Logging & Monitoring

← 03 Scheduling · Index · 05 Application Lifecycle Management →

Contents

• 4.1 Logging and Monitoring Section Introduction
• 4.2 Monitoring Cluster Components
• 4.3 Lab Solution Monitor Cluster Components
• 4.4 Managing Application Logs
• 4.5 Lab Solution Logging Optional

4.1 Logging and Monitoring Section IntroductionCopy Anchor Link

Source: KodeKloud notes

Section scope: observe control plane + nodes, application metrics, logging (pod/node/kubelet), centralized aggregation, dashboards, alerting.

Areas and toolsCopy Anchor Link

• EFK = Elasticsearch + Fluentd/Fluent Bit + Kibana (ELK uses Logstash).
• Metrics vs logs: metrics → trends + alerting; logs → detailed debugging/forensics. Use both for full observability.
• Don't ship sensitive/PII data to external log backends without masking, encryption and strict access control; review retention/access policies first.

Quick troubleshooting commandsCopy Anchor Link

kubectl logs <pod-name> [-c <container-name>]   # -c selects container; otherwise annotation/first container is used
kubectl logs -f <pod-name>                      # follow/stream
kubectl describe pod <pod-name>                 # events at the bottom
kubectl exec -it <pod-name> -- /bin/sh          # shell into a running container

Suggested practice (from the course)Copy Anchor Link

• Deploy metrics-server, validate with kubectl top (§4.2, §4.3).
• Install Prometheus + Grafana; dashboard for cluster CPU/memory.
• Instrument an app with a /metrics endpoint; configure Prometheus to scrape it.
• Debug a failing Pod with kubectl logs; forward logs to Fluent Bit, view in Kibana or Grafana Explore.
• Alertmanager rules (e.g. node memory pressure, Pod CrashLoopBackOff) + test notification routing.

[!NOTE] Beyond the course: CKA scope here is only metrics-server + kubectl top, and kubectl logs/node-level logs. Prometheus, Grafana, EFK, Loki and Alertmanager are not exam tasks. Node/control-plane log locations for troubleshooting:

Copy Text

journalctl -u kubelet -f                 # kubelet runs as a systemd service (not a Pod)
kubectl logs -n kube-system kube-apiserver-controlplane   # static-Pod control-plane components
crictl ps -a; crictl logs <container-id>  # when the API server is down
ls /var/log/pods /var/log/containers      # raw container log files on the node

4.2 Monitoring Cluster ComponentsCopy Anchor Link

Source: KodeKloud notes

Why monitorCopy Anchor Link

• Catch node failures / resource exhaustion early.
• Per-Pod resource usage → troubleshooting and sizing.
• Feed autoscaling (HPA/VPA).
• Historical metrics → capacity planning, trend analysis.

What to monitorCopy Anchor Link

Monitoring solutionsCopy Anchor Link

• Kubernetes has no full built-in monitoring solution. Options:
  • Metrics Server — lightweight, in-memory; source for kubectl top and autoscalers.
  • Prometheus — long-term time-series storage, alerting, PromQL.
  • Elastic Stack (EFK/ELK) — logging + log-based insights.
  • Commercial SaaS — Datadog, Dynatrace.
• Heapster = original monitoring/analysis project; deprecated, replaced by Metrics Server (still seen in old architectures/docs).

Metrics ServerCopy Anchor Link

• Typically one instance per cluster.
• Polls each node's kubelet (which embeds cAdvisor — container Advisor — exposing node + container stats via the Summary API).
• Aggregates in memory and serves via the Kubernetes Metrics API → used by kubectl top and HPA.
• No disk persistence, no history → for historical data/advanced queries use Prometheus or another TSDB.

Diagram (new): kubelet/cAdvisor stats on every node are scraped by metrics-server and exposed through the API server's Metrics API to kubectl top and HPA. API path, port and scrape interval are beyond-course details.

[!NOTE] Beyond the course: the Metrics API is metrics.k8s.io/v1beta1, registered with the API server's aggregation layer via an APIService object (kubectl get apiservice v1beta1.metrics.k8s.io → AVAILABLE=True when healthy). Current metrics-server (v0.6+) scrapes the kubelet's /metrics/resource endpoint on port 10250 (older versions used /stats/summary); default --metric-resolution=15s. You can query raw: kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes.

Deploying Metrics ServerCopy Anchor Link

# Minikube
minikube addons enable metrics-server

# Other environments (course method)
git clone https://github.com/kubernetes-sigs/metrics-server.git
kubectl apply -f metrics-server/deploy/1.8+/

• Give it a short time after deployment before metrics appear.

[!CAUTION] Correction: the deploy/1.8+/ directory no longer exists in the upstream repo (the course itself says the path may change). Current upstream install:

Copy Text

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Installs into kube-system. On lab/kubeadm clusters with self-signed kubelet certs, the Pod may stay not-ready with x509 errors until you add --kubelet-insecure-tls to the container args (lab/dev only).

Viewing live metrics — kubectl topCopy Anchor Link

$ kubectl top nodes
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
kubemaster   166m         8%     1337Mi          70%
kubenode1    36m          1%     1046Mi          55%
kubenode2    39m          1%     1048Mi          55%

[!CAUTION] Correction: the course shows kubectl top pods output with CPU%/MEMORY% columns. Real output for Pods has only NAME CPU(cores) MEMORY(bytes) (percentages exist only for nodes, relative to allocatable) — as seen in the lab output in §4.3.

[!NOTE] Beyond the course — exam-useful flags:

Copy Text

kubectl top pod -A --sort-by=memory          # or --sort-by=cpu; highest first
kubectl top pod <pod> --containers           # per-container usage
kubectl top pod -l app=web -n prod           # filter by label/namespace
kubectl top node --sort-by=cpu --no-headers | head -1   # busiest node, scriptable

Troubleshooting kubectl topCopy Anchor Link

• No data / error: Metrics API not available → check Metrics Server Pods are running and read their logs (kubectl -n kube-system get pods, kubectl logs ...).
• Kubelet Summary API must be reachable from Metrics Server — network policies, firewalls, or RBAC can block it.
• Need history? Metrics Server has none → Prometheus/TSDB.

When to add Prometheus (full stack)Copy Anchor Link

• Long-term retention/history; PromQL / complex dashboards; alerting + recording rules; Grafana, log/trace correlation. Optionally add EFK for logs.

[!WARNING] Exam traps / remember:

• kubectl top needs metrics-server; it is not installed by kubeadm by default.
• Metrics are live only — nothing historical; restart of metrics-server loses everything.
• cAdvisor lives inside the kubelet; there is no separate cAdvisor Pod.
• Heapster = deprecated; any answer mentioning it for current clusters is wrong.

4.3 Lab Solution Monitor Cluster ComponentsCopy Anchor Link

Source: KodeKloud notes

Theory: §4.2. Prereq: kubeconfig with rights to create cluster-scoped objects.

DrillCopy Anchor Link

git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git
cd kubernetes-metrics-server
kubectl create -f .        # creates every manifest in the directory
# Lab output:
# namespace/metrics-server created
# serviceaccount/metrics-server created
# clusterrole.rbac.authorization.k8s.io/system:metrics-server created
# clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
# service/metrics-server created
# deployment.apps/metrics-server created

• Lab repo files (README: development/learning only, insecure, not for production): aggregated-metrics-reader.yaml, auth-delegator.yaml, auth-reader.yaml, metrics-apiservice.yaml, metrics-server-deployment.yaml, metrics-server-service.yaml, resource-reader.yaml. For production follow the official repo kubernetes-sigs/metrics-server.

$ kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
controlplane  470m         1%     1252Mi          0%
node01        57m          0%     349Mi           0%

$ kubectl top pods
NAME      CPU(cores)   MEMORY(bytes)
elephant  20m          32Mi
lion      1m           18Mi
rabbit    131m         252Mi

[!TIP] Answer "most/least" questions without eyeballing: kubectl top pod --sort-by=memory (highest first) or kubectl top node --sort-by=cpu. (--sort-by is beyond the course.)

4.4 Managing Application LogsCopy Anchor Link

Source: KodeKloud notes

PrincipleCopy Anchor Link

• Containerised apps should log to stdout / stderr; the container runtime captures the streams and CLI tools expose them. Same rule in Kubernetes: kubelet (via the runtime) captures them → kubectl logs.

Diagram (new): runtime writes container stdout/stderr to files on the node; kubectl logs goes API server → kubelet → file; node log agents tail the same files. File paths and the API path are beyond-course details.

DockerCopy Anchor Link

docker run kodekloud/event-simulator        # foreground: logs print to terminal
docker run -d kodekloud/event-simulator     # detached: prints container ID, no logs shown
docker logs -f <container-id-or-name>       # stream logs

Sample output: 2018-10-06 15:57:15,937 - root - INFO - USER1 logged in (login/logout/page-view events, one per second).

KubernetesCopy Anchor Link

# event-simulator.yaml
apiVersion: v1
kind: Pod
metadata:
  name: event-simulator-pod
spec:
  containers:
    - name: event-simulator
      image: kodekloud/event-simulator

kubectl create -f event-simulator.yaml
kubectl logs -f event-simulator-pod          # stream, same output as docker logs

Multi-container PodsCopy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: event-simulator-pod
spec:
  containers:
    - name: event-simulator
      image: kodekloud/event-simulator
    - name: image-processor
      image: some-image-processor       # placeholder image from the course

kubectl logs -f event-simulator-pod -c event-simulator   # -c / --container selects the container

• Course lab: omitting -c produced an error asking for a container.

[!CAUTION] Correction: current kubectl logs uses the kubectl.kubernetes.io/default-container annotation, or the first container, when -c is omitted. It may print Defaulted container ... out of: ...; pass -c explicitly to avoid reading the wrong logs. See default-container.

Quick referenceCopy Anchor Link

ProductionCopy Anchor Link

• docker logs / kubectl logs suffice for dev and basic troubleshooting.
• Production: cluster-wide collectors/aggregators (Fluentd / Fluent Bit → Elasticsearch + Kibana, or hosted SaaS) to collect, index, retain, search and alert centrally.
• Next steps: structured (JSON) logging, sidecar log processors, Fluentd/Fluent Bit shipping.

[!NOTE] Beyond the course: dockershim was removed in v1.24 — on cluster nodes use crictl logs <container-id> (find IDs with crictl ps -a), not docker logs. Other kubectl logs flags worth knowing:

Copy Text

kubectl logs <pod> -p                       # --previous: logs of the crashed/restarted container instance
kubectl logs <pod> --all-containers         # every container (incl. init) in the Pod
kubectl logs <pod> --tail=50 --since=1h     # last 50 lines / last hour
kubectl logs <pod> --timestamps
kubectl logs deploy/<name>                  # picks one Pod of the Deployment
kubectl logs -l app=web --all-containers    # by label selector
kubectl logs <pod> -c <init-container>      # init containers too

Logs live on the node and are lost when the Pod is deleted (kubelet rotates them, default 10Mi × 5 files).

[!WARNING] Exam traps / remember:

• Multi-container Pod: always select -c <name>; current kubectl can default to a different container than intended.
• CrashLoopBackOff: current logs may be empty — use kubectl logs -p.
• Task "save logs to a file": kubectl logs <pod> -c <ctr> > /path/file (run on the host the task specifies).

4.5 Lab Solution Logging OptionalCopy Anchor Link

Source: KodeKloud notes

Theory: §4.4.

DrillCopy Anchor Link

Sample log line format: [2022-04-16 19:55:58,923] WARNING in event-simulator: USER7 Order failed as the item is OUT OF STOCK. (first Pod also shows USER7 out-of-stock warnings).

TechniquesCopy Anchor Link

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].name}'   # list container names
kubectl describe pod <pod>                                        # also lists containers
kubectl logs <pod> -c <container>                                 # pick a container
kubectl logs <pod> | grep USER5                                    # filter by user/keyword
kubectl logs <pod> -c simple-webapp | grep -i warning              # only warnings

• Check Pod status first; specify -c for multi-container Pods.
• Scan for repeated WARNING/ERROR lines and correlate timestamps with the reported incident.

[!TIP] The Defaulted container ... out of: ... line (or older error) lists container names. Still select -c explicitly. Use grep on user IDs/keywords instead of scrolling.

← 03 Scheduling · Index · 05 Application Lifecycle Management →

Reference

Based on the KodeKloud Certified Kubernetes Administrator (CKA) course notes.