Anurag's Notes

Application Lifecycle Management

← 04 Logging & Monitoring · Index · 06 Cluster Maintenance →

Contents


5.1 Application Lifecycle Management Section Introduction

Source: KodeKloud notes

Section scope: changing a live app safely, configuring it, scaling it, keeping it healthy. Overlaps heavily with CKAD; if already familiar, prioritise the labs.

TopicKey primitivesQuick example
Rolling updates & rollbacks (§5.2–5.3)Deployment, ReplicaSetkubectl set image deployment/my-app my-app=nginx:1.19 · rollback kubectl rollout undo deployment/my-app
Configuring applications (§5.4–5.12)commands/args, env vars, ConfigMap, Secret, volume mountsenvFrom: [{configMapRef: {name: my-config}}]
Scaling applications (§5.16–5.19)Deployment replicas, HorizontalPodAutoscalerkubectl scale deployment my-app --replicas=5 · kubectl autoscale deployment my-app --min=2 --max=10 --cpu-percent=70
Self-healing applicationsreadinessProbe, livenessProbe, ReplicaSet, DeploymentreadinessProbe: {httpGet: {path: /healthz, port: 8080}, initialDelaySeconds: 5}

Multi-container Pods / init containers follow in §5.13–5.15.

5.2 Rolling Updates and Rollbacks

Source: KodeKloud notes

Rollouts and revisions

  • Creating a Deployment triggers a rollout; each rollout = a new revision (Revision 1 → e.g. nginx:1.7.0, Revision 2 → nginx:1.7.1).
  • Each revision is backed by its own ReplicaSet. Revision history lets you track changes and roll back.

Revision 1 (nginx:1.7.0) and Revision 2 (nginx:1.7.1) as separate Pod sets

kubectl rollout status deployment/myapp-deployment    # live progress: "Waiting for rollout to finish: 3 of 10 updated replicas are available..." -> "successfully rolled out"
kubectl rollout history deployment/myapp-deployment   # REVISION + CHANGE-CAUSE table
deployments "myapp-deployment"
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl apply --filename=deployment-definition.yml --record=true
  • CHANGE-CAUSE is filled only if a change-cause was recorded (historically --record, which is deprecated); otherwise <none>.

Update strategies

StrategyBehaviourDowntimeDefault?
RecreateScale old ReplicaSet to 0, then scale new ReplicaSet upYes ("application down" gap)No
RollingUpdateTake down old Pods / bring up new Pods a few at a timeNoYes (used if strategy omitted)

Recreate (all old Pods down, app down, then all new up) vs RollingUpdate (old and new interleaved)

Triggering an update

  • A rollout (new revision) happens when the Pod template changes: image, labels, env, resources, etc.
# 1. Declarative: edit the manifest, then apply -> new rollout
kubectl apply -f deployment-definition.yml

# 2. Imperative: change image (syntax <container-name>=<new-image>)
kubectl set image deployment/myapp-deployment nginx-container=nginx:1.9.1
  • kubectl set image changes only the live object → your manifest file now drifts from the cluster; update the file too.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  labels:
    app: myapp
    type: front-end
spec:
  replicas: 3
  selector:
    matchLabels:
      type: front-end          # must match template labels
  template:
    metadata:
      name: myapp-pod
      labels:
        app: myapp
        type: front-end
    spec:
      containers:
      - name: nginx-container  # name used in `kubectl set image ... nginx-container=<image>`
        image: nginx:1.7.1     # changing this triggers a rollout

[!CAUTION] Correction: the original lists "modifying replica counts" as an update. Changing spec.replicas (or kubectl scale) only scales the current ReplicaSet — it does not create a new revision/rollout. Only changes under spec.template do.

Telling the strategy apart with kubectl describe deployment

Field / signalRecreateRollingUpdate
StrategyTypeRecreateRollingUpdate
RollingUpdateStrategy lineabsent25% max unavailable, 25% max surge (defaults)
Eventsold RS scaled down to 0, then new RS scaled up to 5interleaved: new RS up to 2, old down to 4, new up to 3, old down to 3 …
Replicas line (course examples)e.g. 5 desired | 5 updated | 5 totale.g. 5 desired | 5 updated | 6 total | 4 available | 2 unavailable
  • Other describe fields: Annotations: deployment.kubernetes.io/revision=2, kubectl.kubernetes.io/last-applied-configuration, kubernetes.io/change-cause; MinReadySeconds; Conditions (Available … MinimumReplicasAvailable, Progressing … NewReplicaSetAvailable/ReplicaSetUpdated); OldReplicaSets / NewReplicaSet.

Under the hood

  • Deployment → creates ReplicaSet-1 → creates Pods.
  • Template change → Deployment creates ReplicaSet-2 and shifts Pods from RS-1 to RS-2 per the strategy.
  • Old ReplicaSet is kept (scaled to 0) — that is what makes rollback possible.

Deployment containing ReplicaSet-1 (old Pods being removed) and ReplicaSet-2 (new Pods being added)

Rollback

kubectl get replicasets                              # before: old RS 0/0/0, new RS 5/5/5
kubectl rollout undo deployment/myapp-deployment     # "deployment ... rolled back"  (`deployment myapp-deployment` form also works)
kubectl get replicasets                              # after: old RS 5/5/5, faulty RS 0/0/0

Command reference

PurposeCommand
Createkubectl create -f deployment-definition.yml
Listkubectl get deployments
Apply manifest changeskubectl apply -f deployment-definition.yml
Update imagekubectl set image deployment/myapp-deployment nginx-container=nginx:1.9.1
Rollout statuskubectl rollout status deployment/myapp-deployment
Rollout historykubectl rollout history deployment/myapp-deployment
Rollbackkubectl rollout undo deployment/myapp-deployment
Inspectkubectl describe deployment myapp-deployment
List ReplicaSetskubectl get replicasets

[!NOTE] Beyond the course:

kubectl rollout history deployment/myapp --revision=2      # show the Pod template of one revision
kubectl rollout undo deployment/myapp --to-revision=1      # roll back to a specific revision
kubectl rollout restart deployment/myapp                   # new rollout with same spec (e.g. pick up changed ConfigMap/Secret env)
kubectl rollout pause deployment/myapp ; kubectl rollout resume deployment/myapp
kubectl annotate deployment/myapp kubernetes.io/change-cause="image -> nginx:1.9.1"   # replacement for --record
  • spec.revisionHistoryLimit (default 10) = how many old ReplicaSets are kept for rollback.
  • A rollback re-uses the old template and gets a new revision number (the old number disappears from history).
  • Strategy fields: spec.strategy.rollingUpdate.maxSurge (extra Pods above replicas, % rounds up) and maxUnavailable (% rounds down); both default 25%; they cannot both be 0.

[!WARNING] Exam traps / remember:

  • Default strategy is RollingUpdate; Recreate causes downtime.
  • set image syntax is <container-name>=<image> — find the container name with kubectl describe deploy / -o yaml, it is not the Deployment name.
  • --record is deprecated; don't rely on CHANGE-CAUSE being populated.

5.3 Lab Solution Rolling update

Source: KodeKloud notes

Lab env: k3s cluster (alias kubectl='k3s kubectl', alias k='kubectl'); Deployment frontend, 4 replicas, served via the lab portal; ./curl-test.sh simulates continuous traffic. Theory: see §5.2.

#TaskCommand / answerGotcha
1Pods / Deployments presentk get pod (4× frontend-…), k get deploy (frontend 4/4)—
2Current imagek describe deploy frontend → Image: kodekloud/webapp-color:v1 (app shows v1, blue)—
3Current strategysame describe → StrategyType: RollingUpdate, RollingUpdateStrategy: 25% max unavailable, 25% max surgeLab also sets MinReadySeconds: 20, slowing the rollout
4Max Pods down at once during rolling update25% of 4 = 1—
5Upgrade to v2k set image deploy frontend simple-webapp=kodekloud/webapp-color:v2container name is simple-webapp
6Observe rolloutk describe deploy frontend → revision 2, 4 desired | 2 updated | 5 total | 3 available | 2 unavailable, OldReplicaSets: frontend-5c74c57d95 (3/3), NewReplicaSet: frontend-d495f944b (2/2)./curl-test.sh shows v1/blue and v2/green interleaved, no errors
7Change strategy to Recreatek edit deploy frontend → spec.strategy.type: Recreate; verify StrategyType: Recreatedelete the rollingUpdate: block (see CAUTION). After the switch, describe deploy frontend loses the RollingUpdateStrategy line; the completed lab rollout shows OldReplicaSets: <none> and frontend-d495f944b (4/4 replicas created)
8Upgrade to v3k set image deploy frontend simple-webapp=kodekloud/webapp-color:v3while old Pods are gone and new not ready → 502 Bad Gateway (nginx ingress); then all v3/red
# kubectl edit deployment frontend  (relevant part after the change)
spec:
  minReadySeconds: 20
  replicas: 4
  selector:
    matchLabels:
      name: webapp
  strategy:
    type: Recreate            # was RollingUpdate; remove the rollingUpdate: {maxSurge, maxUnavailable} block
  template:
    metadata:
      labels:
        name: webapp
    spec:
      containers:
      - name: simple-webapp
        image: kodekloud/webapp-color:v2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080

[!CAUTION] Correction: the original fragment keeps rollingUpdate: {maxSurge: 25%, maxUnavailable: 25%} together with type: Recreate. The API server rejects the rollingUpdate field when the strategy type is Recreate. Remove the rollingUpdate block when switching to Recreate.

StrategyProsCons
RollingUpdate (default)Minimal downtime; traffic keeps flowing; honours maxUnavailable/maxSurgeFull update takes longer; two versions serve simultaneously
RecreateSimple lifecycle; needed when old and new versions can't coexist (some stateful/incompatible changes)Outage until new Pods are ready; use only when the downtime is acceptable or external buffering can absorb it

[!TIP] Fast strategy switch without an editor (Beyond the course): kubectl patch deploy frontend -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

5.4 Commands and Arguments in Docker

Source: KodeKloud notes

Not a direct exam objective, but required to understand §5.5.

Container = its main process

  • Container lives only while its main process runs; process exits → container exits.
  • Containers run one service/job (web server, DB, batch task), not a full OS.
  • Image declares the default process via CMD/ENTRYPOINT: service images (nginx, MySQL) start their daemon; base images (Ubuntu) default to a shell.
docker run ubuntu        # exits immediately
docker ps                # nothing running
docker ps -a             # 45aacca36850  ubuntu  "/bin/bash"  ...  Exited (0) 41 seconds ago
  • Why: Ubuntu's default is bash; bash needs a terminal, Docker attaches no TTY by default → bash finds no terminal and exits.
# nginx (simplified)
CMD ["nginx"]

# mysql (simplified): entrypoint script + default server command
COPY docker-entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
CMD ["mysqld"]

# ubuntu-style base image
FROM ubuntu:14.04
RUN apt-get update && apt-get install -y \
    build-essential software-properties-common \
    byobu curl git htop man unzip vim wget \
    && rm -rf /var/lib/apt/lists/*
ENV HOME /root
WORKDIR /root
CMD ["bash"]

Overriding at runtime / baking into an image

  • Anything after the image name in docker run replaces CMD (or is appended to ENTRYPOINT if one is set).
docker run ubuntu sleep 5     # runs sleep 5; container up 5 s then exits
  • Permanent: build a derived image with CMD.
CMD formSyntaxNotes
ShellCMD sleep 5runs via /bin/sh -c
Exec (JSON) — recommendedCMD ["sleep", "5"]no shell; executable gets signals directly. Each token its own string; first = executable. ["sleep 5"] is wrong
FROM ubuntu
CMD ["sleep", "5"]
docker build -t ubuntu-sleeper .
docker run ubuntu-sleeper        # sleeps 5 s, exits

ENTRYPOINT vs CMD

InstructionRoledocker run behaviour
CMDDefault command or default argsFully replaced by run-time args: CMD ["sleep","5"] + docker run image 10 → runs 10 (fails)
ENTRYPOINTFixed executableRun-time args appended: ENTRYPOINT ["sleep"] + docker run image 10 → sleep 10
ENTRYPOINT + CMDExecutable + default argsdocker run image → sleep 5; docker run image 10 → sleep 10
--entrypoint flagOverride ENTRYPOINT at run timedocker run --entrypoint sleep2.0 image 10 → sleep2.0 10
FROM ubuntu
ENTRYPOINT ["sleep"]
CMD ["5"]
docker run ubuntu-sleeper                           # sleep 5
docker run ubuntu-sleeper 10                        # sleep 10
docker run --entrypoint sleep2.0 ubuntu-sleeper 10  # sleep2.0 10

Best practice

  • Use exec/JSON form for both: ENTRYPOINT ["executable","arg1"], CMD ["default","args"].
  • Avoid shell form ENTRYPOINT executable arg1: a shell wraps the process → signal propagation (SIGTERM) and argument handling change.

[!NOTE] Beyond the course: ENTRYPOINT ["sleep"] without CMD and without run-time args → sleep: missing operand — hence the default CMD ["5"]. Shell-form ENTRYPOINT ignores CMD and docker run args entirely, and PID 1 is /bin/sh, so docker stop SIGTERM doesn't reach the app (killed after the grace period). docker run -it ubuntu keeps bash alive by attaching a TTY.

5.5 Commands and Arguments in Kubernetes

Source: KodeKloud notes

Mapping

DockerfilePod field (spec.containers[])Effect
ENTRYPOINTcommandReplaces image ENTRYPOINT (the executable)
CMDargsReplaces image CMD (default args to the entrypoint)
  • docker run ubuntu-sleeper 10 (extra tokens after image) ≡ args: ["10"].
  • docker run --entrypoint sleep2.0 ubuntu-sleeper 10 ≡ command: ["sleep2.0"] + args: ["10"].

Pod command overrides ENTRYPOINT, args overrides CMD; resulting process for each combination Diagram (new): field mapping and what the container actually runs for each combination (row 3 per Kubernetes docs).

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper-pod
spec:
  containers:
  - name: ubuntu-sleeper
    image: ubuntu-sleeper
    command: ["sleep2.0"]   # overrides ENTRYPOINT ["sleep"]  (omit to keep image entrypoint)
    args: ["10"]            # overrides CMD ["5"]
  • Both fields are arrays of strings. No shell expansion unless you run a shell explicitly: command: ["/bin/sh", "-c", "echo hello"].

[!WARNING] Exam traps / remember:

  • command is not Docker's CMD — command ↔ ENTRYPOINT, args ↔ CMD.
  • Beyond the course: setting command without args drops the image's CMD too (only your command runs).

[!NOTE] Beyond the course — imperative generation:

kubectl run sleeper --image=busybox --command -- sleep 1000                 # --command: tokens after -- become `command`
kubectl run webapp-green --image=kodekloud/webapp-color -- --color=green    # without --command: tokens become `args`
kubectl run sleeper --image=busybox --dry-run=client -o yaml --command -- sleep 1000 > pod.yaml
docker run --name ubuntu-sleeper ubuntu-sleeper      # name the container, use image CMD
docker run --name ubuntu-sleeper ubuntu-sleeper 10   # alternative: override CMD with 10
  • These are alternatives: remove the first stopped container or choose another name before reusing --name.

5.6 Lab Solution Commands and Arguments Optional

Source: KodeKloud notes

Theory: §5.4–5.5.

1. How many Pods? → kubectl get pods (one).

2. Command used by Pod ubuntu-sleeper? → kubectl describe pod ubuntu-sleeper:

Containers:
  ubuntu:
    Image:          ubuntu
    Command:
      sleep
      4800
  • One element per line = array ["sleep", "4800"]. If args are set, a separate Args: block appears.

3. Create ubuntu-sleeper-2 (ubuntu) that sleeps 5000 s — add to the given file either form:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper-2
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    command: ["sleep"]
    args: ["5000"]          # option A: describe shows Command: sleep / Args: 5000
    # command: ["sleep", "5000"]   # option B: executable + arg together in command
kubectl create -f ubuntu-sleeper-2.yaml
kubectl describe pod ubuntu-sleeper-2

4. Fix ubuntu-sleeper-3.yaml — unquoted number in the array:

    command:
    - "sleep"
    - 1200            # YAML int -> rejected
Error from server (BadRequest): ... json: cannot unmarshal number into Go struct field Container.spec.containers.command of type string

Fix: - "1200" → kubectl create -f ubuntu-sleeper-3.yaml.

5. Change ubuntu-sleeper-3 to sleep 2000 — command is immutable on a running Pod; kubectl edit fails with:

spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`,
`spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds`
# edit the file ("1200" -> "2000"), then either:
kubectl delete pod ubuntu-sleeper-3 && kubectl create -f ubuntu-sleeper-3.yaml
kubectl replace --force -f ubuntu-sleeper-3.yaml        # deletes + recreates in one step
  • Delete = graceful shutdown: container gets SIGTERM and may take a while to exit. Beyond the course: sleep as PID 1 ignores SIGTERM, so deletion waits the full grace period (30 s default); add --grace-period=0 --force to skip the wait.

6. Inspect Dockerfiles (/root/webapp-color/):

# /root/webapp-color/Dockerfile
FROM python:3.6-alpine
RUN pip install flask
COPY . /opt/
EXPOSE 8080
WORKDIR /opt
ENTRYPOINT ["python", "app.py"]          # startup command: python app.py

# /root/webapp-color/Dockerfile2  (same, plus:)
ENTRYPOINT ["python", "app.py"]
CMD ["--color", "red"]                   # startup: python app.py --color red

7. webapp-color-2/webapp-color-pod.yaml (dir also has Dockerfile2) — what runs?

apiVersion: v1
kind: Pod
metadata:
  name: webapp-green
  labels:
    name: webapp-green
spec:
  containers:
  - name: simple-webapp
    image: kodekloud/webapp-color
    command: ["--color", "green"]   # replaces ENTRYPOINT -> tries to exec "--color" (broken)
  • Answer: --color green — command overrides the image's python app.py (and its CMD). To pass flags to the existing entrypoint use args: ["--color", "green"] and leave command unset.

[!NOTE] Beyond the course: the lab's final task (create Pod webapp-green, image kodekloud/webapp-color, run with --color=green) in one line: kubectl run webapp-green --image=kodekloud/webapp-color -- --color=green → sets args, keeps ENTRYPOINT.

[!WARNING] Exam traps / remember:

  • Quote every numeric-looking element of command/args ("1200").
  • Pod command/args/env are immutable → recreate (replace --force) — or use a Deployment, which rolls out the change.

5.7 Configuring Environment Variables in Applications

Source: KodeKloud notes

  • Docker: docker run -e APP_COLOR=pink simple-webapp-color.
  • Kubernetes: env = list under the container; each item has name + either value (literal string) or valueFrom (reference).
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    ports:
    - containerPort: 8080
    env:
    - name: APP_COLOR
      value: "pink"                 # literal
    - name: APP_MODE
      valueFrom:
        configMapKeyRef:            # one key from a ConfigMap
          name: my-configmap
          key: app_mode
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:               # one key from a Secret
          name: my-secret
          key: db_password
SourceUse forField
LiteralSimple, non-sensitive stringsvalue:
ConfigMap keyNon-sensitive config managed separatelyvalueFrom.configMapKeyRef {name, key}
Secret keyPasswords, tokensvalueFrom.secretKeyRef {name, key}
  • Referenced ConfigMap/Secret must exist in the Pod's namespace before the Pod starts (kubectl create configmap / kubectl create secret).
  • Never put sensitive data in ConfigMaps; use Secrets (+ encryption at rest, RBAC) — see §5.10, §5.12.

[!NOTE] Beyond the course: value must be a string — quote numbers/booleans (value: "3306", value: "true"). A missing referenced ConfigMap/key leaves the Pod in CreateContainerConfigError unless optional: true is set.

5.8 Configuring ConfigMaps in Applications

Source: KodeKloud notes

Why

  • Pull env vars / small config files out of Pod manifests into a central key-value object; many Pods reuse it; config separate from code.
  • Not for secrets (passwords, tokens) → Secrets.
  • Workflow: (1) create ConfigMap, (2) inject into Pod.

Create

ApproachWhenExample
ImperativeQuick / scripts / CIkubectl create configmap app-config --from-literal=APP_COLOR=blue
DeclarativeSource-controlled, reviewable (GitOps)kubectl apply -f configmap.yaml
kubectl create configmap app-config \
  --from-literal=APP_COLOR=blue \
  --from-literal=APP_MODE=prod               # repeat flag per key

kubectl create configmap app-config --from-file=config.properties   # key = file name, value = file content
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config          # descriptive names: app-config, mysql-config, redis-config
data:                       # note: data, not spec
  APP_COLOR: "blue"
  APP_MODE: "prod"

Inspect

kubectl get configmaps                 # NAME  DATA(=number of keys)  AGE
kubectl describe configmap app-config  # Data section lists each key with its value

Inject into Pods

ConfigMap app-config consumed via envFrom, single env valueFrom, and volume mount Diagram (new): the three consumption methods and what appears inside the container.

# Method 1: envFrom — all keys become env vars (APP_COLOR=blue, APP_MODE=prod)
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    ports:
    - containerPort: 8080
    envFrom:
    - configMapRef:
        name: app-config
---
# Method 2: single key -> single env var
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-single-env
spec:
  containers:
  - name: simple-webapp
    image: simple-webapp-color
    env:
    - name: APP_COLOR
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_COLOR
---
# Method 3: volume — each key becomes a file (/etc/app-config/APP_COLOR contains "blue")
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-config-volume
spec:
  containers:
  - name: simple-webapp
    image: simple-webapp-color
    volumeMounts:
    - name: app-config-volume
      mountPath: /etc/app-config
  volumes:
  - name: app-config-volume
    configMap:
      name: app-config
MethodUse whenResult in container
envFrom + configMapRefMany keys map 1:1 to env varsEvery key → env var
env + valueFrom.configMapKeyRefNeed specific keys / renameOne env var per entry
configMap volumeApp reads config files / structured filesEvery key → file at mountPath

Updates and best practices

  • ConfigMap consumed as env vars → change is not seen until the Pod is recreated/restarted.
  • Volume-mounted ConfigMap → files updated automatically in many cases; whether the app notices depends on how it reads files.
  • Keep ConfigMaps declarative for auditability; name them after their consumer.

[!NOTE] Beyond the course:

  • --from-file=<key>=<path> sets a custom key; --from-file=<dir>/ adds one key per file; --from-env-file=app.env loads KEY=VALUE lines as separate keys.
  • Volume updates arrive after the kubelet sync delay (up to ~1 min); subPath mounts never update.
  • Restart Pods of a Deployment after changing env-sourced config: kubectl rollout restart deployment/<name>.
  • ConfigMap size limit 1 MiB; kubectl create configmap ... --dry-run=client -o yaml generates YAML quickly.

5.9 Lab Solution Env Variables Optional

Source: KodeKloud notes

Theory: §5.7–5.8.

#TaskCommand / answer
1Pods presentkubectl get pods → one Pod webapp-color
2Env var name/valuekubectl describe pod webapp-color → Environment: APP_COLOR: pink (UI background pink)
3Change color to greenkubectl edit pod webapp-color (pink → green) → rejected (immutable) → kubectl replace --force -f /tmp/kubectl-edit-<random>.yaml
4Number of ConfigMaps in defaultkubectl get configmap → 2 (kube-root-ca.crt, db-config)
5DB host in db-configkubectl describe configmap db-config → DB_HOST: SQL01.example.com (also DB_NAME: SQL01, DB_PORT: 3306)
6Create ConfigMap webapp-config-map with APP_COLOR='dark blue'kubectl create configmap webapp-config-map --from-literal=APP_COLOR='dark blue' · verify kubectl get configmap webapp-config-map -o yaml
7Make webapp-color use itkubectl edit pod webapp-color → add envFrom → save (rejected) → kubectl replace --force -f /tmp/kubectl-edit-<random>.yaml

Edit-then-replace technique (immutable Pod fields)

  • kubectl edit on a forbidden field → error + kubectl saves your edited copy to /tmp/kubectl-edit-XXXXXXXXXX.yaml (path printed in the error).
  • kubectl replace --force -f <that file> → pod "webapp-color" deleted / pod/webapp-color replaced.
  • Long-term: use a Deployment, which rolls out template changes.
# container section after step 7
spec:
  containers:
  - name: webapp-color
    image: kodekloud/webapp-color
    imagePullPolicy: Always
    envFrom:
    - configMapRef:
        name: webapp-config-map
    resources: {}
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-2nbfl
      readOnly: true

Verify with kubectl describe pod webapp-color (UI now dark blue):

    Environment Variables from:
      webapp-config-map   ConfigMap  Optional: false
    Environment:    <none>

Reference snippets (Kubernetes docs dapi-test-pod)

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
  - name: test-container
    image: registry.k8s.io/busybox
    command: ["/bin/sh", "-c", "env"]     # prints env and exits
    env:
    - name: SPECIAL_LEVEL_KEY             # single key, renamed
      valueFrom:
        configMapKeyRef:
          name: special-config
          key: special.how
    # alternative — import all keys:
    # envFrom:
    # - configMapRef:
    #     name: special-config
  restartPolicy: Never

[!CAUTION] Correction: the original uses k8s.gcr.io/busybox. The k8s.gcr.io registry is frozen; use registry.k8s.io/busybox (or plain busybox).

[!NOTE] Beyond the course: instead of relying on the /tmp file, dump first: kubectl get pod webapp-color -o yaml > p.yaml, edit, kubectl replace --force -f p.yaml. Changing env via a Deployment instead: kubectl set env deployment/<name> --from=configmap/webapp-config-map.

[!WARNING] Exam traps / remember:

  • kube-root-ca.crt exists in every namespace (auto-created) — count it when asked "how many ConfigMaps".
  • describe pod shows envFrom sources under Environment Variables from, not under Environment.
  • Values with spaces need quoting in the shell: --from-literal=APP_COLOR='dark blue'.

5.10 Secrets

Source: KodeKloud notes

Why Secrets

  • Anti-pattern: credentials hard-coded in app code. The course example is a Flask app calling mysql.connector.connect(host='mysql', database='mysql', user='root', password='paswrd').
  • Non-sensitive config (hostnames, flags) belongs in a ConfigMap (see §5.8). Passwords, tokens and keys belong in a Secret.
  • Secret values are base64-encoded, not encrypted. In production, add encryption at rest (§5.12), RBAC, and/or an external KMS / secrets manager.
ConfigMapSecret
Intended forNon-sensitive configSensitive data (passwords, tokens, keys)
Storage formatPlain textBase64-encoded (encoding only, not encryption)
Typical useHostnames, feature flagsDB credentials, API keys
# Contrast only: a password should NOT go in a ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_Host: mysql
  DB_User: root
  DB_Password: paswrd     # wrong place for this; use a Secret

Create: imperative

# From literals (repeat --from-literal per key)
kubectl create secret generic app-secret \
  --from-literal=DB_Host=mysql \
  --from-literal=DB_User=root \
  --from-literal=DB_Password=paswrd

# From a file: the key is the file name, the value is the file content
kubectl create secret generic app-secret --from-file=app_secret.properties

Create: declarative (values under data must be base64)

echo -n 'mysql'  | base64    # bXlzcWw=
echo -n 'root'   | base64    # cm9vdA==
echo -n 'paswrd' | base64    # cGFzd3Jk
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque              # generic secret type
data:
  DB_Host: bXlzcWw=
  DB_User: cm9vdA==
  DB_Password: cGFzd3Jk
kubectl create -f secret-data.yaml    # or: kubectl apply -f secret-data.yaml

Inspect and decode

kubectl get secrets                        # NAME / TYPE / DATA (key count) / AGE
kubectl describe secret app-secret         # shows keys and sizes only, e.g. DB_Password: 6 bytes
kubectl get secret app-secret -o yaml      # shows base64 values
echo 'bXlzcWw=' | base64 --decode          # -> mysql  (GNU: --decode or -d; older macOS: -D)

Example kubectl get secrets output in the course: app-secret Opaque 3 and default-token-mvtkv kubernetes.io/service-account-token 3.

Inject into pods

Two ways: as environment variables (one key or all keys at once) or as files through a secret volume.

# All keys -> env vars (DB_Host, DB_User, DB_Password)
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app-container
    image: your-image:latest
    envFrom:
    - secretRef:
        name: app-secret
  restartPolicy: Never
# Secret as a volume: each key becomes a file holding the decoded value
apiVersion: v1
kind: Pod
metadata:
  name: app-pod-with-secret-volume
spec:
  containers:
  - name: app-container
    image: your-image:latest
    volumeMounts:
    - name: app-secret-volume
      mountPath: /opt/app-secret-volumes
      readOnly: true
  volumes:
  - name: app-secret-volume
    secret:
      secretName: app-secret      # note: secretName (a ConfigMap volume uses name)
  restartPolicy: Never
ls /opt/app-secret-volumes               # DB_Host  DB_Password  DB_User
cat /opt/app-secret-volumes/DB_Password  # paswrd

Best practices

  • No plaintext credentials in ConfigMaps.
  • Treat base64 as encoding. Enable encryption at rest for Secrets.
  • Restrict Secret access with RBAC.
  • For higher assurance, use an external secrets manager or KMS (HashiCorp Vault, AWS KMS, GCP KMS, Azure Key Vault).

[!WARNING] Exam traps / remember:

  • Use echo -n. Without -n the trailing newline gets encoded too and the password is wrong.
  • data: needs base64 values. Plain text there is either rejected (illegal base64 data) or silently decoded into garbage. Use stringData: for plain text.
  • The command is kubectl create secret generic …. The generic subtype is required.
  • describe shows key sizes for Opaque Secrets; ServiceAccount-token Secrets are an exception and show the token. Use -o yaml and decode for Opaque values.
  • Volume source field: secret.secretName. envFrom field: secretRef.name.

[!NOTE] Beyond the course:

  • Single key as an env var:
    env:
    - name: DB_Password
      valueFrom:
        secretKeyRef:
          name: app-secret
          key: DB_Password
  • stringData: accepts plain-text values. The API server encodes them into data.
  • Decode one key in one go: kubectl get secret app-secret -o jsonpath='{.data.DB_Password}' | base64 -d.
  • Generate YAML fast: kubectl create secret generic app-secret --from-literal=k=v --dry-run=client -o yaml > s.yaml.
  • Other types: kubectl create secret docker-registry … (kubernetes.io/dockerconfigjson, used by imagePullSecrets) and kubectl create secret tls <n> --cert=… --key=… (kubernetes.io/tls).
  • Updated Secrets propagate to volume mounts (unless mounted with subPath) after a short delay. Env vars do not update until the pod restarts.
  • Since v1.24 no default-token-xxxxx Secret is auto-created per ServiceAccount (see §7.20). Pods get projected, time-bound tokens. Use kubectl create token <sa> for a token.

5.11 Lab Solution Secrets Optional

Source: KodeKloud notes

Drill

#TaskCommand / answerGotcha
1How many Secrets in defaultkubectl get secrets → 1 (default-token-cr4sr)Only the SA token secret exists (older cluster)
2Data keys in the default token secretkubectl describe secret default-token-cr4sr → ca.crt, namespace, tokendescribe lists key sizes (ca.crt: 570 bytes, namespace: 7 bytes), but prints token: in full
3Type of that secretSame output → kubernetes.io/service-account-token
4Which key is NOT in itAnything other than ca.crt/namespace/token, e.g. DB_Host
5Inspect the appkubectl get deploy (none), kubectl get pods (webapp-pod, mysql), kubectl get svc (webapp-service NodePort 8080:30080, sql01 ClusterIP 3306), kubectl get secretDB service name is sql01 → that is the DB_Host value
6Why does the web app failUI shows DB_Host=Not Set; DB_Database=Not Set; DB_User=Not Set; DB_Password=Not Set; 2003: Can't connect to MySQL server on 'localhost:3306'Root cause: no Secret created or injected
7Create db-secretkubectl create secret generic db-secret --from-literal=DB_Host=sql01 --from-literal=DB_User=root --from-literal=DB_Password=password123Keys are case-sensitive and become env var names exactly. Verify: kubectl describe secret db-secret → 3 keys, type Opaque
8Load all keys of db-secret as env vars into webapp-podAdd envFrom (below) to the container, then recreate the podA running pod's env cannot be edited in place
9Verifykubectl describe pod webapp-pod → Environment Variables from: db-secret Secret Optional: falseThen the app UI shows a successful MySQL connection
# inside spec.containers[0] of webapp-pod
    envFrom:
    - secretRef:
        name: db-secret
kubectl edit pod webapp-pod
# Saving is rejected (pod spec is immutable). kubectl writes your edit to /tmp/kubectl-edit-XXXX.yaml
kubectl replace --force -f /tmp/kubectl-edit-123695794.yaml
# pod "webapp-pod" deleted
# pod/webapp-pod replaced

Reference pattern from the Kubernetes docs (all keys of mysecret as env):

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  USER_NAME: YWRtaW4=           # admin
  PASSWORD: MWYyZDFiMmU2N2Rm    # 1f2d1b2e67df
---
apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod
spec:
  containers:
  - name: test-container
    image: registry.k8s.io/busybox
    command: [ "/bin/sh", "-c", "env" ]
    envFrom:
    - secretRef:
        name: mysecret
  restartPolicy: Never

Security notes (Kubernetes docs, shown in lab)

Kubernetes docs caution: Secrets are stored unencrypted in etcd by default; enable encryption at rest and RBAC

  • By default Secrets sit unencrypted in etcd. Anyone with API access, or with etcd access, can read or modify them.
  • Anyone allowed to create a Pod in a namespace can read every Secret in that namespace. This includes indirect access, e.g. permission to create a Deployment.
  • Minimum steps: enable encryption at rest; use RBAC to restrict reading and writing Secrets; use RBAC to limit who can create or replace Secrets.
  • Three ways a Pod uses a Secret: files in a volume, container env vars, and the kubelet pulling images (imagePullSecrets). The control plane also uses Secrets, e.g. bootstrap token Secrets for node registration.
  • Alternatives to Secrets:
    • ServiceAccount tokens to authenticate to in-cluster apps.
    • Third-party secret managers (e.g. Vault, AWS Secrets Manager).
    • A custom X.509 signer plus CertificateSigningRequests to issue certs to Pods.
    • A device plugin that exposes node-local encryption hardware (e.g. a TPM) to trusted Pods.
  • Don't print or log secret values. Env vars and mounted files can leak through logs, process listings or debug output.

[!WARNING] Exam traps / remember:

  • kubectl get pods READY/STATUS won't tell you the env is missing. Check kubectl describe pod → Environment Variables from:.
  • kubectl replace --force -f deletes and recreates the pod. That is the standard way to apply immutable pod changes.

[!CAUTION] Correction: the docs example's image k8s.gcr.io/busybox comes from a frozen registry. Use registry.k8s.io/busybox or plain busybox.

[!NOTE] Beyond the course: on v1.24+ clusters no default-token-* Secret exists, so Q1 to Q4 may show No resources found or a different count. Answer from what the cluster actually shows.

5.12 Demo Encrypting Secret Data at Rest

Source: KodeKloud notes

Setup: single-node kubeadm control plane, root/sudo on the node (to edit static pod manifests).

Diagram of the API server encrypting Secrets with enc.yaml before writing to etcd; old Secrets stay plaintext until rewritten Diagram (new): the API server loads enc.yaml at startup and encrypts new Secret writes with the first provider. Secrets written earlier stay plaintext until they are rewritten.

1. Show the default (base64 only)

kubectl create secret generic my-secret --from-literal=key1=supersecret
kubectl get secret my-secret                 # Opaque, DATA 1
kubectl describe secret my-secret            # key1: 11 bytes
kubectl get secret my-secret -o yaml         # data.key1: c3VwZXJzZWNyZXQ=
echo 'c3VwZXJzZWNyZXQ=' | base64 --decode    # supersecret

2. Read the raw etcd key

  • etcd key path: /registry/secrets/<namespace>/<name>.
  • Needs etcdctl with the v3 API plus etcd client TLS certs. On kubeadm, the API server's etcd client cert is accepted.
sudo apt-get update && sudo apt-get install -y etcd-client   # Debian/Ubuntu

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  get /registry/secrets/default/my-secret | hexdump -C
# Without encryption the plaintext "supersecret" is visible in the dump

3. Is encryption already enabled?

grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml
ps aux | grep kube-apiserver | grep -v grep     # look for --encryption-provider-config=...

If the flag is missing, encryption at rest is off.

4. Write the EncryptionConfiguration

head -c 32 /dev/urandom | base64      # aescbc needs a 32-byte key, base64-encoded
# enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets                # only listed resources are encrypted
    providers:                 # FIRST provider encrypts new writes; all are tried for reads
      - aescbc:
          keys:
            - name: key1
              secret: y0xTt+U6xgRdNxe4nDYYsij0GgRDoUYC+wAwOKeNfPs=
      - identity: {}           # no encryption; fallback so old plaintext data can still be read
sudo mkdir -p /etc/kubernetes/enc
sudo mv enc.yaml /etc/kubernetes/enc/enc.yaml
sudo chown root:root /etc/kubernetes/enc/enc.yaml
sudo chmod 600 /etc/kubernetes/enc/enc.yaml

5. Wire it into the kube-apiserver static pod

Edit /etc/kubernetes/manifests/kube-apiserver.yaml and make three changes:

spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml   # (1) flag
    # ...existing flags...
    volumeMounts:
    - name: enc                                                   # (2) mount into the container
      mountPath: /etc/kubernetes/enc
      readOnly: true
    # ...existing mounts...
  volumes:
  - name: enc                                                     # (3) pod-level hostPath volume
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate
  # ...existing volumes...
  • On save, the kubelet recreates the static pod and the API server restarts with the new config. On a single node this causes a brief API outage.
  • Verify: ps aux | grep kube-apiserver | grep -v grep shows --encryption-provider-config=/etc/kubernetes/enc/enc.yaml. Watch it come back with crictl ps and then kubectl -n kube-system get pods.

6. Verify that new Secrets are encrypted

kubectl create secret generic my-secret-2 --from-literal=key2=topsecret
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  get /registry/secrets/default/my-secret-2 | hexdump -C
# value starts with k8s:enc:aescbc:v1:key1: followed by ciphertext; "topsecret" not visible

7. Re-encrypt existing Secrets

Encryption applies only to new writes. Old objects stay as they were until they are written again.

kubectl get secret my-secret -o yaml | kubectl replace -f -          # rewrite one Secret
kubectl annotate secret my-secret reencrypt="$(date +%s)" --overwrite # any update also rewrites it
kubectl get secrets --all-namespaces -o json | kubectl replace -f -  # rewrite ALL Secrets (careful)

Then re-check the etcd entry of my-secret. It should now be encrypted.

TopicValue
etcd key/registry/secrets/<ns>/<name>
API server flag--encryption-provider-config=/etc/kubernetes/enc/enc.yaml
Config kindEncryptionConfiguration, apiVersion: apiserver.config.k8s.io/v1
ScopeOnly what is listed under resources (here secrets)
Key generationhead -c 32 /dev/urandom | base64
Stored prefixk8s:enc:aescbc:v1:<keyName>:

[!WARNING] Exam traps / remember:

  • If identity: {} is listed first, nothing gets encrypted. The first provider is the one used for writing.
  • Adding the flag is not enough. The file must be inside the container, so you also need the volumeMounts and volumes entries, with matching paths. Otherwise the API server won't start.
  • Existing Secrets are not encrypted until rewritten (kubectl get secrets -A -o json | kubectl replace -f -).
  • Never commit Secret YAML (base64 data) to public repos. Restrict etcd access and the etcd client certs. Rotate encryption keys regularly.

[!CAUTION] Correction: the original suggests docker ps to watch the API server restart. On current (containerd) nodes, use crictl ps / crictl ps -a | grep kube-apiserver.

[!NOTE] Beyond the course:

  • If the API server doesn't come back, check the manifest for YAML or path errors. Then read the logs: crictl ps -a | grep apiserver, crictl logs <id>, or /var/log/pods/kube-system_kube-apiserver-*/.
  • Other providers: identity, aescbc, aesgcm, secretbox, kms (v2). Upstream docs now recommend kms v2 or secretbox over aescbc. For the exam, use whatever the task specifies.
  • kubectl get secret always returns decrypted data. Encryption is only visible in etcd.
  • The upstream docs task uses --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key. Either cert pair works on kubeadm.

5.13 Multi Container Pods

Source: KodeKloud notes

Why

  • Breaking a monolith into microservices lets you develop, deploy and scale each part independently.
  • Some processes still need to run side by side: an app plus a logging agent, proxy or helper. You want them as separate images and pipelines, but they should scale, start and stop together, and share network and storage without Services or cross-pod networking.
  • The Kubernetes primitive for this is the multi-container Pod. Common patterns: sidecar (logging, proxy, metrics), ambassador, adapter.

Multi-container Pod: two containers share lifecycle, network and storage

SharedMeaningExample
LifecycleCreated and terminated together. The pod-level restartPolicy governs restartsReplacing a container means updating or recreating the pod
NetworkOne network namespace and one Pod IP. Containers talk over localhostweb-app on 8080 is reachable from the sidecar at localhost:8080
StoragePod-level volumes can be mounted into any containerShare logs or config via emptyDir or a PV

YAML

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    name: simple-webapp
spec:
  containers:            # it's an array: add more items for more containers
  - name: web-app
    image: web-app
    ports:
    - containerPort: 8080
  - name: main-app       # second container; reaches web-app at localhost:8080
    image: main-app

Best practices

  • Sidecar: log shippers, proxies, agents next to the app. Ambassador/adapter: a container that proxies or translates between the app and external services.
  • Set resources.requests and resources.limits per container to avoid noisy neighbours inside the pod.
  • Give each container its own livenessProbe / readinessProbe.

[!WARNING] Exam traps / remember:

  • Containers share the Pod IP, so two containers can't bind the same port. A port collision stops the second one from starting.
  • containers: is a list. Add another - name: item under the same key. Container names must be unique within the pod.

[!NOTE] Beyond the course: kubectl run can't create multi-container pods. Generate a base with kubectl run p --image=nginx --dry-run=client -o yaml > p.yaml and add list items. Address a specific container with kubectl logs <pod> -c <container> and kubectl exec -it <pod> -c <container> -- sh.

5.14 Multi Container Pods Design Patterns

Source: KodeKloud notes

Three patterns: co-located containers, regular init containers, sidecar containers

PatternWhere definedStartupTypical use
Co-located containerscontainersConcurrent, no guaranteed orderProcesses that run together and talk over localhost or shared volumes
Init containersinitContainersSequential. Each must exit 0 before the next starts. All finish before containers startDB readiness checks, schema migrations, provisioning files
Sidecarcontainers (as taught)Concurrent, long-running helperLogging, metrics/telemetry, proxy, cache, adapter

Co-located containers

  • Multiple entries in containers. They run for the pod's whole lifetime, and their start order is non-deterministic.
  • If one terminates, the pod restartPolicy decides whether it is restarted.
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    app: simple-webapp
spec:
  containers:
    - name: web-app
      image: web-app
      ports:
        - containerPort: 8080
    - name: main-app
      image: main-app

Init containers

  • Separate initContainers field. They run one at a time, in listed order, to completion.
  • A failed init container is restarted according to the pod restartPolicy. The app containers wait until all init containers succeed.
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    app: simple-webapp
spec:
  initContainers:                       # run in order: db-checker, then api-checker
    - name: db-checker
      image: busybox
      command: ["sh", "-c", "/scripts/wait-for-db-to-start.sh"]
    - name: api-checker
      image: busybox
      command: ["sh", "-c", "/scripts/wait-for-another-api.sh"]
  containers:
    - name: web-app
      image: web-app
      ports:
        - containerPort: 8080

Sidecar containers

  • A helper that runs next to the main app for the pod's lifetime. It shares the network namespace (reaches the app via localhost) and the volumes.
  • As taught, a sidecar is just another containers entry, so it has no startup ordering with the app. If ordering matters, coordinate with readiness probes, shared volumes/files, or init containers.
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    app: simple-webapp
spec:
  restartPolicy: Always          # pod-level field
  containers:
    - name: web-app
      image: web-app
      ports:
        - containerPort: 8080
    - name: log-shipper          # sidecar
      image: docker.elastic.co/beats/filebeat:7.10.0
      command: ["filebeat", "-e", "-c", "/etc/filebeat/filebeat.yml"]

Logging example

A Filebeat sidecar ships the app's logs to Elasticsearch, and Kibana displays them. It should start with the app (to catch init logs) and keep running until after the app stops (to catch crash and termination logs).

Logging pipeline: app pod with Filebeat sidecar to Elasticsearch to Kibana

Best practices

  • If startup order matters, use initContainers or explicit coordination (readiness probes, shared files, in-pod health checks).
  • Keep sidecars lightweight with a single responsibility. Use shared volumes to exchange logs or config.
  • In production, manage pods through Deployments or StatefulSets.

[!WARNING] Exam traps / remember:

  • Init containers: sequential, run to completion, must exit 0. Regular containers: concurrent, no order.
  • Init containers don't count toward READY (1/1 = regular containers only). While they run, STATUS shows Init:N/M.

[!CAUTION] Correction: the original says restartPolicy defaults to Always "for Pods created by controllers". The default is Always for every Pod. Deployments and ReplicaSets only allow Always.

[!NOTE] Beyond the course: native sidecar containers (alpha in v1.28, on by default since v1.29, GA in v1.33). Declare the sidecar under initContainers with restartPolicy: Always. It starts before the app containers and keeps running. Later init containers and the app start once it has started (if it defines a startupProbe, once that probe succeeds). It is stopped after the app containers, and it doesn't block Job completion. This gives the startup and shutdown guarantees the Filebeat example needs.

spec:
  initContainers:
  - name: log-shipper
    image: docker.elastic.co/beats/filebeat:7.10.0
    restartPolicy: Always        # container-level: makes it a native sidecar
  containers:
  - name: web-app
    image: web-app

5.15 Lab Solution Init Containers Optional

Source: KodeKloud notes

Theory: see §5.14.

#TaskCommand / answerGotcha
1Which pod has an init container (red 1/1, green 2/2, blue 1/1)kubectl describe pod blue → Init Containers: init-myserviceget pods doesn't list init containers. green 2/2 means two regular containers
2Image of that init containerbusybox (app container uses busybox:1.28)Read the Init Containers: block, not Containers:
3State of the init containerTerminated, Reason: Completed, Exit Code: 0It ran (sh -c sleep 5) and exited before the app started
4Pod purple statuskubectl get pod purple → 0/1 Init:0/2 and pod Status: PendingInit containers are still running
5Purple's init containers and stateskubectl describe pod purple → warm-up-1 (sleep 600, Running), warm-up-2 (sleep 1200, Waiting PodInitializing). App container Waiting PodInitializing. Condition Initialized False
6How long until purple's app startsSum of the init durations: 600 s + 1200 s = 30 minSequential, so add them up
7Add an init container to red: image busybox, sleep 20Edit the manifest and recreate the pod (below)Can't add initContainers to a running pod. Delete and recreate, or replace --force
8Pod orange brokenkubectl get pod orange → Init:CrashLoopBackOff
9Find the causekubectl describe pod orange → init init-myservice Terminated / Error / Exit Code: 127. kubectl logs orange fails (waiting to start: PodInitializing). kubectl logs orange -c init-myservice → sh: sleeeep: not foundExit 127 = command not found. Pass -c <init-name> to get the init container's logs
10Fix orangeChange the command to sleep 2;, recreate the pod → init Completed / 0, pod 1/1 Running
# Task 7: add to red's spec (sibling of containers)
initContainers:
- name: busybox-init
  image: busybox
  imagePullPolicy: IfNotPresent
  command: ["sh", "-c", "sleep 20"]
# Task 10: fixed init container for orange
initContainers:
- name: init-myservice
  image: busybox
  imagePullPolicy: Always
  command: ["sh", "-c", "sleep 2;"]     # was: sleeeep 2;
# Recreate workflow (red / orange)
kubectl get pod red -o yaml > red.yaml      # or: kubectl edit pod red  -> saved to /tmp/kubectl-edit-*.yaml
vi red.yaml                                  # add initContainers
kubectl replace --force -f red.yaml          # = delete + create
# or: kubectl delete pod red && kubectl apply -f red.yaml
kubectl describe pod red                      # init container Terminated / Completed / Exit Code 0

[!WARNING] Exam traps / remember:

  • Init:X/Y = X of Y init containers have finished. Init:CrashLoopBackOff = an init container keeps failing. PodInitializing = the app containers are waiting.
  • kubectl logs <pod> targets the app container, which hasn't started. Use -c <init-container>.
  • Many pod fields are immutable. Use replace --force (or delete + apply) carefully.

5.16 Introduction to Autoscaling 2025 Updates

Source: KodeKloud notes

Scaling basics

  • Vertical (scale up/down): change the CPU or memory of a single machine.
  • Horizontal (scale out/in): add or remove machines/instances to share the load.

Vertical scaling adds CPU/memory to one server; horizontal scaling adds servers

Mapping to Kubernetes (two levels: cluster infra = nodes, workloads = pods)

HorizontalVertical
Cluster (nodes), manualProvision nodes and kubeadm join (or cloud tooling)Resize/replace node VMs, drain workloads (disruptive)
Cluster (nodes), automatedCluster Autoscaler: adds/removes nodes based on pending (unschedulable) pods and low utilization. Uses cloud provider APIs(none covered)
Workload (pods), manualkubectl scale on a Deployment/ReplicaSet/StatefulSetChange requests/limits in the manifest or with kubectl edit (pods restart)
Workload (pods), automatedHPA: replicas from CPU/memory/custom/external metrics (§5.17)VPA: recommends or applies requests; may evict pods (§5.19)
kubeadm join <control-plane-endpoint> --token <token> --discovery-token-ca-cert-hash sha256:<hash>   # add a node
kubectl scale deployment my-deployment --replicas=5                                                 # manual horizontal
kubectl edit deployment my-deployment                                                               # manual vertical (resources)
AutoscalerScopeUse caseSource
Cluster AutoscalerCluster, horizontalAdd nodes for unschedulable pods; remove under-used nodesgithub.com/kubernetes/autoscaler (cluster-autoscaler)
HPAWorkload, horizontalScale replicas on metricsBuilt in (kubernetes.io HPA docs)
VPAWorkload, verticalRecommend/apply requests; may evict podsgithub.com/kubernetes/autoscaler (vertical-pod-autoscaler)

[!TIP] Exam focus: how HPA changes the replica count from metrics, and how the Cluster Autoscaler adds and removes nodes (cloud integration). Know VPA conceptually: it changes pod requests and may evict pods. Deep VPA configuration is usually out of scope.

5.17 Horizontal Pod Autoscaler HPA 2025 Updates

Source: KodeKloud notes

Target workload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: nginx
        resources:
          requests:
            cpu: "250m"     # HPA % utilization is computed against THIS (and scheduler uses it)
          limits:
            cpu: "500m"     # caps usage; does not affect the HPA calculation

Manual way (what HPA automates)

kubectl top pod my-app-pod                       # needs metrics-server
kubectl scale deployment my-app --replicas=3

Example: usage 450m against a 250m request → 450/250 × 100 = 180 %. Against a 50 % target, you would scale out by hand. That approach is reactive and error-prone.

What HPA does

HPA observes metrics, adds pods, balances thresholds, tracks multiple metrics

  • Continuously watches one or more metrics (CPU, memory, custom, external) and adjusts the replicas of a Deployment, ReplicaSet or StatefulSet.
  • Scales out above the target and in below it, always within minReplicas and maxReplicas.
  • Built into the control plane (part of kube-controller-manager), so there is nothing to install for HPA itself. It needs a metrics source: metrics-server for resource metrics, or adapters for custom/external metrics.

Formula

desiredReplicas = ceil(currentReplicas × currentAverageUtilization / targetUtilization) Example: ceil(1 × 180 / 50) = ceil(3.6) = 4 replicas.

Imperative

kubectl autoscale deployment my-app --cpu-percent=50 --min=1 --max=10   # HPA named my-app
kubectl get hpa
# NAME     REFERENCE           TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
# my-app   Deployment/my-app   180%/50%   1         10        1          1m
kubectl delete hpa my-app

TARGETS = current/target.

Declarative (autoscaling/v2, supports multiple metric types)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:               # what to scale
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource            # Resource | Pods | Object | External | ContainerResource
      resource:
        name: cpu
        target:
          type: Utilization     # % of requests
          averageUtilization: 50

Metric sources

SourceUseProvided by
Resource metricsCPU/memory of pods (and nodes)metrics-server
Custom metricsIn-cluster, per-pod app metricsCustom metrics adapter (e.g. Prometheus Adapter)
External metricsSystems outside the cluster: queue length, SaaS/APM signalsExternal adapter (e.g. Datadog, Dynatrace)

HPA reads metrics-server and a custom metrics adapter internally, and an external adapter for Datadog/Dynatrace

Configuring each adapter is adapter-specific and out of scope.

[!WARNING] Exam traps / remember:

  • Utilization is computed against requests, not limits. A low request means a high % and aggressive scale-out.
  • No requests set, or no metrics-server → TARGETS shows <unknown> and nothing scales.
  • The HPA never goes outside min/max. kubectl autoscale names the HPA after the target (my-app).

[!NOTE] Beyond the course:

  • autoscaling/v1 supports CPU only (targetCPUUtilizationPercentage). Use autoscaling/v2 for memory, multiple metrics and behavior.
  • Defaults: about 15 s sync loop; about 10 % tolerance (no action within ±10 % of the target); scale-down stabilization window of 300 s (spec.behavior.scaleDown.stabilizationWindowSeconds), so scale-in is deliberately slow. Scale-up has no stabilization by default.
  • Debug with kubectl describe hpa <n> (events, conditions such as ScalingActive/AbleToScale). Watch with kubectl get hpa -w.
  • Memory target: resource: {name: memory, target: {type: AverageValue, averageValue: 500Mi}} (or type: Utilization).
  • Newer kubectl versions also accept kubectl autoscale … --cpu=50% (and --memory) and mark --cpu-percent deprecated. Check kubectl autoscale -h on the exam terminal.
  • A fast way to get the YAML: kubectl autoscale deployment my-app --cpu-percent=50 --min=1 --max=10 --dry-run=client -o yaml.

5.18 In place Resize of Pods 2025 Updates

Source: KodeKloud notes

Default behaviour: delete and recreate

Changing resources.requests/limits in a Deployment's pod template replaces the pods. That disrupts stateful and long-running workloads.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: nginx
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "1"
            memory: "512Mi"

In-place resize

  • Some resource fields of a running container can change without deleting the pod. The kubelet applies the change live, or restarts only that container if the policy requires it.
  • Per-container resizePolicy controls whether a change to a given resource needs a container restart:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: nginx
        resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired        # apply CPU change live (default)
        - resourceName: memory
          restartPolicy: RestartContainer   # memory change restarts this container
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Feature gate

# on kube-apiserver AND kubelet (plus other components as needed), via their flags/config
--feature-gates=InPlacePodVerticalScaling=true

How you set it depends on how the cluster was built (kubeadm, managed, kind, minikube…). On managed control planes, check whether the gate is exposed or an alternative setting exists.

Resize example from the course

kubectl patch deployment my-app --patch '{
  "spec": {"template": {"spec": {"containers": [
    {"name": "my-app", "resources": {"requests": {"cpu": "1"}}}
  ]}}}
}'

Course claim: if policy and node allow it, CPU grows without deleting the pod, while memory may restart the container depending on restartPolicy and current usage. (See the correction below.)

Limitations

LimitationDetail
ResourcesOnly cpu and memory
QoS classCan't change through a resize
Container typesInit and ephemeral containers can't be resized
RemovalRequests/limits that are set can't be removed
Memory decreaseA limit below current usage can't be applied. The resize stays "InProgress" (pending) until it becomes feasible
OSWindows pods not supported

[!CAUTION] Correction: patching a Deployment's pod template (as in the course example) always triggers a normal rollout: a new ReplicaSet and new pods. It is never an in-place resize. In-place resize works on the Pod object itself, and since v1.33 only through the resize subresource (kubectl v1.32+):

kubectl patch pod <pod> --subresource resize --patch \
  '{"spec":{"containers":[{"name":"my-app","resources":{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].resources}'   # what is actually applied

A plain kubectl edit pod that changes resources is still rejected.

[!NOTE] Beyond the course:

  • InPlacePodVerticalScaling was alpha in v1.27, beta and on by default since v1.33, and GA in v1.35. On recent clusters there is no gate to set.
  • Resize status: older versions used status.resize (Proposed/InProgress/Deferred/Infeasible). From v1.33 it shows as the pod conditions PodResizePending (reason Deferred/Infeasible) and PodResizeInProgress.
  • resizePolicy.restartPolicy defaults to NotRequired.

5.19 Vertical Pod Autoscaling VPA 2025 Updates

Source: KodeKloud notes

Manual vertical scaling

kubectl top pod my-app-pod           # needs metrics-server (Metrics API)
kubectl edit deployment my-app       # change requests/limits → Deployment replaces pods

What VPA does

  • Watches container usage continuously, computes recommended CPU/memory requests, and can apply them depending on the update mode.
  • Goal: cut over-provisioning by fitting requests to real usage.
  • Needs metrics-server.

Install (add-on, not core Kubernetes)

kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vertical-pod-autoscaler.yaml   # as shown in course
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxxx   Running
# vpa-recommender-xxxx            Running
# vpa-updater-xxxx                Running

Components

ComponentRole
RecommenderPulls historical and live usage from the Metrics API and computes recommended CPU/memory requests
UpdaterFinds pods with suboptimal requests and evicts them (when the mode allows), so their controller recreates them
Admission controllerMutates pod specs at creation and injects the recommended requests

VPA components: recommender writes recommendations, updater evicts, admission controller sets requests on new pods Diagram (new): the three VPA components and which update modes activate each one.

VPA object (CRD; there is no kubectl autoscale equivalent, so write YAML)

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"            # Off | Initial | Recreate | Auto
  resourcePolicy:
    containerPolicies:
      - containerName: "my-app"
        minAllowed:
          cpu: "250m"
        maxAllowed:
          cpu: "2"
        controlledResources: ["cpu"]   # only manage CPU
kubectl apply -f my-app-vpa.yaml
kubectl describe vpa my-app-vpa
#   Recommendation:
#     Container: my-app
#       Target:          cpu: 1500m
#       Uncapped Target: cpu: 1.5        (before min/maxAllowed are applied)
#       Lower Bound:     cpu: 250m
#       Upper Bound:     cpu: 2

Update modes

ModeBehaviour
OffRecommender only. Recommendations are recorded but never applied (visibility only)
InitialAdmission controller applies requests only at pod creation. No evictions
RecreateUpdater evicts suboptimal running pods. Controllers recreate them with the new requests. Admission also applies to new pods
AutoCurrently the same as Recreate. Intended to use in-place resize (§5.18) once that is safely supported

[!WARNING] Recreate/Auto cause evictions and restarts. Configure controllers and PodDisruptionBudgets to keep the app available, and test outside production first.

VPA vs HPA

VPAHPA
ActionAdjusts per-pod requests/limitsChanges the replica count
Best forStateful / resource-heavy apps (databases, JVMs)Stateless web frontends, APIs
Reaction to spikesSlower (pod restarts or resizes)Fast (adds replicas)
Availability impactMay restart podsAdds/removes pods without restarting existing ones
EfficiencyCuts over-provisioning per podAvoids idle capacity by scaling out/in
StrategyRight-size each podAbsorb load spikes
  • Combining them works well (VPA right-sizes requests, HPA scales count), but tune update modes and PDBs to avoid conflicts.

[!WARNING] Exam traps / remember:

  • VPA = CRD + 3 components in kube-system. HPA is built in. Both depend on metrics-server.
  • Off = recommend only. Initial = only new pods. Recreate/Auto = evicts.
  • VPA changes requests. It doesn't add replicas.

[!CAUTION] Correction: the releases/latest/download/vertical-pod-autoscaler.yaml URL in the course is not the documented install method and may not resolve. Upstream installs with:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler && ./hack/vpa-up.sh

In a lab, use whatever manifest or path the task provides.

[!NOTE] Beyond the course:

  • Don't let HPA and VPA act on the same CPU/memory metric for one workload, because they fight each other. If you combine them, drive HPA from custom/external metrics, or run VPA in Off mode.
  • Newer VPA releases add the InPlaceOrRecreate update mode, which tries an in-place resize (§5.18) before falling back to eviction.
  • By default VPA keeps the limit-to-request ratio when it changes requests (controlledValues: RequestsAndLimits). kubectl get vpa shows the current target in short form.

← 04 Logging & Monitoring · Index · 06 Cluster Maintenance →


Reference

Based on the KodeKloud Certified Kubernetes Administrator (CKA) course notes.

On this page

5.1 Application Lifecycle Management Section Introduction5.2 Rolling Updates and RollbacksRollouts and revisionsUpdate strategiesTriggering an updateTelling the strategy apart with kubectl describe deploymentUnder the hoodRollbackCommand reference5.3 Lab Solution Rolling update5.4 Commands and Arguments in DockerContainer = its main processOverriding at runtime / baking into an imageENTRYPOINT vs CMDBest practice5.5 Commands and Arguments in KubernetesMapping5.6 Lab Solution Commands and Arguments Optional5.7 Configuring Environment Variables in Applications5.8 Configuring ConfigMaps in ApplicationsWhyCreateInspectInject into PodsUpdates and best practices5.9 Lab Solution Env Variables OptionalEdit-then-replace technique (immutable Pod fields)Reference snippets (Kubernetes docs dapi-test-pod)5.10 SecretsWhy SecretsCreate: imperativeCreate: declarative (values under data must be base64)Inspect and decodeInject into podsBest practices5.11 Lab Solution Secrets OptionalDrillSecurity notes (Kubernetes docs, shown in lab)5.12 Demo Encrypting Secret Data at Rest1. Show the default (base64 only)2. Read the raw etcd key3. Is encryption already enabled?4. Write the EncryptionConfiguration5. Wire it into the kube-apiserver static pod6. Verify that new Secrets are encrypted7. Re-encrypt existing Secrets5.13 Multi Container PodsWhyYAMLBest practices5.14 Multi Container Pods Design PatternsCo-located containersInit containersSidecar containersLogging exampleBest practices5.15 Lab Solution Init Containers Optional5.16 Introduction to Autoscaling 2025 UpdatesScaling basicsMapping to Kubernetes (two levels: cluster infra = nodes, workloads = pods)5.17 Horizontal Pod Autoscaler HPA 2025 UpdatesTarget workloadManual way (what HPA automates)What HPA doesFormulaImperativeDeclarative (autoscaling/v2, supports multiple metric types)Metric sources5.18 In place Resize of Pods 2025 UpdatesDefault behaviour: delete and recreateIn-place resizeFeature gateResize example from the courseLimitations5.19 Vertical Pod Autoscaling VPA 2025 UpdatesManual vertical scalingWhat VPA doesInstall (add-on, not core Kubernetes)ComponentsVPA object (CRD; there is no kubectl autoscale equivalent, so write YAML)Update modesVPA vs HPA