Cluster Maintenance

← 05 Application Lifecycle Management · Index · 07 Security →

Contents

• 6.1 Cluster Maintenance Section Introduction
• 6.2 OS Upgrades
• 6.3 Lab Solution OS Upgrades optional
• 6.4 Kubernetes Releases
• 6.5 Cluster Upgrade Introduction
• 6.6 Demo Cluster Upgrade
• 6.7 Lab Solution Cluster Upgrade Process
• 6.8 Backup and Restore Methods

6.1 Cluster Maintenance Section IntroductionCopy Anchor Link

Source: KodeKloud notes

Section scopeCopy Anchor Link

• OS upgrades and how they affect node availability. Safe evacuation of a node for planned maintenance and bringing it back (§6.2, §6.3).
• Releases and versioning, control-plane vs kubelet version compatibility, choosing a target version (§6.4, §6.5).
• End-to-end hands-on cluster upgrade: control plane, then nodes, with minimal disruption (§6.6, §6.7).
• Backup and restore of cluster state and application data, plus a disaster-recovery exercise (§6.8).
• Before any OS or Kubernetes upgrade, plan the maintenance window and take backups. Rehearse the upgrade and the rollback in a staging cluster that mirrors production.

Node maintenance scenariosCopy Anchor Link

Safe drain → restore sequenceCopy Anchor Link

kubectl cordon <node-name>                     # 1. mark unschedulable (no new pods)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force   # 2. evict pods (current flags; see CAUTION)
# 3. perform OS/firmware maintenance or reboot the node
kubectl get nodes                              # 4. node back to Ready?
journalctl -u kubelet                          #    kubelet healthy? (run on the node)
kubectl uncordon <node-name>                   # 5. allow scheduling again
kubectl get pods -o wide                       # 6. pods scheduling? then app health checks + metrics

• Use --force only when you have to. Don't throw away local data of stateful workloads unless they are replicated.
• Drain respects PodDisruptionBudgets. If a PDB blocks eviction, either wait or adjust the budget for the maintenance window.
• PDBs and readiness probes together limit how many replicas can be evicted at once, which keeps the app up during maintenance.

[!CAUTION] Correction: --delete-local-data is removed in current kubectl. Use --delete-emptydir-data. Also, kubectl drain cordons the node itself, so the separate cordon step is redundant (but harmless).

Upgrade planning and version compatibilityCopy Anchor Link

• Releases come on a regular cadence. Every minor release brings features, deprecations and bug fixes, so read its release notes for breaking changes.
• Version skew policy: only limited skew is allowed between the control plane (API server, controller-manager, scheduler) and kubelet/node components. Therefore upgrade the control plane first, then the nodes (details in §6.5). Check the official skew policy for your exact versions.
• Pick a target version that is: stable and non-deprecated; supported by your cloud provider or distribution; validated in staging against your workloads and third-party controllers.
• Pre-upgrade checklist:
  • etcd snapshot + persistent application data backed up.
  • Monitoring, logging and alerting working.
  • Third-party controllers, CRDs and operators support the target version.
  • Rollback plan written and tested in staging.

Typical upgrade flow (high level)Copy Anchor Link

1. Back up etcd and application data.
2. Upgrade the control plane (API server, controller-manager, scheduler). On kubeadm clusters: kubeadm upgrade apply <version>. On managed control planes (GKE/EKS/AKS): use the provider's upgrade workflow.
3. Check control-plane health and API compatibility.
4. Upgrade kube-proxy and CoreDNS/core add-ons if needed.
5. Upgrade worker nodes one at a time: cordon + drain → upgrade kubelet/kube-proxy and the container runtime → uncordon.
6. Watch workloads and metrics after each node. Roll back if something breaks.

Hands-on end-to-end exercise:

• Pre-upgrade: full etcd snapshot. Validate app backups and test a restore in staging. All nodes Ready, no pods stuck in restart loops.
• Control plane: follow the documented steps for your cluster type (kubeadm, hosted, cloud). Check component health with kubectl get componentstatuses / kubectl get cs.
• Nodes, one at a time: kubectl cordon → kubectl drain → OS/kernel/runtime patches → restart kubelet → validate → kubectl uncordon.
• Post-upgrade: integration, smoke and performance tests. Watch metrics and logs for anything abnormal.

[!CAUTION] Correction: the course’s kubectl get componentstatuses (cs) has been deprecated since v1.19 and is unreliable. Check control-plane health with kubectl get pods -n kube-system and kubectl get --raw='/readyz?verbose'.

Backup and restore strategies (details in §6.8)Copy Anchor Link

Disaster-recovery exercise:

1. Take backups of etcd and app data, and verify them.
2. Simulate a disaster, e.g. corrupt the control plane or wipe nodes.
3. Restore etcd from the snapshot, or rebuild the control plane following the provider docs.
4. Recreate resources. Restore PVs/app data from Velero or DB backups.
5. Validate app state, connectivity and data integrity.

[!WARNING] Exam traps / remember:

• Order is always control plane → workers, never the reverse.
• Automate backups and regularly test restores. A backup you have never restored is unproven.
• Monitor during and after maintenance. Keep a current rollback plan and rehearse it periodically.

6.2 OS UpgradesCopy Anchor Link

Source: KodeKloud notes

What happens when a node goes downCopy Anchor Link

• Its pods become unreachable. User impact depends on how they were deployed:
  • Multi-replica Deployment/ReplicaSet (the "blue" app): the other replicas keep serving, so users are usually unaffected.
  • Single standalone pod (the "green" app): downtime.
• If the node comes back quickly (kubelet resumes), its pods just keep running.
• If it stays unreachable longer than the eviction timeout, the control plane treats its pods as dead and evicts them.
• When the node controller sees a node NotReady/unreachable, it adds a NoExecute taint, e.g. node.kubernetes.io/unreachable. Pods without a matching toleration are evicted after the grace period.
• Timeout: kube-controller-manager --pod-eviction-timeout, default 5m0s. Node back in time → pods stay. Otherwise → evicted, and controllers create replacements on other nodes.
• A pod can tolerate the taint and set its own limit for staying on an unhealthy node:

apiVersion: v1
kind: Pod
metadata:
  name: tolerant-app
spec:
  containers:
  - name: app
    image: nginx
  tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 3600        # stay up to 1h on an unreachable node before eviction

Node-offline sequence: node NotReady → node controller adds NoExecute taint → waits up to --pod-eviction-timeout (5 min) → evicts pods without a toleration, and controller-managed pods are recreated elsewhere. (Timeline diagram: §2.8.)

[!CAUTION] Correction: --pod-eviction-timeout was removed from kube-controller-manager (v1.27). Eviction is purely taint-based now. The DefaultTolerationSeconds admission plugin gives every pod tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable (NoExecute) with tolerationSeconds: 300, so the default is still ~5 min. Tune it per pod with tolerationSeconds as in the YAML above.

Planned maintenance: drain / cordon / uncordonCopy Anchor Link

kubectl drain node-1        # cordon + evict pods (pods recreated elsewhere by their controllers)
kubectl cordon node-1       # only mark unschedulable; existing pods keep running
kubectl uncordon node-1     # make schedulable again

kubectl drain:

• Cordons the node (unschedulable).
• Evicts pods gracefully, subject to PDBs. It won't evict if that would violate a PDB.
• Mirror pods (static pods) are ignored.
• DaemonSet pods: ignored only when you pass --ignore-daemonsets. Otherwise drain aborts (see CAUTION below).
• Unmanaged (standalone) pods: drain refuses to evict them unless you pass --force.

Behaviour to remember:

• Pods are not "moved". Drained pods are terminated, and their controllers notice the missing replicas and create new pods on other nodes.
• After uncordon, pods do not move back automatically. Only newly scheduled pods can land on the node again.

Diagram (new): drain sequence and what happens to each pod type, with the flag it needs.

Common kubectl drain flagsCopy Anchor Link

[!CAUTION] Correction: The course table calls --ignore-daemonsets "default behaviour". It isn't. Without the flag, drain aborts with cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore) (see §6.3). --delete-local-data was replaced by --delete-emptydir-data; the old flag is removed in current kubectl (unknown flag).

[!NOTE] Beyond the course: Check PDBs before draining with kubectl get pdb -A. While a PDB blocks it, drain keeps retrying the eviction. --timeout=<dur> bounds that wait, and --disable-eviction bypasses PDBs by deleting pods directly (use with care).

Practical recommendationsCopy Anchor Link

• For planned OS upgrades or reboots, drain first so replicas are recreated elsewhere before the node restarts.
• If the workload is highly available and the node will be back quickly, a short reboot without draining can work. It is risky, though: if the node stays down past the eviction timeout, its pods get evicted anyway.
• Before draining, look for PDBs and critical unmanaged pods. Drain can be blocked until the PDB's availability guarantee can be met.
• Static/mirror pods behave differently: the kubelet recreates them on the same node.
• Test drain/uncordon behaviour (including PDB interaction) in staging.

[!WARNING] Exam traps / remember:

• drain = cordon + evict. cordon = no new pods, existing ones stay. uncordon = schedulable again, but nothing moves back.
• Standalone pods are lost on --force drain or eviction. Only controller-managed pods come back elsewhere.
• Using --force / --delete-emptydir-data can cause data loss (emptyDir) or downtime. Prefer replication and graceful eviction.

6.3 Lab Solution OS Upgrades optionalCopy Anchor Link

Source: KodeKloud notes

Lab: controlplane + node01 (v1.20.0). Deployment blue (3 replicas) all running on node01. Theory: §6.2.

Command summary: drain <node> (cordon + evict controller-managed pods) · --ignore-daemonsets (skip node-local DS pods) · --force (delete unmanaged pods, data loss) · cordon · uncordon.

[!NOTE] Beyond the course: The lab's k describe node output shows kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock and the role control-plane,master. Both are pre-v1.24: dockershim is gone, so today's socket is unix:///var/run/containerd/containerd.sock, and the master role/taint was replaced by node-role.kubernetes.io/control-plane.

[!WARNING] Exam traps / remember:

• A failed drain leaves the node cordoned. If you abandon the drain, uncordon it.
• Answer "make node unschedulable without moving pods" with cordon. Answer "evacuate / prepare for maintenance" with drain --ignore-daemonsets.
• Run critical workloads under controllers so they survive drains.

[!NOTE] Beyond the course: kubeadm normally taints control-plane nodes node-role.kubernetes.io/control-plane:NoSchedule. The lab deliberately permits workloads there.

6.4 Kubernetes ReleasesCopy Anchor Link

Source: KodeKloud notes

Check the cluster versionCopy Anchor Link

kubectl get nodes    # VERSION column = kubelet version reported by each node

NAME           STATUS   ROLES          AGE   VERSION
controlplane   Ready    control-plane  14m   v1.33.0
node01         Ready    <none>         13m   v1.33.0

Semantic versioning: vMAJOR.MINOR.PATCHCopy Anchor Link

Release history and supportCopy Anchor Link

• v1.0: July 2015. Timeline from the course slide: v0.12 (Mar 2015) · v0.20 (Jun 2015) · v1.5 (Dec 2016) · v1.10 (Mar 2018) · v1.20 (Dec 2020) · v1.33 (Apr 2025).
• The project keeps release branches for the most recent three minor releases (e.g. 1.33, 1.32, 1.31). v1.19+ gets ~1 year of patch support.
• Release notes, changelogs and downloads: https://kubernetes.io/releases/ and https://github.com/kubernetes/kubernetes/releases

[!CAUTION] Correction: the course says a minor release roughly every three months; since 2021 (v1.21 onward) Kubernetes ships three minor releases a year (~every 15 weeks, roughly 4 months), not four. Patch support per minor is ~14 months (12 months + a 2-month upgrade window).

Feature maturity stagesCopy Anchor Link

[!NOTE] Beyond the course: Since v1.24, new beta APIs are disabled by default and need --runtime-config on the API server. Beta feature gates are still generally on.

Release artifacts and component versionsCopy Anchor Link

• The release tarball kubernetes.tar.gz contains binaries that all share the same version: kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, kube-proxy, kubectl.
• etcd and CoreDNS are separate projects with their own versions. Each Kubernetes release documents the etcd versions it supports/tested and the CoreDNS compatibility. Check these in the release notes before installing or upgrading.

[!WARNING] Exam traps / remember:

• kubectl get nodes VERSION is the kubelet version, not the control-plane version (check that with kubectl version or kubeadm upgrade plan).
• Minor = features, patch = bug/security fixes. Alpha features are off by default.
• etcd/CoreDNS versions do not follow Kubernetes versions.

• Patch support per minor: v1.19+ ≈ 1 year; v1.18 and older ≈ 9 months.

6.5 Cluster Upgrade IntroductionCopy Anchor Link

Source: KodeKloud notes

Focus: kubeadm-managed clusters; the control-plane components (kube-apiserver, kube-controller-manager, kube-scheduler) and the node components (kubelet, kube-proxy). External dependencies (etcd, CoreDNS) need their own consideration in production.

Version skew rulesCopy Anchor Link

Components don't all have to run the same version, but they must stay within the skew policy. That is what makes live, one-component-at-a-time upgrades possible.

Example with apiserver v1.10: controller-manager/scheduler v1.10 or v1.9. kubelet/kube-proxy v1.10, v1.9 or v1.8. Nothing at v1.11 (newer than the API server), except kubectl may be.

Diagram (new): skew window per component. The dashed X-3 cell is the current upstream policy.

[!NOTE] Beyond the course: Since v1.28, kubelet and kube-proxy may be up to three minor versions older than the API server (X-3). Neither may be newer than the API server. In HA control planes, all kube-apiserver instances must be within one minor version of each other.

When and how to upgradeCopy Anchor Link

The method depends on how the cluster was provisioned:

• Managed (cloud provider): the provider upgrades it, via UI/API (often a few clicks).
• kubeadm: kubeadm upgrade plan / kubeadm upgrade apply, then upgrade the node packages yourself.
• Hard way / custom: upgrade each component manually.

kubeadm upgrade plan               # health/preflight checks, current vs available versions
kubeadm upgrade apply <version>    # upgrade control-plane components

Phase 1: control planeCopy Anchor Link

• Upgrade kubeadm on the host first, then kubeadm upgrade apply updates API server, controller-manager, scheduler and the cluster manifests.
• Control-plane components restart briefly. During that window:
  • Workloads on the workers keep serving users.
  • Management is unavailable: no kubectl/API access, no new scheduling, and no controller actions (a failed pod is not replaced).
• Afterwards kubectl get nodes still shows the old versions, because it reports kubelet versions.

Phase 2: worker nodes, strategiesCopy Anchor Link

kubeadm upgrade plan output (1.11 cluster)Copy Anchor Link

[upgrade/versions] Cluster version: v1.11.8
[upgrade/versions] kubeadm version: v1.11.3
[upgrade/versions] Latest stable version: v1.13.4
[upgrade/versions] Latest version in the v1.11 series: v1.11.8

Components that must be upgraded manually after you have
upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT        AVAILABLE
Kubelet     3 x v1.11.3    v1.13.4

Upgrade to the latest stable version:
COMPONENT            CURRENT     AVAILABLE
API Server           v1.11.8     v1.13.4
Controller Manager   v1.11.8     v1.13.4
Scheduler            v1.11.8     v1.13.4
Kube Proxy           v1.11.8     v1.13.4
CoreDNS              1.1.3       1.1.3
Etcd                 3.2.18      N/A

You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.13.4
Note: Before you can perform this upgrade, you have to update kubeadm to v1.13.4.

• The plan shows: cluster version, kubeadm version, latest stable, latest in the current series, the components to upgrade, and the reminder that kubelets are upgraded manually.
• kubeadm does not install or upgrade kubelet. Upgrade the kubeadm binary to the target minor first (e.g. v1.12.x before kubeadm upgrade apply v1.12.x). Then upgrade the control plane, then kubelet/kube-proxy on each node.

Control plane 1.11 → 1.12 (course commands)Copy Anchor Link

apt-get update
apt-get install -y kubeadm=1.12.0-00           # 1. kubeadm to the target minor first
kubeadm upgrade apply v1.12.0                  # 2. control plane
# [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0".
# [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets ...
kubectl get nodes                              # still shows v1.11.3 (kubelet versions)

# kubeadm runs control-plane components as static pods, so masters run a kubelet too:
apt-get install -y kubelet=1.12.0-00
systemctl restart kubelet
kubectl get nodes                              # master v1.12.0, node-1/node-2 v1.11.3

Workers, one at a time (course commands)Copy Anchor Link

kubectl drain node-1                           # evict pods (respects PDBs), mark unschedulable
# on node-1:
apt-get update
apt-get install -y kubeadm=1.12.0-00
apt-get install -y kubelet=1.12.0-00
kubeadm upgrade node config --kubelet-version v1.12.0   # update node's kubelet config (old syntax, see CAUTION)
systemctl restart kubelet
kubectl uncordon node-1                        # schedulable again
# repeat: drain node-2 → upgrade → uncordon node-2; then node-3 …

• kube-proxy: when it runs as a DaemonSet from cluster-level manifests (kubeadm), kubeadm upgrade apply may update its image, so verify afterwards. In other setups, upgrade it manually.
• To reach v1.13, repeat the whole cycle: kubeadm on the control plane → kubeadm upgrade apply v1.13.x → kubelet on each node, one at a time (+ kube-proxy if needed).

[!CAUTION] Correction (outdated commands):

• kubeadm upgrade node config --kubelet-version v1.12.0 no longer exists. Run plain kubeadm upgrade node on each worker (and on additional control-plane nodes).
• kubeadm=1.12.0-00 is the version format of the frozen apt.kubernetes.io repo. With pkgs.k8s.io it is 1.29.3-1.1, and you must switch the repo for each minor (see §6.6).
• kubectl drain node-1 fails on any real node because of DaemonSet pods. Use kubectl drain node-1 --ignore-daemonsets.
• Run systemctl daemon-reload before systemctl restart kubelet after a package upgrade.

Checklist before each minor stepCopy Anchor Link

1. Read release notes for every intermediate minor version.
2. Back up etcd (cluster state).
3. Upgrade kubeadm on the machine that runs the upgrade.
4. kubeadm upgrade plan, and confirm the versions.
5. kubeadm upgrade apply <target> for the control plane.
6. Upgrade kubelet (and kube-proxy where needed) on nodes, using drain/uncordon for a rolling upgrade.
7. Validate the cluster and workloads.

[!WARNING] Exam traps / remember:

• Back up etcd and read the release notes before upgrading. Some versions need manual steps for third-party components (etcd, CNI). Skipping either can cause outages you cannot recover from.
• Upgrade kubeadm before upgrade apply. kubeadm can't apply a minor newer than itself.
• upgrade apply leaves the kubelet untouched. Every node, control plane included, needs its kubelet package upgraded and restarted.
• Draining marks the node unschedulable and uncordon reverses it. Where pods land afterwards is up to the scheduler.

6.6 Demo Cluster UpgradeCopy Anchor Link

Source: KodeKloud notes

Demo: kubeadm cluster controlplane + node01, v1.28.0 → v1.29.3, Ubuntu 20.04.6, following the official guide https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ (in the exam, open this docs page and follow it).

• That docs page covers 1.28.x → 1.29.x and 1.29.x → 1.29.y. Skipping MINOR versions is unsupported. Older paths have their own pages.
• Docs workflow: (1) upgrade the primary control-plane node → (2) additional control-plane nodes → (3) worker nodes.

Diagram (new): order of steps on the control-plane node, then on each worker.

Pre-checksCopy Anchor Link

kubectl get nodes          # both nodes Ready, v1.28.0
cat /etc/*-release         # OS family decides repo steps (Debian/Ubuntu → apt)

Package repository (pkgs.k8s.io)Copy Anchor Link

• Legacy Google-hosted repos apt.kubernetes.io / yum.kubernetes.io: deprecated Aug 31, 2023, frozen Sep 13, 2023, removed early 2024. Moving to the new repos was an opt-in, manual migration.
• The community repos at pkgs.k8s.io are required for versions released after Sep 13, 2023, and carry v1.24.0 onward.
• There is one repo per minor version. To upgrade, point the repo at the target minor (v1.29) on every node, control plane and workers.

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt-cache madison kubeadm     # list versions; pick the LATEST patch, e.g. 1.29.3-1.1

kubeadm | 1.29.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages
kubeadm | 1.29.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.29/deb  Packages
...

Note the exact package string (1.29.3-1.1). You use it for kubeadm, kubelet and kubectl.

[!NOTE] Beyond the course: if the node already uses pkgs.k8s.io, you only need to change the version in the repo file: sudo sed -i 's/v1.28/v1.29/' /etc/apt/sources.list.d/kubernetes.list && sudo apt update. The signing key is the same for every minor.

DrillCopy Anchor Link

[!CAUTION] Correction: The course's worker step says to run sudo kubeadm upgrade node "from a control-plane node". It must run on the worker itself, because it rewrites that node's local kubelet config. On additional control-plane nodes, kubeadm upgrade node replaces kubeadm upgrade apply (no plan needed there).

[!NOTE] Beyond the course: kubeadm upgrade apply also renews the control-plane certificates by default (--certificate-renewal=true), and it asks for [y/N] confirmation. Add -y/--yes to skip the prompt.

[!WARNING] Exam traps / remember:

• After kubeadm upgrade apply, nodes keep reporting the old version until their kubelet is upgraded and restarted.
• Change the repo to the target minor on every node before apt-get install. Otherwise apt-cache madison only lists the old minor.
• Use the latest patch of the target minor, with the exact package string. Upgrade one node at a time.
• Check CNI/add-on docs for version-specific steps, and test in staging first.

6.7 Lab Solution Cluster Upgrade ProcessCopy Anchor Link

Source: KodeKloud notes

Lab: controlplane + node01 on v1.19.0 → upgrade to v1.20.0. Deployment blue with 5 replicas spread over both nodes. Constraints: no new VMs and no user impact. Procedure: §6.6.

Verify between steps: kubectl get nodes, kubectl get pods -A (the course also uses kubectl get cs, which is deprecated, see §6.1). Pods reschedule according to Deployment settings and any PDBs.

[!CAUTION] Correction: 1.20.0-00 and http://apt.kubernetes.io kubernetes-xenial are from the frozen legacy repo. Current labs/exam use pkgs.k8s.io: switch the repo to the target minor first, then use package versions like 1.xx.y-1.1 (§6.6).

[!WARNING] Exam traps / remember:

• Confirm add-ons (CNI, CoreDNS, kube-proxy) support the target version. Back up etcd and test the path outside production first.
• Exam-style tasks often say "upgrade the control plane and node01 to exactly vX.Y.Z". Use that exact version in kubeadm upgrade apply and in the package versions.
• kubeadm refuses to apply a minor version newer than the kubeadm binary itself.

kubeadm upgrade plan (kubeadm v1.19.0) output: Latest version in the v1.19 series: v1.19.16 · kubelet 2 x v1.19.0 → v1.19.16 (manual) · You can now apply the upgrade by executing: kubeadm upgrade apply v1.19.16 · Note: Before you can perform this upgrade, you have to update kubeadm to v1.19.16. → so the plan only offers targets up to kubeadm's own minor.

• Docs 'Before you begin': read the release notes; the cluster must run a static-pod control plane and etcd pods (or external etcd); back up important components, e.g. app-level state in databases (kubeadm upgrade does not touch workloads).

6.8 Backup and Restore MethodsCopy Anchor Link

Source: KodeKloud notes

Backup candidatesCopy Anchor Link

Resource configurationCopy Anchor Link

• Declarative manifests kept in Git (GitHub/GitLab) are the best practice: reproducible, auditable, CI/CD/GitOps-friendly. If the cluster is lost, redeploy from the repo.

# pod-definition.yml
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
    type: front-end
spec:
  containers:
  - name: nginx-container
    image: nginx

kubectl apply -f pod-definition.yml

• Objects created imperatively have no manifest file, so export the live state from the API:

kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml   # quick capture of common workload objects

• get all is not everything. It skips PVCs, PVs, CRDs, ClusterRoleBindings and more. For full coverage, use a tool such as Velero (https://velero.io, resources + PVs), or a script that enumerates types from kubectl api-resources.

[!NOTE] Beyond the course: a scripted export of every listable namespaced type:

Copy Text

for r in $(kubectl api-resources --verbs=list --namespaced -o name); do
  kubectl get "$r" -A -o yaml > "backup-${r}.yaml"
done

etcd backup (snapshot)Copy Anchor Link

• etcd holds object definitions, node membership and most cluster metadata. Back it up by copying its data directory or by taking a snapshot. Basics: §2.5/§2.6. HA: §10.4.

ETCDCTL_API=3 etcdctl snapshot save snapshot.db     # basic form (use the v3 API)

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.pem \
  --cert=/etc/etcd/kubernetes.pem \
  --key=/etc/etcd/kubernetes-key.pem \
  snapshot save snapshot.db                          # TLS-secured etcd: endpoint + CA + client cert/key required

ls -l snapshot.db                                    # verify the file exists

• If more than one etcdctl binary is installed, make sure you use the v3 client (ETCDCTL_API=3) and that it is compatible with the server version.
• On TLS-enabled etcd, always pass --endpoints, --cacert, --cert and --key. Without them etcdctl can't connect.

etcd restore (systemd-managed etcd, course flow)Copy Anchor Link

1. Stop kube-apiserver so it doesn't talk to etcd during the restore.
2. etcdctl snapshot restore writes a new data directory from the snapshot.
3. Point etcd at that directory (--data-dir) with correct bootstrap flags, restart etcd, then start kube-apiserver.

sudo systemctl stop kube-apiserver
ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-from-backup
# I | mvcc: restore compact to 475629

# /etc/systemd/system/etcd.service (excerpt): point --data-dir at the restored directory
[Service]
ExecStart=/usr/local/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/etcd/kubernetes.pem \
  --key-file=/etc/etcd/kubernetes-key.pem \
  --peer-cert-file=/etc/etcd/kubernetes.pem \
  --peer-key-file=/etc/etcd/kubernetes-key.pem \
  --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-peer-urls https://${INTERNAL_IP}:2380 \
  --listen-client-urls https://${INTERNAL_IP}:2379,https://127.0.0.1:2379 \
  --advertise-client-urls https://${INTERNAL_IP}:2379 \
  --initial-cluster-token etcd-cluster-0 \
  --initial-cluster controller-0=https://${CONTROLLER0_IP}:2380,controller-1=https://${CONTROLLER1_IP}:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd-from-backup

sudo systemctl daemon-reload
sudo systemctl restart etcd
sudo systemctl start kube-apiserver

• A restore creates fresh cluster metadata (new member IDs). A restored data dir does not rejoin an existing cluster on its own. Set --initial-cluster, --name and --initial-cluster-state to match your intent: form a new cluster, or rejoin.
• On kubeadm clusters, etcd and kube-apiserver are static pods. Stop or replace them through their manifests rather than with systemctl.

[!NOTE] Beyond the course (kubeadm restore, the usual exam case):

Copy Text

# Cert paths: cat /etc/kubernetes/manifests/etcd.yaml (or kubectl -n kube-system describe pod etcd-controlplane)
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /opt/snapshot.db
etcdutl snapshot status /opt/snapshot.db -w table          # verify (or: etcdctl snapshot status)

# restore is local file work: no TLS flags needed; target dir must not already exist
etcdutl snapshot restore /opt/snapshot.db --data-dir /var/lib/etcd-from-backup
#   (deprecated but still works: ETCDCTL_API=3 etcdctl snapshot restore ...)
# then edit /etc/kubernetes/manifests/etcd.yaml → volumes: etcd-data hostPath.path: /var/lib/etcd-from-backup
# the kubelet recreates the etcd static pod; watch with: crictl ps | grep etcd ; kubectl get pods -n kube-system

etcdctl snapshot restore is deprecated in etcd 3.5 in favour of etcdutl snapshot restore. Both may be present in the exam environment.

Choosing a methodCopy Anchor Link

• Managed K8s (EKS/GKE/AKS): usually no etcd access. Use the provider's backup features, or manifests + PV snapshots.
• Secrets: an etcd snapshot includes them. Manifest-only backups can lose them, so keep them in an external secret store or back them up securely.
• Consistency: coordinate etcd snapshots with PV snapshots. App-consistent backups need the app quiesced, or storage-provider volume snapshots.

[!WARNING] Exam traps / remember:

• etcdctl snapshot save on a TLS cluster needs all four flags: --endpoints, --cacert, --cert, --key. Take the paths from the etcd manifest/unit.
• Restore into a new directory, then repoint etcd's data dir (--data-dir in the unit, or the hostPath in the static-pod manifest). Stop kube-apiserver first on systemd setups.
• kubectl get all misses many resource types. etcd snapshots don't contain PV data.

[!NOTE] Beyond the course: the export loop above covers namespaced types only. Also export cluster-scoped objects (RBAC, PVs, CRDs, StorageClasses, etc.):

Copy Text

for r in $(kubectl api-resources --verbs=list --namespaced=false -o name); do
  kubectl get "$r" -o yaml > "backup-cluster-${r}.yaml"
done

Resource exports do not back up volume data or make all exported objects immediately restorable.

← 05 Application Lifecycle Management · Index · 07 Security →

Reference

Based on the KodeKloud Certified Kubernetes Administrator (CKA) course notes.