Scheduling

← 02 Core Concepts · Index · 04 Logging & Monitoring →

Contents

• 3.1 Scheduling Section Introduction
• 3.2 Manual Scheduling
• 3.3 Lab Solution Manual Scheduling optional
• 3.4 Labels and Selectors
• 3.5 Lab Solution Labels and Selectors
• 3.6 Taints and Tolerations
• 3.7 Lab Solution Taints and Toleration Optional
• 3.8 Node Selectors
• 3.9 Node Affinity
• 3.10 Lab Solution Node Affinity Optional
• 3.11 Taints and Tolerations vs Node Affinity
• 3.12 Resource Requirements
• 3.13 Lab Solution Resource Limits
• 3.14 DaemonSets
• 3.15 Lab Solution DaemonSets optional
• 3.16 Static Pods
• 3.17 Lab Solution Static Pods Optional
• 3.18 Priority Classes
• 3.19 Multiple Schedulers
• 3.20 Lab Solution Multiple Scheduler
• 3.21 Configuring Scheduler Profiles
• 3.22 2025 Updates Admission Controllers
• 3.23 2025 Updates Lab Solution Admission Controllers
• 3.24 2025 Updates Validating and Mutating Admission Controllers
• 3.25 2025 Updates Lab Solution Validating and Mutating Admission Controllers

3.1 Scheduling Section IntroductionCopy Anchor Link

Source: KodeKloud notes

Section scope: how the scheduler picks a node for a Pod, and every lever you have to influence that choice.

Also covered later in the section: taints/tolerations (§3.6), node selectors (§3.8), node affinity (§3.9), taints vs node affinity combined (§3.11), resource requirements (§3.12), DaemonSets (§3.14), static Pods (§3.16), priority classes (§3.18), multiple schedulers + their events (§3.19–3.20), scheduler profiles (§3.21), admission controllers (§3.22).

[!CAUTION] Correction: the course intro table lists kubectl create daemonset as an example — that command does not exist. Create a DaemonSet from YAML (e.g. generate a Deployment with kubectl create deployment ... --dry-run=client -o yaml, change kind to DaemonSet, remove replicas/strategy). See §3.14.

3.2 Manual SchedulingCopy Anchor Link

Source: KodeKloud notes

How the scheduler binds a PodCopy Anchor Link

• Every Pod has spec.nodeName; normally not set in your manifest.
• Scheduler watches for Pods with empty nodeName → picks a node → creates a Binding object (v1 Binding) for the Pod → API server sets nodeName → kubelet on that node starts it.
• No scheduler running → Pods without nodeName stay Pending until manually bound or a scheduler is started.
• Use cases for manual scheduling: no scheduler in the cluster, don't want to rely on it, need exact placement.

Diagram (new): scheduler watches unscheduled Pods and POSTs a Binding; the two manual alternatives bypass or mimic that step.

Option 1 — nodeName at creation (simplest)Copy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  nodeName: node02          # bypasses the scheduler; kubelet on node02 runs it
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 8080

kubectl apply -f pod.yaml
kubectl get pods -o wide      # NODE column shows node02; Pending -> Running

• nodeName cannot be set or changed via kubectl edit/apply/patch on an existing Pod. Only post-creation path = a Binding (Option 2), which works once, on an unscheduled Pod. Once set, immutable.

Option 2 — POST a Binding for an existing Pod (mimics the scheduler)Copy Anchor Link

{
  "apiVersion": "v1",
  "kind": "Binding",
  "metadata": { "name": "nginx" },
  "target": { "apiVersion": "v1", "kind": "Node", "name": "node02" }
}

kubectl proxy &                                   # local unauthenticated proxy on 127.0.0.1:8001
curl -X POST -H "Content-Type: application/json" \
  --data-binary @binding.json \
  http://127.0.0.1:8001/api/v1/namespaces/default/pods/nginx/binding

• Endpoint: POST /api/v1/namespaces/<namespace>/pods/<pod>/binding; metadata.name = Pod name, target = Node.
• Requires RBAC permission to create the pods/binding subresource — normally only the scheduler/controllers have it.
• Without kubectl proxy, talk to the API server directly with a client cert/token.

• Best practice: prefer controllers (Deployment, DaemonSet, StatefulSet) + nodeSelector / node affinity / taints & tolerations; manual scheduling is for labs, debugging, very specific placement. Control who can create Bindings via RBAC.

[!WARNING] Exam traps / remember:

• nodeName bypasses the scheduler → scheduler filtering/scoring is skipped, so NoSchedule taints don't block the Pod (kubelet-side checks still apply, see note below).
• Can't edit nodeName on a running/pending Pod → delete and recreate (kubectl replace --force -f), see §3.3.
• Pod Pending + Node: <none> + no events → suspect scheduler is down.

[!NOTE] Beyond the course: nodeName skips only the scheduler. The kubelet still runs admission checks: nodeSelector / required node affinity, host ports, CPU/memory fit. Failing Pod → phase Failed, reason NodeAffinity / OutOfcpu / OutOfmemory (not Pending). NoSchedule taints are ignored, but an untolerated NoExecute taint still evicts the Pod.

3.3 Lab Solution Manual Scheduling optionalCopy Anchor Link

Source: KodeKloud notes

Environment: nodes controlplane, node01; files nginx.yaml (Pod nginx, image nginx, no nodeName), sample.yaml. Theory: §3.2.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  nodeName: node01        # added; controlplane for the second task
  containers:
  - image: nginx
    name: nginx

• Deletion can take several seconds: runtime sends SIGTERM; Pod waits up to terminationGracePeriodSeconds (default 30 s) depending on how the process handles the signal — expected.
• describe output shows default tolerations node.kubernetes.io/not-ready:NoExecute op=Exists for 300s and node.kubernetes.io/unreachable:NoExecute op=Exists for 300s (auto-added to every Pod; see §3.6 tolerationSeconds). QoS Class: BestEffort = no requests/limits.

[!NOTE] Beyond the course:

• On kubeadm clusters the scheduler is a static Pod: manifest /etc/kubernetes/manifests/kube-scheduler.yaml (see §3.16). In an exam "Pod stuck Pending, no events" task, check that file / crictl ps on the control plane rather than only using nodeName.
• Faster than editing: kubectl delete pod nginx --force --grace-period=0 then kubectl apply -f nginx.yaml.
• Lab output is v1.20: roles control-plane,master and a default-token-xxxxx Secret volume. Since v1.22 Pods mount a projected kube-api-access-xxxxx token volume instead of the Secret; since v1.24 the ROLES column shows only control-plane.

3.4 Labels and SelectorsCopy Anchor Link

Source: KodeKloud notes

ConceptsCopy Anchor Link

• Labels = arbitrary key/value string pairs in metadata.labels on any object; as many as needed. Identify app, function/role, environment, owner, tier…
• Selectors = queries that filter objects by labels (e.g. color=green → all green; color=green AND class=bird → green birds).
• Grouping mechanism for hundreds/thousands of objects by app, environment, purpose.

Label a Pod and select itCopy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp
  labels:
    app: app1
    function: front-end
spec:
  containers:
  - name: simple-webapp
    image: simple-webapp
    ports:
    - containerPort: 8080

kubectl get pods --selector app=app1          # long form
kubectl get pods -l app=app1                  # short form
kubectl get pods -l app=app1,function=front-end   # comma = AND

Labels in controllers (ReplicaSet example)Copy Anchor Link

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: simple-webapp
  labels:                 # labels of the ReplicaSet object
    app: app1
    function: front-end
  annotations:
    buildversion: "1.34"  # non-identifying metadata
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app1           # must match template labels
  template:
    metadata:
      labels:             # Pod labels
        app: app1
        function: front-end
    spec:
      containers:
      - name: simple-webapp
        image: simple-webapp

• Selector/template mismatch = common error (RS won't manage Pods / is rejected — see §3.5).
• Services: spec.selector matches Pod labels → traffic routed to every matching Pod.

Annotations vs labelsCopy Anchor Link

• Annotations: same key/value structure but non-identifying info (build version, CI/CD data, contact details, links to external tools). Never used for selection.

Label syntax rulesCopy Anchor Link

• Key = optional DNS-subdomain prefix (≤ 253 chars) + / + name (≤ 63 chars), e.g. example.com/key.
• Value ≤ 63 chars, may be empty; must start/end with alphanumeric; -, _, . allowed in between.
• Conventionally lowercase; keys and values are case-sensitive.

[!CAUTION] Correction: the course says the selector "must match the pod template labels exactly". Precisely: every selector label must be present (same key and value) in the template labels; the template may have extra labels — the course's own example has function: front-end only in the template. A selector the template doesn't satisfy is rejected.

[!NOTE] Beyond the course:

Copy Text

kubectl get pods --show-labels                 # show all labels
kubectl get pods -L env,tier                   # labels as columns
kubectl get pods -l 'env in (prod,dev)'        # set-based
kubectl get pods -l 'env!=prod,!tier'          # not-equal / key absent
kubectl label pod web tier=frontend            # add
kubectl label pod web tier=backend --overwrite # change existing value
kubectl label pod web tier-                    # remove label key
kubectl annotate pod web owner=team-a          # add annotation

apps/v1 controllers require spec.selector, and it is immutable after creation.

3.5 Lab Solution Labels and SelectorsCopy Anchor Link

Source: KodeKloud notes

Pods labelled with env, bu, tier. Theory: §3.4.

Error seen:

spec.template.metadata.labels: Invalid value: map[string]string{"tier":"nginx"}: `selector` does not match template `labels`

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: replicaset-1
spec:
  replicas: 2
  selector:
    matchLabels:
      tier: nginx        # was front-end -> mismatch with template
  template:
    metadata:
      labels:
        tier: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

[!WARNING] Exam traps / remember:

• On selector/template errors check spelling, hyphens (front-end vs frontend) and case on both sides.
• kubectl get pods --show-labels to see current Pod labels.
• Count questions: always --no-headers | wc -l; read whether the question asks for Pods or "objects" (get all).

• Use kubectl --help for command discovery; use <subcommand> --help for its flags.

3.6 Taints and TolerationsCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Scheduling mechanism, not a security control. Analogy: node = person, taint = bug spray, pod = bug, toleration = bug's tolerance for that spray.
• Taint → node: repels Pods that don't tolerate it. Toleration → Pod: lets it be scheduled on a node with a matching taint.
• By default Pods have no tolerations → no Pod lands on a tainted node.
• Taints restrict what a node accepts; they do not attract a Pod to that node. A tolerating Pod can still land on any untainted node. To direct Pods to nodes use nodeSelector / node affinity (podAffinity/podAntiAffinity for Pod-relative placement).

Taint commandsCopy Anchor Link

kubectl taint nodes <node-name> <key>=<value>:<effect>
kubectl taint nodes node1 app=blue:NoSchedule        # add
kubectl taint nodes node1 app=blue:NoSchedule-       # remove (trailing -)
kubectl describe node kube-master | grep Taints      # inspect one node
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints   # all nodes

Taint effectsCopy Anchor Link

Toleration in a Pod (spec.tolerations)Copy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: nginx-container
    image: nginx
  tolerations:
  - key: "app"
    operator: "Equal"     # Equal: key+value must match; Exists: key only (omit value)
    value: "blue"
    effect: "NoSchedule"  # must match the taint's effect

• Must match taint key, value (with Equal) and effect. Quote values (they're strings).

NoExecute and tolerationSecondsCopy Anchor Link

tolerations:
- key: "app"
  operator: "Equal"
  value: "blue"
  effect: "NoExecute"
  tolerationSeconds: 300   # stay 300 s after taint is added, then evicted

Control-plane taintCopy Anchor Link

• Control-plane nodes are tainted by default so workloads don't run there (keeps control-plane components isolated). Removing it exposes the control plane to user workloads — avoid unless intended.

kubectl describe node kube-master | grep Taints
# Taints:   node-role.kubernetes.io/master:NoSchedule   (course, old clusters)

[!CAUTION] Correction: node-role.kubernetes.io/master was replaced by node-role.kubernetes.io/control-plane. Current kubeadm clusters taint control-plane nodes with node-role.kubernetes.io/control-plane:NoSchedule.

[!WARNING] Exam traps / remember:

• Taint = node, toleration = Pod. A toleration does not guarantee the Pod goes to that node — combine with node affinity to dedicate nodes (§3.11).
• tolerationSeconds only makes sense with NoExecute.
• Removal needs the trailing -, and the key/effect must match the existing taint.
• operator defaults to Equal if omitted; with Exists don't set value.

[!NOTE] Beyond the course:

• kubectl taint nodes node1 app:NoSchedule- (key+effect) or kubectl taint nodes node1 app- (all effects for key) also remove taints.
• Toleration with operator: Exists and no key tolerates every taint (used by many DaemonSets).
• Every Pod automatically gets node.kubernetes.io/not-ready and node.kubernetes.io/unreachable NoExecute tolerations for 300 s (seen in §3.3 describe output) → Pods are evicted ~5 min after their node fails.

3.7 Lab Solution Taints and Toleration OptionalCopy Anchor Link

Source: KodeKloud notes

Two nodes: controlplane, node01. Theory: §3.6.

Scheduler event for mosquito:

Warning  FailedScheduling  default-scheduler  0/2 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master:NoSchedule}, that the pod didn't tolerate, 1 node(s) had taint {spray=mortein:NoSchedule}, that the pod didn't tolerate.

apiVersion: v1
kind: Pod
metadata:
  name: bee
  labels:
    run: bee
spec:
  containers:
  - name: bee
    image: nginx
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  tolerations:              # same level as containers
  - key: "spray"
    operator: "Equal"
    value: "mortein"
    effect: "NoSchedule"

• Toleration syntax: search "taints tolerations" on kubernetes.io docs (allowed in exam). Two forms:

tolerations:
- key: "key1"
  operator: "Equal"     # key + value + effect must match
  value: "value1"
  effect: "NoSchedule"

Or use the alternative fragment:

tolerations:
- key: "key1"
  operator: "Exists"    # key + effect only, no value
  effect: "NoSchedule"

[!CAUTION] Correction: on current clusters the control-plane taint is node-role.kubernetes.io/control-plane:NoSchedule → remove with kubectl taint node controlplane node-role.kubernetes.io/control-plane:NoSchedule-. Check describe output and copy the exact key.

[!WARNING] Exam traps / remember: untaint the control plane only when the task asks (single-node/test clusters) — not good practice in production.

3.8 Node SelectorsCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Problem: by default any Pod can land on any node with enough resources → a heavy batch/data-processing Pod may land on a small node.
• nodeSelector = simplest way to require nodes with given labels. Steps: label node → add nodeSelector to Pod → scheduler places Pod only on nodes having all listed labels.

kubectl label nodes node01 size=large       # kubectl label nodes <node> <key>=<value>
kubectl get nodes --show-labels             # verify
kubectl create -f pod-definition.yaml

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: data-processor
    image: data-processor
  nodeSelector:
    size: large          # exact, case-sensitive match on node label

LimitationsCopy Anchor Link

• Equality only — no OR (large or medium) and no NOT (not small).
• Everything is required; no "preferred".
• For these → node affinity (§3.9); for placement relative to other Pods → pod affinity / anti-affinity.

[!WARNING] Exam traps / remember: label the node before (or soon after) creating the Pod — with no matching node the Pod sits Pending. nodeSelector sits under Pod spec (for Deployments: spec.template.spec).

3.9 Node AffinityCopy Anchor Link

Source: KodeKloud notes

WhyCopy Anchor Link

• Goal: ensure Pods land on particular nodes (e.g. data-processing Pod on large node01). Node affinity = nodeSelector + operators, logical combinations, existence checks and soft preferences.

nodeSelector vs equivalent node affinityCopy Anchor Link

# nodeSelector form
spec:
  nodeSelector:
    size: Large

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
spec:
  containers:
  - name: data-processor
    image: data-processor
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:          # list; terms are ORed
        - matchExpressions:         # list; expressions in one term are ANDed
          - key: size
            operator: In
            values:
            - Large

Match patternsCopy Anchor Link

# Large OR Medium
- key: size
  operator: In
  values: [Large, Medium]
# anything except Small
- key: size
  operator: NotIn
  values: [Small]
# label key present, any value (no values field)
- key: size
  operator: Exists

Diagram (new): terms ORed, expressions within a term ANDed, values within In ORed; required vs preferred behaviour.

Affinity typesCopy Anchor Link

"DuringScheduling" = when the Pod is first placed; "DuringExecution" = Pod already running and node labels change.

Preferred (weighted)Copy Anchor Link

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 50              # 1-100; scheduler maximises summed weight of matched preferences
      preference:             # a single nodeSelectorTerm (no nodeSelectorTerms list here)
        matchExpressions:
        - key: size
          operator: In
          values:
          - Large

ChoosingCopy Anchor Link

• nodeSelector for simple exact match; nodeAffinity for multi-value, OR/NOT, existence, numeric, or soft preferences.
• required... when placement is mandatory; preferred... to keep workloads schedulable when preferred nodes are busy.
• IgnoredDuringExecution → relabelling a node never evicts running Pods. Test rules in staging.

[!WARNING] Exam traps / remember:

• Structure: affinity → nodeAffinity → required... → nodeSelectorTerms → - matchExpressions → - key/operator/values. Under preferred... it's - weight + preference → matchExpressions.
• values is always a list (even for one value); omit it for Exists/DoesNotExist; Gt/Lt take a single integer as string.
• Label values are case-sensitive (Large ≠ large).
• Copy the YAML block from kubernetes.io ("Assign Pods to Nodes") rather than typing it.

[!NOTE] Beyond the course: if a Pod has both nodeSelector and nodeAffinity, both must be satisfied.

3.10 Lab Solution Node Affinity OptionalCopy Anchor Link

Source: KodeKloud notes

Theory: §3.9.

node01 labels: beta.kubernetes.io/arch=amd64, beta.kubernetes.io/os=linux, kubernetes.io/arch=amd64, kubernetes.io/hostname=node01, kubernetes.io/os=linux.

blue (relevant part of the Deployment):

spec:
  replicas: 3
  selector:
    matchLabels:
      app: blue
  template:
    metadata:
      labels:
        app: blue
    spec:
      affinity:                      # sibling of containers
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: color
                operator: In
                values:
                - blue
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always

red.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: red
  labels:
    app: red
spec:
  replicas: 2
  selector:
    matchLabels:
      app: red
  template:
    metadata:
      labels:
        app: red
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/master   # course (v1.20); now node-role.kubernetes.io/control-plane
                operator: Exists                      # key present, value empty
      containers:
      - name: nginx
        image: nginx
        resources: {}

[!WARNING] Exam traps / remember:

• Indentation errors (e.g. did not find expected key) are the most common failure when editing --dry-run=client -o yaml output — affinity must sit directly under template.spec, aligned with containers.
• Affinity only attracts; it doesn't override taints. In this lab the control plane had Taints: <none>. On a default cluster the control-plane taint would leave red Pending unless you add a toleration for node-role.kubernetes.io/control-plane:NoSchedule (or the task says to remove the taint).

[!CAUTION] Correction: kubeadm has labelled control-plane nodes node-role.kubernetes.io/control-plane since v1.20 (ROLES control-plane,master in the lab = both labels present). The .../master label was dropped in v1.24 → use node-role.kubernetes.io/control-plane with Exists; it works on both old and new clusters. Confirm with kubectl get node controlplane --show-labels.

[!TIP] Quick checks: kubectl get nodes -l color=blue (which nodes match a label) and kubectl get nodes --show-labels. kubectl edit on the Deployment triggers a rollout, so the new Pods replace the old ones automatically.

3.11 Taints and Tolerations vs Node AffinityCopy Anchor Link

Source: KodeKloud notes

ScenarioCopy Anchor Link

• Shared cluster with other teams' nodes and pods. You own 3 nodes (blue, red, green) and 3 pods (blue, red, green).
• Goal: each colour pod runs only on its colour node, and no other team's pod lands on your nodes. The second goal also means your pods stay off the other teams' nodes.

Taints + tolerations alone (see §3.6)Copy Anchor Link

kubectl taint nodes node-blue color=blue:NoSchedule   # node repels pods without a matching toleration

apiVersion: v1
kind: Pod
metadata:
  name: pod-blue
spec:
  containers:
  - name: nginx
    image: nginx
  tolerations:                 # lets the pod onto node-blue; does NOT pull it there
  - key: "color"
    operator: "Equal"
    value: "blue"
    effect: "NoSchedule"

• Other pods are kept off node-blue. That part works.
• A toleration only permits placement on a tainted node. pod-blue can still be scheduled on any untainted node (another team's node), so you get no guaranteed placement.

Node affinity alone (see §3.9)Copy Anchor Link

kubectl label node node-blue  color=blue
kubectl label node node-red   color=red
kubectl label node node-green color=green

apiVersion: v1
kind: Pod
metadata:
  name: pod-blue
spec:
  containers:
  - name: nginx
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:   # hard rule
        nodeSelectorTerms:
        - matchExpressions:
          - key: color
            operator: In
            values:
            - blue

• pod-blue now lands only on nodes labelled color=blue.
• Nothing stops other teams' pods from landing on node-blue, because affinity does not repel anything.

Combine both = dedicated nodesCopy Anchor Link

1. Taint each colour node: kubectl taint nodes node-blue color=blue:NoSchedule
2. Label each colour node: kubectl label node node-blue color=blue
3. The pod spec carries both the toleration and the required node affinity:

apiVersion: v1
kind: Pod
metadata:
  name: pod-blue
spec:
  containers:
  - name: nginx
    image: nginx
  tolerations:                       # taint keeps everyone else off
  - key: "color"
    operator: "Equal"
    value: "blue"
    effect: "NoSchedule"
  affinity:                          # affinity keeps this pod on the blue node
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: color
            operator: In
            values:
            - blue

Repeat the same setup for red and green.

Diagram (new): the taint keeps foreign pods off the colour nodes, and required affinity keeps each colour pod on its own node.

[!WARNING] Exam traps / remember:

• Taint = node-side repel. Toleration = pod-side permission, not attraction. Affinity/nodeSelector = pod-side attraction.
• Exclusivity needs the hard rule requiredDuringSchedulingIgnoredDuringExecution. preferredDuringSchedulingIgnoredDuringExecution is only a soft preference and gives no exclusivity.
• Use a NoSchedule taint on the dedicated nodes, and give the toleration only to the pods meant to run there.

3.12 Resource RequirementsCopy Anchor Link

Source: KodeKloud notes

Scheduling on resourcesCopy Anchor Link

• Each node advertises CPU and memory capacity. Every pod placed on it consumes some of that capacity, e.g. a pod requesting 2 CPU and 1 memory unit.
• The scheduler compares the pod's requests with each node's free capacity. If no node fits, the pod stays Pending, and kubectl describe pod shows events like Insufficient cpu / Insufficient memory.

RequestsCopy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
    name: simple-webapp-color
spec:
  containers:
    - name: simple-webapp-color
      image: simple-webapp-color
      ports:
        - containerPort: 8080
      resources:            # per container, not per pod
        requests:
          memory: "4Gi"
          cpu: "2"

UnitsCopy Anchor Link

• CPU: whole numbers (1, 2) or millicores (100m = 0.1 CPU). The smallest unit is 1m. 1 CPU = 1 AWS vCPU = 1 GCP core = 1 Azure core = 1 hyperthread.
• Memory: decimal suffixes K, M, G (powers of 1000) and binary suffixes Ki, Mi, Gi (powers of 1024). Use binary units for precise sizing.

LimitsCopy Anchor Link

• With no limit set, a container can use as much as the node has free.

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
    name: simple-webapp-color
spec:
  containers:
    - name: simple-webapp-color
      image: simple-webapp-color
      ports:
        - containerPort: 8080
      resources:
        requests:
          memory: "1Gi"
          cpu: "2"
        limits:
          memory: "2Gi"
          cpu: "2"

Exceeding limits: CPU vs memoryCopy Anchor Link

Request/limit combinationsCopy Anchor Link

No requests or limits are set by default, unless a LimitRange applies. Without requests the scheduler reserves nothing, so one pod can monopolise a node.

• CPU: set requests and leave limits unset so bursty workloads can use idle CPU.
• Memory: set both requests and limits, and size conservatively, because overuse leads to OOM kills.

LimitRange: per-container defaults and bounds in a namespaceCopy Anchor Link

• Namespaced. Supplies a default limit (default), a default request (defaultRequest), and min/max bounds for containers.
• Applied at pod creation (admission). Only pods created after the LimitRange exists are affected. Existing pods are not changed.

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint
spec:
  limits:
    - type: Container
      default:          # default LIMIT if none is specified
        cpu: 500m
      defaultRequest:   # default REQUEST if none is specified
        cpu: 500m
      max:
        cpu: "1"
      min:
        cpu: 100m
---
apiVersion: v1
kind: LimitRange
metadata:
  name: memory-resource-constraint
spec:
  limits:
    - type: Container
      default:
        memory: 1Gi
      defaultRequest:
        memory: 1Gi
      max:
        memory: 1Gi
      min:
        memory: 500Mi

ResourceQuota: cap the namespace totalCopy Anchor Link

• Hard cap on the aggregate across all pods in a namespace: requests, limits, and also object counts. Useful in multi-tenant clusters.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: my-resource-quota
spec:
  hard:
    requests.cpu: "4"        # sum of all CPU requests in the ns <= 4
    requests.memory: "4Gi"
    limits.cpu: "10"         # sum of all CPU limits <= 10
    limits.memory: "10Gi"

[!WARNING] Exam traps / remember:

• Requests drive scheduling. Limits drive runtime enforcement. A pod stuck in Pending with Insufficient cpu/memory has a request problem.
• Limit without a request means request = limit.
• CPU over the limit gets throttled. Memory over the limit gets OOMKilled.
• 0.1 = 100m. 1G ≠ 1Gi.
• LimitRange works per container and has no effect on existing pods. ResourceQuota works on the namespace total.

[!NOTE] Beyond the course:

• QoS classes follow from these settings. Every container has requests == limits for both CPU and memory → Guaranteed. At least one request or limit set → Burstable. Nothing set → BestEffort, which is evicted first under node pressure. Check with kubectl get pod <p> -o jsonpath='{.status.qosClass}'.
• If a ResourceQuota covers requests.cpu/limits.memory etc., new pods that don't specify those values are rejected, unless a LimitRange supplies defaults.
• Imperative: kubectl set resources deployment <d> --requests=cpu=100m,memory=128Mi --limits=cpu=200m,memory=256Mi. kubectl create quota <name> --hard=requests.cpu=4,limits.memory=10Gi -n <ns>.
• kubectl describe node <n> → "Allocated resources" shows how much is already requested on that node.

3.13 Lab Solution Resource LimitsCopy Anchor Link

Source: KodeKloud notes

Why elephant fails: the container runs stress --vm 1 --vm-bytes 15M --vm-hang 1 (image polinux/stress, container mem-stress) and allocates 15M, but limits.memory: 10Mi (requests.memory: 5Mi). Fix: raise the limit above the allocation, e.g. 20Mi.

Editing a "locked" pod fieldCopy Anchor Link

kubectl edit pod elephant          # change limits.memory: 20Mi, then save → rejected:
# pods "elephant" was not valid:
# * spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,
#   `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations`
#   (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds`
# kubectl saves your edited copy to /tmp/kubectl-edit-XXXX.yaml. Use it, or export afresh:
kubectl get pod elephant -o yaml > /tmp/elephant-pod.yaml   # then edit resources.limits.memory: 20Mi
kubectl replace --force -f /tmp/elephant-pod.yaml           # deletes + recreates the pod
# pod "elephant" deleted
# pod/elephant replaced
kubectl get pods                        # elephant 1/1 Running
kubectl describe pod elephant           # Limits: memory 20Mi; Requests: memory 5Mi; Restart Count 0

# relevant fragment of the fixed spec
spec:
  containers:
  - name: mem-stress
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "15M", "--vm-hang", "1"]
    resources:
      limits:
        memory: 20Mi
      requests:
        memory: 5Mi

[!TIP] Pods are mostly immutable. In practice the editable fields are image, activeDeadlineSeconds and tolerations (add only). terminationGracePeriodSeconds is listed but only in an edge case (set to 1 if it was previously negative; newer error messages say so). For anything else, use edit → save to file → kubectl replace --force -f <file>. For a Deployment, just kubectl edit deploy, because the pod template is mutable and a rollout follows.

[!NOTE] Beyond the course: newer Kubernetes supports in-place pod resize (see §5.18) through the resize subresource, e.g. kubectl patch pod <p> --subresource resize --patch '...'. A plain kubectl edit pod of resources is still rejected, so in the exam, delete and recreate unless a task explicitly asks for an in-place resize.

3.14 DaemonSetsCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Runs one copy of a pod on every node, or on a selected subset. It aims for node coverage, not a replica count, unlike ReplicaSet/Deployment.
• When a node joins, the DS pod is added to it. When a node is removed, its DS pod is garbage-collected.
• Target a subset of nodes with nodeSelector, node affinity, or tolerations, which also let DS pods onto tainted nodes.
• Pick a DaemonSet for node-scoped agents. Pick a Deployment/ReplicaSet for stateless apps that scale by replica count.

ManifestCopy Anchor Link

Same as a ReplicaSet except kind, and there is no replicas:

apiVersion: apps/v1
kind: DaemonSet                  # ReplicaSet version: kind: ReplicaSet (+ replicas)
metadata:
  name: monitoring-daemon
spec:
  selector:
    matchLabels:
      app: monitoring-agent      # must match the template labels
  template:
    metadata:
      labels:
        app: monitoring-agent
    spec:
      containers:
      - name: monitoring-agent
        image: monitoring-agent:latest

CommandsCopy Anchor Link

kubectl apply -f daemonset.yaml       # use kubectl create -f or kubectl apply -f
kubectl get daemonsets                # short name: ds
# NAME               DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  AGE
# monitoring-daemon  3        3        3      3           3          41m
kubectl describe daemonset monitoring-daemon

[!CAUTION] Correction: the original lists kubectl create daemonset monitoring-daemon --image=.... That subcommand does not exist. To generate a DS imperatively, use kubectl create deployment <n> --image=<img> --dry-run=client -o yaml > ds.yaml, then change kind: DaemonSet and remove replicas, strategy and status (see §3.15). Or copy the example from the docs.

How DS pods get scheduledCopy Anchor Link

• Before v1.12: the DS controller set spec.nodeName on each pod itself, so pods were bound directly and bypassed the scheduler (as in manual scheduling, §3.2).
• From v1.12: the controller creates pods without nodeName. It adds node affinity for the target node, and the default kube-scheduler places them. Taints, node conditions and other constraints are therefore respected like for any workload.

[!WARNING] Exam traps / remember:

• spec.selector is mandatory in apps/v1 and must match spec.template.metadata.labels (selector ⊆ template labels; the template may carry extra labels).
• A DS has no replicas field. DESIRED equals the number of eligible nodes.
• To run on tainted nodes such as the control plane, the DS pod template needs a matching toleration.

[!NOTE] Beyond the course: the DS controller automatically adds tolerations such as node.kubernetes.io/not-ready, unreachable (NoExecute), disk-pressure, memory-pressure, pid-pressure and unschedulable (NoSchedule). That is why DS pods survive cordons and node problems. They are not given a toleration for the control-plane taint. For production CNI DaemonSets, use the vendor's manifests (e.g. the Calico docs).

3.15 Lab Solution DaemonSets optionalCopy Anchor Link

Source: KodeKloud notes

Useful describe ds fields: kube-proxy Image: k8s.gcr.io/kube-proxy:v1.20.0 (lab cluster v1.20; kubectl get nodes → controlplane Ready control-plane,master v1.20.0), Selector: k8s-app=kube-proxy, Node-Selector: kubernetes.io/os=linux, Number of Nodes Scheduled with Up-to-date Pods, ... with Available Pods, Number of Nodes Misscheduled: 0, Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed. The kube-proxy container runs /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=$(NODE_NAME), with NODE_NAME taken from spec.nodeName. Mounts: /lib/modules (ro), /run/xtables.lock (rw). The get ds -A output also has a NODE SELECTOR column (kube-proxy: kubernetes.io/os=linux, flannel: <none>).

Creating the DS (Deployment → DaemonSet trick)Copy Anchor Link

kubectl create deployment fluentd-elasticsearch -n kube-system \
  --image=quay.io/fluentd_elasticsearch/fluentd:v2.5.2 \
  --dry-run=client -o yaml > fluentd.yaml
# edit: kind: Deployment -> DaemonSet; delete replicas:, strategy: {}, status: {}
# add tolerations / volumes / volumeMounts as needed
kubectl apply -f fluentd.yaml          # daemonset.apps/fluentd-elasticsearch created
kubectl get daemonsets -n kube-system

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
spec:
  selector:
    matchLabels:
      name: fluentd-elasticsearch
  template:
    metadata:
      labels:
        name: fluentd-elasticsearch
    spec:
      tolerations:                         # lets it run on control-plane nodes; remove if not wanted
      - key: node-role.kubernetes.io/control-plane   # course: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

• Running agents on control-plane nodes can affect stability and security. Only add the toleration deliberately.
• This is a demo manifest. Production also needs RBAC, a ConfigMap for the fluentd config, volume permissions, security contexts, resource tuning and node selectors/affinity.

[!CAUTION] Correction: the course toleration key node-role.kubernetes.io/master is obsolete. Current kubeadm control-plane nodes are tainted node-role.kubernetes.io/control-plane:NoSchedule, so use that key. On containerd/CRI-O nodes, container logs are under /var/log/pods and /var/log/containers, which the /var/log mount already covers. /var/lib/docker/containers only matters on Docker-based nodes.

[!NOTE] Beyond the course: images from k8s.gcr.io now come from registry.k8s.io, so expect e.g. registry.k8s.io/kube-proxy:vX.Y.Z in current labs.

3.16 Static PodsCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Normally the kubelet asks the kube-apiserver which pods to run, i.e. the ones the scheduler assigned to its node. It can also run pods without any control plane. It periodically reads pod manifests from a local directory, creates the pods through the container runtime, and keeps them running.
• Only Pods. ReplicaSets, Deployments and Services need the control plane (API server, controllers, scheduler, etcd), so dropping their YAML into the directory does nothing.

Configuring the manifest pathCopy Anchor Link

# course example of a kubelet unit using the flag
ExecStart=/usr/local/bin/kubelet \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --network-plugin=cni \
  --register-node=true \
  --pod-manifest-path=/etc/kubernetes/manifests \
  --v=2

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
staticPodPath: /etc/kubernetes/manifests

How to find the path on an unknown node: systemctl cat kubelet or cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf → find --config=<file> → grep staticPodPath <file>. If the unit has --pod-manifest-path, that is the answer.

[!CAUTION] Correction: in the course unit file, --network-plugin=cni (removed with dockershim in v1.24) and --container-runtime=remote (removed in v1.27) no longer exist, and --pod-manifest-path is deprecated in favour of staticPodPath in the config file. Current kubeadm control-plane manifests are named kube-apiserver.yaml, kube-controller-manager.yaml, kube-scheduler.yaml and etcd.yaml. The course lists apiserver.yaml and controller-manager.yaml.

Inspecting static podsCopy Anchor Link

• With no API server, kubectl doesn't work. Use the runtime CLI. The course showed docker ps, whose containers are named k8s_<container>_<pod>-<node>_<ns>_<uid>_0, plus one pause (sandbox) container per pod named k8s_POD_<pod>-<node>_<ns>_<uid>_0, image k8s.gcr.io/pause:3.1 (e.g. k8s_POD_myapp-pod-host01_default_..._0).
• On containerd/CRI-O nodes, use crictl ps (or ctr -n k8s.io containers ls).
• In a cluster: the kubelet also creates a read-only mirror pod in the API server, so kubectl get pods shows it. The name has the node name appended, e.g. static-web-node01. You can't edit it through the API. Changes go into the manifest file on the node.

Diagram (new): static pod lifecycle. The node's manifest file is the source of truth. The API server only holds a read-only mirror.

Control plane as static pods (kubeadm)Copy Anchor Link

• kubeadm writes the control-plane component manifests into the kubelet's static pod directory. The kubelet then starts and manages them without any API server, which is how kubeadm bootstraps the control plane.
• Typical files: /etc/kubernetes/manifests/{kube-apiserver,kube-controller-manager,kube-scheduler,etcd}.yaml.

Static pods vs DaemonSetsCopy Anchor Link

[!WARNING] Exam traps / remember:

• Static pods, and pods with spec.nodeName preset (manual scheduling, §3.2), bypass the kube-scheduler. DaemonSet pods are placed by the default scheduler (since v1.12).
• kubectl delete pod <static-pod> only deletes the mirror, and the kubelet recreates it. To remove a static pod for good, delete the manifest file on the node that runs it.
• The static pod path may be non-default. Always check the kubelet config on that node.

[!NOTE] Beyond the course: a static pod spec cannot reference other API objects such as ServiceAccounts, ConfigMaps or Secrets. The pod's ownerReferences shows kind: Node.

3.17 Lab Solution Static Pods OptionalCopy Anchor Link

Source: KodeKloud notes

# Q1: kubectl get pod kube-apiserver-controlplane -n kube-system -o yaml (static pod)
ownerReferences:
- apiVersion: v1
  controller: true
  kind: Node
  name: controlplane
# vs coredns pod: apiVersion: apps/v1, kind: ReplicaSet, name: coredns-74ff55c5b

# Q7: kubectl options BEFORE --, container command AFTER --
kubectl run static-busybox --image=busybox --restart=Never \
  --dry-run=client -o yaml --command -- sleep 1000 > static-busybox.yaml
cp static-busybox.yaml /etc/kubernetes/manifests/
kubectl get pods            # static-busybox-controlplane  1/1 Running  (node name suffixed)

apiVersion: v1
kind: Pod
metadata:
  name: static-busybox
  labels:
    run: static-busybox
spec:
  containers:
  - name: static-busybox
    image: busybox            # Q8: change to busybox:1.28.4 in the file on the node
    command: ["sleep", "1000"]
  restartPolicy: Never

# Q9: permanently remove static-greenbox
kubectl get pods -o wide                  # static-greenbox-node01 ... NODE node01
ssh node01                                # course used: ssh root@<node01-IP>
grep staticPodPath /var/lib/kubelet/config.yaml
# staticPodPath: /etc/just-to-mess-with-you      <- non-default path, deliberately
ls /etc/just-to-mess-with-you             # greenbox.yaml
rm /etc/just-to-mess-with-you/greenbox.yaml
exit
kubectl get pods --watch                  # static-greenbox-node01 Terminating → gone

[!WARNING] Exam traps / remember:

• The static pod lives on the node whose suffix it carries. Edit or delete the file on that node, not on controlplane.
• Non-default staticPodPath values appear in labs and the exam. Always read the kubelet config.
• A typo in an image or tag in a static manifest causes pull failures or restart loops. Check the final state after editing.
• Editing or deleting control-plane manifests in /etc/kubernetes/manifests restarts or removes that component, so be deliberate.

[!NOTE] Beyond the course: if the kubelet config location is unknown, run ps -ef | grep kubelet and look at --config=.... The same check works on any node.

3.18 Priority ClassesCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Assigns an integer priority to pods. The scheduler places higher-priority pods first. If one doesn't fit, it can preempt (evict) lower-priority pods to make room.
• PriorityClass is cluster-scoped (non-namespaced), so pods in any namespace can reference it.
• Range: signed 32-bit, -2,147,483,648 to 2,147,483,647. A larger number means higher priority. System classes use about 2,000,000,000, above the app range, which tops out at 1,000,000,000.
• Typical order: Kubernetes components > databases > critical apps > jobs.

kubectl get priorityclass       # short name: pc
# NAME                      VALUE        GLOBAL-DEFAULT   AGE     PREEMPTIONPOLICY
# system-cluster-critical   2000000000   false            7m33s   PreemptLowerPriority
# system-node-critical      2000001000   false            7m33s   PreemptLowerPriority

FieldsCopy Anchor Link

• With no priorityClassName and no global default, the pod's priority is 0.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000000
description: "Priority class for mission critical pods"
globalDefault: true
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 8080
  priorityClassName: high-priority     # in the pod spec, not under containers

Scheduling and preemptionCopy Anchor Link

• Pending pods are queued by priority, so higher priority is scheduled first. A priority-7 pod goes before a priority-5 pod.
• If a high-priority pod doesn't fit, the scheduler checks that pending pod's preemptionPolicy:
  • PreemptLowerPriority: lower-priority pods may be evicted to free resources.
  • Never: no eviction. The pod waits in the queue until resources free up, but it still sits ahead of lower-priority pods in the queue.
• Preemption happens only if evicting lower-priority pods actually makes the pod schedulable. The scheduler minimises the number of victims.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000000
description: "Priority class for mission critical pods"
preemptionPolicy: PreemptLowerPriority   # the default, written explicitly
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority-nonpreempt
value: 1000000000
description: "Non-preempting priority class"
preemptionPolicy: Never                  # never evicts running pods

Best practice: use high values sparingly and document them, because aggressive priorities cause eviction churn. Reserve very large values for system components. Watch evictions with kubectl get events, metrics and logs.

[!WARNING] Exam traps / remember:

• PriorityClass is not namespaced. priorityClassName goes at pod spec level.
• The default preemptionPolicy is PreemptLowerPriority. Never means the pod is still queued first but evicts nobody.
• Only one globalDefault: true is allowed. Without one, priority defaults to 0.

[!NOTE] Beyond the course:

• User-defined classes may use at most 1,000,000,000. Higher values are reserved for the built-in system-* classes.
• Imperative: kubectl create priorityclass high-priority --value=1000000 --description="..." --preemption-policy=Never (add --global-default to make it the default).
• The resolved number is stored in the pod: kubectl get pod <p> -o jsonpath='{.spec.priority}'. You can list them with kubectl get pods -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priorityClassName.
• A pod that references a non-existent class is rejected. Changing globalDefault later does not change existing pods.

3.19 Multiple SchedulersCopy Anchor Link

Source: KodeKloud notes

ConceptCopy Anchor Link

• Default scheduler = default-scheduler (the kube-scheduler process); honours taints/tolerations, node affinity etc.
• Need custom placement (own scoring, extra checks, different algorithm)? Run additional schedulers alongside the default — each with a unique schedulerName — or your own scheduler binary.
• A Pod picks its scheduler with spec.schedulerName; omitted → default-scheduler.
• Each scheduler instance gets its own config file (KubeSchedulerConfiguration) naming its schedulerName.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  schedulerName: my-scheduler      # must match a running scheduler's profile name
  containers:
  - name: nginx
    image: nginx

Scheduler config fileCopy Anchor Link

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: my-scheduler      # name Pods reference
leaderElection:
  leaderElect: true                # needed when running >1 replica of THIS scheduler (HA)
  resourceNamespace: kube-system
  resourceName: lock-object-my-scheduler   # lock name; unique per scheduler deployment

• Leader election: with several replicas of the same scheduler, only the elected leader schedules. Single replica → leaderElect: false is fine. Multiple distinct custom schedulers → each needs its own resourceName so locks don't collide.

Ways to run itCopy Anchor Link

Run as a standalone Pod (course example)Copy Anchor Link

apiVersion: v1
kind: Pod
metadata:
  name: my-scheduler
  namespace: kube-system
spec:
  containers:
  - name: kube-scheduler
    image: k8s.gcr.io/kube-scheduler-amd64:v1.22.0
    command:
    - kube-scheduler
    - --address=127.0.0.1                                # removed in newer versions, see CAUTION
    - --kubeconfig=/etc/kubernetes/scheduler.conf        # credentials to reach the API server
    - --config=/etc/kubernetes/my-scheduler-config.yaml  # KubeSchedulerConfiguration file
    volumeMounts:
    - name: config-volume
      mountPath: /etc/kubernetes
  volumes:
  - name: config-volume
    configMap:
      name: my-scheduler-config

• Create the referenced ConfigMap and kubeconfig before applying the Pod.

[!CAUTION] Correction: --address (and the insecure healthz port 10251 seen in the course's log output) were removed from kube-scheduler; it now serves only on the secure port 10259 (HTTPS). k8s.gcr.io is frozen → images are now on registry.k8s.io. Also, mounting a ConfigMap at /etc/kubernetes hides any other file in that directory (e.g. scheduler.conf) unless the kubeconfig is in the same ConfigMap — the lab version (§3.20) avoids this by mounting at /etc/kubernetes/my-scheduler and using the Pod's ServiceAccount instead of --kubeconfig.

Run as a Deployment (production pattern)Copy Anchor Link

Steps: build/package scheduler binary into an image → push to registry → ServiceAccount + ClusterRoleBinding → config via ConfigMap mounted into the Deployment and passed with --config → leader election for HA.

git clone https://github.com/kubernetes/kubernetes.git && cd kubernetes && make   # build from upstream
# Dockerfile: FROM busybox + copy the kube-scheduler binary
docker build -t gcr.io/my-gcp-project/my-kube-scheduler:1.0 .
gcloud auth configure-docker                                   # auth Docker to GCR
docker push gcr.io/my-gcp-project/my-kube-scheduler:1.0

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler        # built-in role with the default scheduler's permissions
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: false
    profiles:
    - schedulerName: my-scheduler
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-scheduler
  namespace: kube-system
  labels: {component: scheduler, tier: control-plane}
spec:
  replicas: 1
  selector:
    matchLabels: {component: scheduler, tier: control-plane}
  template:
    metadata:
      labels: {component: scheduler, tier: control-plane, version: second}
    spec:
      serviceAccountName: my-scheduler          # identity used to talk to the API server
      containers:
      - name: kube-second-scheduler
        image: gcr.io/my-gcp-project/my-kube-scheduler:1.0
        command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
        livenessProbe:
          httpGet: {path: /healthz, port: 10259, scheme: HTTPS}   # scheduler secure port
          initialDelaySeconds: 15
        readinessProbe:
          httpGet: {path: /healthz, port: 10259, scheme: HTTPS}
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
        - name: config-volume
          mountPath: /etc/kubernetes/my-scheduler
      volumes:
      - name: config-volume
        configMap:
          name: my-scheduler-config

Apply, use and verifyCopy Anchor Link

kubectl apply -f my-scheduler-resources.yaml
kubectl get pods -n kube-system                    # custom scheduler Pod(s) Running
kubectl apply -f nginx-with-custom-scheduler.yaml  # Pod with schedulerName: my-scheduler
kubectl describe pod nginx                         # stuck Pending? read Events
kubectl get events -o wide                         # SOURCE column of the "Scheduled" event = which scheduler
kubectl logs my-scheduler -n kube-system           # version, cache sync, leader lease, scheduling errors

REASON     SOURCE           MESSAGE
Scheduled  my-scheduler     Successfully assigned default/nginx to node01
Pulling    kubelet, node01  pulling image "nginx"
...
# scheduler logs: "attempting to acquire leader lease kube-system/my-scheduler..." → "successfully acquired lease"

TroubleshootingCopy Anchor Link

• Pod never leaves Pending → kubectl describe pod <pod> / events.
• Scheduler Pod healthy (liveness/readiness)? Correct --config (and --kubeconfig if used)?
• RBAC: ServiceAccount must have system:kube-scheduler (or equivalent).
• Several distinct scheduler deployments with leader election → unique resourceName each.

[!WARNING] Exam traps / remember:

• Field is spec.schedulerName (Pod spec level, not container level). Name must match the schedulerName in the scheduler's config profile exactly.
• Verify who scheduled a Pod: kubectl get events -o wide → SOURCE of the Scheduled event.
• Custom scheduler lives in kube-system; its config arrives via ConfigMap + --config.

[!NOTE] Beyond the course:

• A Pod whose schedulerName matches no running scheduler stays Pending with no scheduling events at all — a quick tell that the name is wrong or the scheduler is down.
• schedulerName can't be changed on an existing Pod → delete and recreate (kubectl replace --force -f pod.yaml). For Deployments, set it under spec.template.spec.
• Quick check: kubectl get pod nginx -o jsonpath='{.spec.schedulerName}'.
• Newer kubectl get events -o wide columns differ from the course sample (LAST SEEN, TYPE, REASON, OBJECT, SUBOBJECT, SOURCE, MESSAGE…); SOURCE is still there.
• system:kube-scheduler only allows lease get/update for the lease named kube-scheduler; a custom scheduler with leaderElect: true and its own resourceName needs that name added to the role (upstream docs) or extra RBAC — another reason the lab uses leaderElect: false.

[!NOTE] Beyond the course: the upstream scheduler manifest also binds system:volume-scheduler, and in kube-system binds extension-apiserver-authentication-reader. These cover volume scheduling and reading delegated-authentication configuration; include them for the full deployment.

3.20 Lab Solution Multiple SchedulerCopy Anchor Link

Source: KodeKloud notes

Theory: see §3.19. Flow: find default scheduler image → ConfigMap with config → second scheduler Pod → test Pod with schedulerName.

DrillCopy Anchor Link

Provided config file (key = filename my-scheduler-config.yaml inside the ConfigMap):

apiVersion: kubescheduler.config.k8s.io/v1beta2   # see CAUTION: use v1 on current clusters
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: my-scheduler
leaderElection:
  leaderElect: false        # single replica → no leader election

Final /root/my-scheduler.yaml:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: my-scheduler
  name: my-scheduler
  namespace: kube-system
spec:
  serviceAccountName: my-scheduler
  containers:
  - command:
    - /usr/local/bin/kube-scheduler
    - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml   # = mountPath + ConfigMap key
    image: k8s.gcr.io/kube-scheduler:v1.23.0                          # same as kube-scheduler-controlplane
    name: kube-second-scheduler
    livenessProbe:
      httpGet: {path: /healthz, port: 10259, scheme: HTTPS}
      initialDelaySeconds: 15
    readinessProbe:
      httpGet: {path: /healthz, port: 10259, scheme: HTTPS}
    resources:
      requests:
        cpu: "0.1"
    securityContext:
      privileged: false
    volumeMounts:
    - name: config-volume
      mountPath: /etc/kubernetes/my-scheduler
  hostNetwork: false
  hostPID: false
  volumes:
  - name: config-volume
    configMap:
      name: my-scheduler-config

Test Pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-customsched
spec:
  schedulerName: my-scheduler
  containers:
  - name: nginx
    image: nginx:1.21
    ports:
    - containerPort: 80

TroubleshootingCopy Anchor Link

• Scheduler Pod won't start → kubectl describe pod my-scheduler -n kube-system, kubectl logs my-scheduler -n kube-system.
• Test Pod Pending → events + scheduler logs: is it receiving the Pod / blocked by RBAC? Can it reach the API?
• Image version mismatch with the control plane → behaviour/API mismatches.

[!CAUTION] Correction: kubescheduler.config.k8s.io/v1beta2 was removed in Kubernetes v1.28 (v1beta3 too, in v1.29); on current clusters use kubescheduler.config.k8s.io/v1, or the second scheduler crash-loops on startup — check its logs. The lab's "ClusterRole allowing get/list/watch pods" is not enough on its own: a scheduler must also create pods/binding and events and read nodes etc. — bind system:kube-scheduler plus system:volume-scheduler (full RBAC: §3.19). The sample kubectl get sa output showing SECRETS 1 predates v1.24 (token Secrets no longer auto-created → 0).

[!TIP] Image without describe: kubectl get pod kube-scheduler-controlplane -n kube-system -o jsonpath='{.spec.containers[0].image}', or grep image: /etc/kubernetes/manifests/kube-scheduler.yaml. Get the ConfigMap key right: the --config path must be <mountPath>/<key>, and --from-file uses the file's basename as the key.

3.21 Configuring Scheduler ProfilesCopy Anchor Link

Source: KodeKloud notes

Scheduling pipeline (recap)Copy Anchor Link

1. Scheduling queue: Pods wait, sorted by priority (higher first; PriorityClass, §3.18).
2. Filter: drop Nodes that can't run the Pod (e.g. Pod requests 10 CPU → only Nodes with ≥10 CPU free remain).
3. Score: rank the remaining Nodes by score (e.g. more free resources left after placement = higher score).
4. Bind: Pod bound to the highest-scoring Node.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "High priority class for important pods."
---
apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
  priorityClassName: high-priority     # moves the Pod up the scheduling queue
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    resources:
      requests:
        memory: "1Gi"
        cpu: "10"                      # only nodes with 10 free CPUs pass filtering

Plugins per phaseCopy Anchor Link

• One plugin can serve several phases (e.g. NodeResourcesFit in Filter and Score; TaintToleration, NodeAffinity likewise).

Extension pointsCopy Anchor Link

Plugins hook into extension points; each plugin can implement any subset and be configured per point:

• queueSort
• preFilter → filter → postFilter
• preScore → score → reserve
• permit → preBind → bind → postBind

Plugins seen in action:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  nodeName: controlplane     # scheduler skips a pre-bound Pod; kubelet on controlplane admits it
  containers:
  - name: nginx
    image: nginx

kubectl cordon controlplane          # NodeUnschedulable filter now excludes this node
kubectl describe node controlplane   # Taints: node.kubernetes.io/unschedulable:NoSchedule / Unschedulable: true

Multiple binaries vs multiple profilesCopy Anchor Link

• Old way: separate scheduler processes (default-scheduler, my-scheduler, my-scheduler-2) → more to operate; independent schedulers can race (place Pods on the same node without knowing about each other).
• Since v1.18: one kube-scheduler process can run multiple profiles; each profile acts as its own scheduler with its own schedulerName.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: my-scheduler-2
  plugins:
    score:
      enabled:
      - name: MyCustomPluginA
      - name: MyCustomPluginB
      disabled:
      - name: TaintToleration
- schedulerName: my-scheduler-3
  plugins:
    preScore:
      disabled:
      - name: '*'              # wildcard = disable every plugin at this extension point
    score:
      disabled:
      - name: '*'
- schedulerName: my-scheduler-4   # own customisations (or none)
- schedulerName: default-scheduler

• Under plugins:, every extension point takes enabled: / disabled: lists of exact plugin names or '*'.
• Profile selection: Pod's spec.schedulerName = profile's schedulerName; unset → default-scheduler profile.
• References: KEP-1451 (multi-scheduling profiles), Scheduling Framework docs.

[!WARNING] Exam traps / remember:

• Filter = feasibility (removes nodes); Score = preference (ranks, never removes). Queue ordering = PrioritySort; binding = DefaultBinder.
• Multi-profile support since 1.18; profiles live under profiles: in one KubeSchedulerConfiguration, each with a schedulerName.
• Cordon = taint node.kubernetes.io/unschedulable:NoSchedule + Unschedulable: true; nodeName bypasses scheduling entirely (§3.2).

[!NOTE] Beyond the course:

• All profiles in one process must use the same queueSort plugin (single shared queue).
• Extra extension points exist (preEnqueue, multiPoint = enable/disable a plugin at all its points at once); score plugins can take a weight.
• Loading the config on a kubeadm cluster: put the file on the control plane, add --config=<file> to /etc/kubernetes/manifests/kube-scheduler.yaml and mount it via a hostPath volume; kubelet restarts the static Pod (§3.16).

[!CAUTION] Correction: the course illustrates the NodeName scheduler plugin with spec.nodeName. A Pod created with that field already set bypasses the scheduler entirely (§3.2); the plugin is not invoked for this example.

3.22 2025 Updates Admission ControllersCopy Anchor Link

Source: KodeKloud notes

Where admission fitsCopy Anchor Link

kubectl → HTTPS → kube-apiserver: Authentication (who: e.g. client cert from kubeconfig) → Authorization (allowed: e.g. RBAC) → Admission controllers (validate / mutate) → persist to etcd. See the diagram in §3.24.

• RBAC works at API-operation level (verb + resource, optionally resourceNames); it never looks inside or changes the object.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "get", "create", "update", "delete"]
# variant from the course: verbs: ["create"] + resourceNames: ["blue", "orange"]

[!CAUTION] Correction: the course shows resourceNames restricting create to Pods named blue/orange. RBAC cannot restrict create (or deletecollection) by name — the name isn't known at authorization time — so such a rule matches no create requests at all. resourceNames works for get, update, patch, delete. Name-based checks on create are an admission-control job.

What admission controllers can doCopy Anchor Link

• Validate (accept/reject), mutate (change object before it's stored), or cause other side effects (e.g. create a missing namespace).
• Use cases: allow only an internal registry (block Docker Hub); forbid :latest tag; deny containers running as root; require labels/metadata; inject sidecars or default env vars.

[!CAUTION] Correction: the course table calls it EventRateLimitAdmissionController; the plugin name to pass to --enable-admission-plugins is EventRateLimit (it also needs --admission-control-config-file).

NamespaceExists vs NamespaceAutoProvisionCopy Anchor Link

kubectl run nginx --image nginx --namespace blue
# NamespaceExists/NamespaceLifecycle: Error from server (NotFound): namespaces "blue" not found
# NamespaceAutoProvision enabled:      pod/nginx created   → kubectl get namespaces shows "blue Active"

[!CAUTION] Correction: NamespaceLifecycle replaces the namespace-existence check and protects default, kube-system, kube-public; it never creates namespaces. NamespaceAutoProvision is a separate opt-in plugin, unlike the course’s claim that both were replaced. See admission controllers.

View / change enabled pluginsCopy Anchor Link

kube-apiserver -h | grep enable-admission-plugins     # lists default-enabled plugins
# kubeadm (apiserver is a static Pod): run the binary inside its Pod
kubectl exec -it kube-apiserver-controlplane -n kube-system -- kube-apiserver -h | grep enable-admission-plugins

• Flags: --enable-admission-plugins=A,B,... (in addition to defaults) · --disable-admission-plugins=X,Y,....
• Where to edit: kubeadm → static Pod manifest /etc/kubernetes/manifests/kube-apiserver.yaml (kubelet recreates the Pod on save); "hard way" → systemd unit ExecStart.

# systemd unit (kube-apiserver.service), relevant flags
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=${INTERNAL_IP} \
  --allow-privileged=true \
  --apiserver-count=3 \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --enable-swagger-ui=true \
  --etcd-servers=https://127.0.0.1:2379 \
  --event-ttl=1h \
  --runtime-config=api/all \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --v=2 \
  --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision

# /etc/kubernetes/manifests/kube-apiserver.yaml (kubeadm), excerpt
apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    image: k8s.gcr.io/kube-apiserver-amd64:v1.11.3
    command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=172.17.0.107
    - --allow-privileged=true
    - --enable-bootstrap-token-auth=true
    - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision

Best practicesCopy Anchor Link

• Custom policy → validating admission/webhooks (e.g. allowed registries, securityContext); defaults/sidecars → mutating admission/webhooks.
• Prefer central cluster-wide policy (PodSecurity admission, OPA/Gatekeeper) over ad-hoc scripts.
• Test API-server flag changes on a non-production cluster first.

[!WARNING] Exam traps / remember:

• Order: AuthN → AuthZ → Admission → etcd. Admission controllers don't authenticate.
• --enable-admission-plugins adds to the defaults; to turn a default off you need --disable-admission-plugins.
• Editing the static Pod manifest restarts kube-apiserver — kubectl fails ("connection refused") until it's back.

[!NOTE] Beyond the course: while the API server restarts, watch it with watch crictl ps (or crictl ps -a | grep kube-apiserver) on the control-plane node; if it never comes back, a typo in the flags is likely → crictl logs <id> or /var/log/pods/kube-system_kube-apiserver-*.

• AuthN source = kubeconfig (~/.kube/config): clusters[].cluster.certificate-authority-data (base64 CA) + server: https://<apiserver-ip>:6443, plus the user's client cert/key. kube-apiserver checks these before AuthZ/admission.

[!CAUTION] Correction: the systemd/static-Pod blocks above are legacy course excerpts. --enable-swagger-ui no longer exists; current runtime-config syntax is --runtime-config=api/all=true; --apiserver-count is deprecated. Current images use registry.k8s.io/kube-apiserver:<version>. Preserve the cluster’s existing flags/image and change only the required admission-plugin settings.

3.23 2025 Updates Lab Solution Admission ControllersCopy Anchor Link

Source: KodeKloud notes

Theory: see §3.22. Lab assumes kube-apiserver runs as a static Pod (kubeadm).

DrillCopy Anchor Link

CommandsCopy Anchor Link

# Q2: default-enabled plugins
kubectl get pods -n kube-system
kubectl exec -it kube-apiserver-controlplane -n kube-system -- kube-apiserver -h | grep 'enable-admission-plugins'

--enable-admission-plugins strings  admission plugins that should be enabled in addition to default enabled ones
(NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds,
DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval,
CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook,
ValidatingAdmissionWebhook, ResourceQuota).

# Q4: first attempt fails
kubectl run nginx --image=nginx -n blue        # Error from server (NotFound): namespaces "blue" not found
vi /etc/kubernetes/manifests/kube-apiserver.yaml
#   - --enable-admission-plugins=NodeRestriction                         (before)
#   - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision  (after)
# save → kubelet recreates the static Pod; wait until kubectl responds again
kubectl run nginx --image=nginx -n blue        # pod/nginx created
kubectl get ns                                 # blue  Active  7s

# Q5: resulting manifest excerpt
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.49.219.9
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision
    - --disable-admission-plugins=DefaultStorageClass   # separate flag, not a "-" prefix in the enable list
    - --enable-bootstrap-token-auth=true

# Q6: verify (three ways)
ps -ef | grep kube-apiserver | grep -v grep                     # running process args = effective config
grep -E 'enable-admission-plugins|disable-admission-plugins' /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl exec -it <kube-apiserver-pod> -n kube-system -- kube-apiserver -h | grep enable-admission-plugins

• Q5 check after restart: create a PVC without storageClassName → it no longer gets the default class.

[!WARNING] Exam traps / remember:

• The manifest is the source of truth kubelet uses; ps shows what is actually running — if they differ, the new Pod hasn't come up (likely a flag typo).
• NamespaceLifecycle does not auto-create namespaces. Create them explicitly; enable NamespaceAutoProvision only when the task demands it.
• kube-apiserver -h shows the default list for that binary, not what this cluster has added/disabled.

[!NOTE] Beyond the course: the default-enabled list grows over versions (newer releases also include e.g. PodSecurity, ValidatingAdmissionPolicy, ClusterTrustBundleAttest). Always read it from the cluster's own binary rather than memorising it. ps -ef | grep kube-apiserver | tr ' ' '\n' | grep admission gives a readable view.

3.24 2025 Updates Validating and Mutating Admission ControllersCopy Anchor Link

Source: KodeKloud notes

Mutating vs validatingCopy Anchor Link

• Admission controllers run after AuthN/AuthZ, before persistence. Mutating = change the object; validating = allow/deny. Some do both.

• Why order matters: NamespaceAutoProvision must create the namespace before NamespaceExists checks it, otherwise the request would be rejected.
• Any controller rejecting → API server returns an error, nothing applied.

Diagram (new): request path through kube-apiserver; mutating runs before validating, webhook plugins call out to an external server. (Schema-validation step and HTTP codes are beyond the course.)

Admission webhooks (custom logic)Copy Anchor Link

• Two built-in plugins call your external server: MutatingAdmissionWebhook (may return a JSONPatch) and ValidatingAdmissionWebhook (allow/deny only).
• API server POSTs an AdmissionReview (JSON, admission.k8s.io/v1) and expects an AdmissionReview back with the same uid and allowed: true|false; denials may carry a status message; mutating responses add patch (base64 JSONPatch) + patchType: "JSONPatch".

{ "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
  "request": {
    "uid": "705ab4f5-6393-11e8-b7cc-42010a800002",
    "kind": {"group": "autoscaling", "version": "v1", "kind": "Scale"},
    "resource": {"group": "apps", "version": "v1", "resource": "deployments"},
    "subResource": "scale",
    "requestKind": {"group": "autoscaling", "version": "v1", "kind": "Scale"},
    "requestResource": {"group": "apps", "version": "v1", "resource": "deployments"} } }

{ "apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
  "response": { "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", "allowed": true } }

Step 1 — deploy the webhook serverCopy Anchor Link

• Any language that can receive/return AdmissionReview JSON. Course shows a Go handler (k8s.io/api/admission/v1 types): read body (400 if empty) → unmarshal AdmissionReview → call admit() → wrap result in a response AdmissionReview (copy TypeMeta), set Response.UID = Request.UID → marshal → write with Content-Type: application/json (errors become metav1.Status{Message}).
• Mutating example (Python/Flask): add label users=<requesting user> via JSONPatch; the patch must be JSON-encoded then base64-encoded.

from flask import Flask, request, jsonify
import base64, json

app = Flask(__name__)

@app.route("/mutate", methods=["POST"])
def mutate():
    req = request.json
    uid = req["request"]["uid"]
    user_name = req["request"]["userInfo"]["name"]
    patch = [{"op": "add", "path": "/metadata/labels/users", "value": user_name}]
    patch_b64 = base64.b64encode(json.dumps(patch).encode("utf-8")).decode("utf-8")
    return jsonify({
        "apiVersion": "admission.k8s.io/v1",   # required in v1 responses (missing in the course snippet)
        "kind": "AdmissionReview",
        "response": {"uid": uid, "allowed": True, "patch": patch_b64, "patchType": "JSONPatch"}
    })
# serve over TLS (the API server only calls webhooks via HTTPS)

[!CAUTION] Correction: with admissionReviewVersions: ["v1"] the reply must include apiVersion: admission.k8s.io/v1 and kind: AdmissionReview; the course's Flask snippet returned only response, which the API server rejects. (Also, a JSONPatch add to /metadata/labels/users fails if the object has no labels map.)

• Hosting: in cluster as Deployment + Service (e.g. webhook-service), or outside the cluster via URL. TLS always required.
• TLS: server cert SAN must contain the Service DNS name (<svc>.<ns>.svc); clientConfig.caBundle = base64 CA that signed it.

[!TIP] Exam scope: you won't write webhook servers — know the flow (AdmissionReview in/out, mutating before validating), the configuration objects, and the TLS/caBundle requirement.

Step 2 — register itCopy Anchor Link

MutatingWebhookConfiguration (mutating) / ValidatingWebhookConfiguration (validating), group admissionregistration.k8s.io/v1.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy.example.com
webhooks:
- name: pod-policy.example.com
  clientConfig:
    service:
      name: webhook-service
      namespace: default
      path: "/validate"
    caBundle: "<base64-ca-bundle>"     # CA that signed the webhook server cert
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None

Runtime behaviourCopy Anchor Link

• API server matches requests against rules → sends AdmissionReview → mutating patches applied → validators run on the mutated object.
• Webhook unreachable → request may be denied, depending on failurePolicy.
• Debug: test in a dev cluster with self-signed certs; log AdmissionReview requests/responses in the server.

[!WARNING] Exam traps / remember:

• Mutating runs before validating. Validating webhooks can't change objects.
• Webhooks need TLS; cert must match <service>.<namespace>.svc; caBundle must hold the signing CA (base64).
• Response uid must equal the request uid.

[!NOTE] Beyond the course: failurePolicy defaults to Fail in v1 (Ignore lets requests through when the webhook is down) — a broken Fail webhook on Pods can block all Pod creation, so delete the configuration to recover (kubectl delete validatingwebhookconfiguration <name>). Other useful fields: namespaceSelector, objectSelector, timeoutSeconds (default 10), clientConfig.service.port (default 443). Kubernetes also has in-process ValidatingAdmissionPolicy (CEL expressions, GA v1.30) as a webhook-free alternative.

[!CAUTION] Correction: a webhook certificate needs the Service DNS name in its SAN; a Common Name alone does not pass current hostname verification. See webhook authentication.

3.25 2025 Updates Lab Solution Validating and Mutating Admission ControllersCopy Anchor Link

Source: KodeKloud notes

Theory: see §3.24.

Quiz answersCopy Anchor Link

• Pairing: NamespaceAutoProvision = mutating (creates namespaces), NamespaceExists = validating (accept/reject).
• Invocation order: mutating first, then validating on the (possibly mutated) object.

DrillCopy Anchor Link

Files in the working directory: keys/, pod-with-defaults.yaml, pod-with-conflict.yaml, pod-with-override.yaml, webhook-configuration.yaml, webhook-deployment.yaml, webhook-service.yaml.