Security

← 06 Cluster Maintenance · Index · 08 Storage →

Contents

• 7.1 Security Section Introduction
• 7.2 Kubernetes Security Primitives
• 7.3 Authentication
• 7.4 TLS Introduction
• 7.5 TLS Basics
• 7.6 TLS in Kubernetes
• 7.7 TLS in Kubernetes Certificate Creation
• 7.8 View Certificate Details
• 7.9 Lab Solution View Certification Details
• 7.10 Certificates API
• 7.11 Lab Solution Certificates API
• 7.12 KubeConfig
• 7.13 Lab Solution KubeConfig
• 7.14 API Groups
• 7.15 Authorization
• 7.16 Role Based Access Controls
• 7.17 Lab Role Based Access Controls
• 7.18 Cluster Roles
• 7.19 Lab Solution Cluster Roles
• 7.20 Service Accounts
• 7.21 Lab Solution Service Accounts
• 7.22 Image Security
• 7.23 Lab Solution Image Security
• 7.24 Prerequisite Security in Docker
• 7.25 Security Contexts
• 7.26 Lab Solution Security Contexts
• 7.27 Network Policies
• 7.28 Developing Network Policies
• 7.29 Lab Solution Network Policies optional
• 7.30 2025 Updates Custom Resource Definition CRD
• 7.31 2025 Updates Custom Controllers
• 7.32 2025 Updates Operator Framework

7.1 Security Section IntroductionCopy Anchor Link

Source: KodeKloud notes

Section mapCopy Anchor Link

Course order: security primitives → securing the persistent key-value store (etcd) → authentication and authorization → security contexts → TLS certificates for cluster components → image security → network policies.

• Section goals: understand how users and components authenticate. Read and interpret certificate config and logs. Apply least-privilege RBAC. Harden workloads with security contexts and image policies. Restrict traffic with NetworkPolicies.
• TLS gets extra depth, from basics through to cluster-level troubleshooting. Certificate problems are the most common issue when you build or debug a cluster yourself. If you already know TLS, focus on the Kubernetes-specific parts and on authorization.

7.2 Kubernetes Security PrimitivesCopy Anchor Link

Source: KodeKloud notes

Host security (control plane and worker nodes)Copy Anchor Link

• A compromised host gives the attacker a path into workloads and cluster services, so lock down host access first.
• Disable root login and password-based SSH. Allow SSH key-based auth only.
• Restrict SSH with allowlists (source IP, bastion host, IAM-based access).
• OS hardening: SELinux/AppArmor, minimal packages, current kernels.
• Lock down cloud/virtualisation management consoles and instance metadata APIs.
• Use centralised logging and host intrusion detection/monitoring.

kube-apiserver: first line of defenceCopy Anchor Link

Every operation (kubectl, client libraries, controllers) goes through the kube-apiserver. Securing it means answering two questions:

• Best practice: narrowly scoped Roles/ClusterRoles. Prefer RoleBinding over ClusterRoleBinding unless you really need cluster-wide rights.

TLS between componentsCopy Anchor Link

• All traffic between etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet and kube-proxy is TLS-encrypted. Production clusters use mutual TLS (mTLS).
• Protect etcd with TLS and restricted access. Enable encryption at rest (§5.12).
• Use mTLS between apiserver ↔ kubelet and between control plane components.
• Automate certificate rotation and lifecycle.
• Expose the API server only to trusted networks/load balancers. Use API aggregation and admission controllers (e.g. PodSecurity, OPA/Gatekeeper) for extra enforcement.

Network security between workloadsCopy Anchor Link

• Default: all pods can reach all pods across the cluster. Restrict with NetworkPolicy (details in §7.27).
• Enforcement is done by the CNI plugin. If the CNI doesn't support NetworkPolicy, policies are accepted but have no effect.
• Principles: start with default-deny ingress/egress per namespace, then allow only the traffic you need. Select with labels. Test in non-prod. Audit traffic with policy logging or flow exporters.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:            # pods this policy protects
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:        # only frontend pods (same namespace) may connect
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

[!WARNING] Exam traps / remember:

• Authentication always runs before authorization.
• NetworkPolicy objects do nothing without a CNI that enforces them.
• With no NetworkPolicy selecting a pod, all traffic to and from it is allowed.

7.3 AuthenticationCopy Anchor Link

Source: KodeKloud notes

Who accesses the clusterCopy Anchor Link

• Actors: admins, developers, end users, third-party services/bots. End-user auth to apps is handled by the apps themselves and is out of scope. This lesson covers management access to kube-apiserver.
• Two identity types:
  • Humans (admins, developers): Kubernetes has no User object. You can't create users with kubectl or list them. Identity comes from outside the cluster: static files, X.509 certs, or an IdP (OIDC/LDAP/Kerberos).
  • Robots (processes, controllers, CI/CD, integrations): ServiceAccounts. These are real API objects that the cluster creates and manages (§7.20).
• Every request (kubectl, SDK, curl) is authenticated by kube-apiserver before authorization and admission:

kubectl get pods --all-namespaces
curl -k https://kube-server-ip:6443/          # -k = skip server cert verification (labs only)

Authentication mechanismsCopy Anchor Link

Static token fileCopy Anchor Link

CSV columns: token,username,uid,"group1,group2" (the group column is optional. Quote it when it holds several groups).

KpjcVbI7rCFAHYPkByTIzRb7gu1cUc4B,user10,u0010,group1
rJjncHmvtXHc6MlWQddhtvNyyhgTdxSC,user11,u0011,group1
mjpOFIEiFoKL9toikaRNtt59ePtczZSq,user12,u0012,group2
PG41IXhs7OjqwWkmBkvgGT9gloyUqZij,user13,u0013,group2

# kube-apiserver flag (systemd ExecStart or static-pod command)
--token-auth-file=/path/to/user-token-details.csv

# use the token as a bearer token
curl -k -H "Authorization: Bearer KpjcVbI7rCFAHYPkByTIzRb7gu1cUc4B" https://master-node-ip:6443/

• Basic-auth file (concept from the course): maps usernames to plaintext passwords plus uid/groups.
• kubeadm runs kube-apiserver as a static pod (systemd setups run it as a host process), so the CSV has to be reachable where the apiserver runs. For a static pod, mount it into the container or follow your distribution's method for supplying extra files.

[!NOTE] Beyond the course: a hostPath mount for the token file in a kubeadm static pod:

Copy Text

# /etc/kubernetes/manifests/kube-apiserver.yaml (relevant fragments only)
spec:
  containers:
  - command:
    - kube-apiserver
    - --token-auth-file=/tmp/users/user-token-details.csv   # path as seen inside the container
    volumeMounts:
    - mountPath: /tmp/users
      name: usr-details
      readOnly: true
  volumes:
  - hostPath:
      path: /tmp/users            # directory on the control-plane host holding the CSV
      type: DirectoryOrCreate
    name: usr-details

• A token only proves identity. The user still needs authorization, e.g. RBAC Role + RoleBinding (§7.16).
• Production: certificates for machine identities, OIDC/LDAP for humans. Both give better security, auditability and IdP integration.

[!CAUTION] Correction: the basic-auth (password) file flag --basic-auth-file was removed in v1.19. Only --token-auth-file still exists for static credentials, and it is insecure (clear text, no rotation, apiserver restart needed for changes).

[!WARNING] Exam traps / remember:

• "List all users" is impossible: there is no User resource. ServiceAccounts are the only identities stored in the cluster.
• Static token and password files store credentials in clear text. Fine for labs, never for production.
• When you add a flag to a static-pod manifest that points at a file, also add a volume/volumeMount, or the apiserver won't start.

7.4 TLS IntroductionCopy Anchor Link

Source: KodeKloud notes

Goals of the TLS lessonsCopy Anchor Link

What TLS certificates are · how to generate, view and configure them · how Kubernetes uses them · how to troubleshoot them.

• Concepts covered: certificate, CA, private key, CSR, SAN, chain.
• How kube-apiserver, kubelet and ingresses consume certificates.

7.5 TLS BasicsCopy Anchor Link

Source: KodeKloud notes

Why TLSCopy Anchor Link

• Protects data in transit (passwords, tokens, financial data). Without it, credentials cross the network as plaintext and can be sniffed.
• A certificate establishes trust: it proves the server (or client) is who it claims to be.

Symmetric vs asymmetric encryptionCopy Anchor Link

SSH (asymmetric key auth)Copy Anchor Link

ssh-keygen                       # creates id_rsa (private) + id_rsa.pub (public)
cat ~/.ssh/authorized_keys       # on the server: public keys allowed to log in
# ssh-rsa AAAAB3NzaC1yc...KhtUBfoTz1BqR V1NThvOo4opzEwRQo1mWx user1
ssh -i id_rsa user1@server1      # authenticate with your private key

• The same public key can go on many servers. Other users generate their own pairs, and you add their public keys to authorized_keys.
• Whoever holds your private key can log in as you.

[!NOTE] Beyond the course: OpenSSH ≥ 9.5 ssh-keygen defaults to Ed25519 (id_ed25519/id_ed25519.pub). Use ssh-keygen -t rsa for the classic id_rsa names.

HTTPS handshake and CA trustCopy Anchor Link

Diagram (new): PKI in two phases: CA issuance of the server certificate, then the per-connection handshake with symmetric session key.

1. Client connects and asks for a secure session.
2. Server sends its certificate (public key + identity).
3. Client validates the chain against its trust store and checks the hostname against the SANs and the expiry.
4. Key exchange produces a shared symmetric session key, by either:
   • RSA key transport: the client encrypts a symmetric key with the server's public key, or
   • ephemeral Diffie-Hellman (ECDHE): preferred in modern TLS because it gives forward secrecy.
5. All further traffic uses the symmetric key.

• Why signatures matter: anyone can create a certificate that claims to be my-bank.com. A phishing or man-in-the-middle server can present one. Clients only trust certificates signed by a CA in their trust store.
• Self-signed cert: signed by its own key and not trusted by default. The browser shows "Your connection is not private" (also shown for expired or untrusted-CA certs).
• Public CAs (Symantec, DigiCert, GlobalSign, Comodo…): their public certs ship with browsers and operating systems.
• Private/internal CA: for internal sites. Its CA certificate must be distributed to every client (e.g. installed in employee browsers) before clients trust what it issues.
• Getting a cert: generate key + CSR → send the CSR to the CA → the CA validates domain ownership and signs with the CA's private key → certificate.

openssl genrsa -out my-bank.key 2048                          # private key
openssl req -new -key my-bank.key -out my-bank.csr \
  -subj "/C=US/ST=CA/O=MyOrg Inc./CN=my-bank.com"             # CSR: public key + identity

Fields to read on a certificate (openssl x509 -in my-bank.crt -text -noout, trimmed):

Serial Number: 420327018966204255
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=<signing CA>
Validity
    Not After : Feb  9 13:41:28 2020 GMT
Subject: CN=my-bank.com
X509v3 Subject Alternative Name:
    DNS:mybank.com, DNS:i-bank.com, DNS:we-bank.com
Subject Public Key Info: ...

mTLS, signing vs encryption, PKICopy Anchor Link

• mTLS: the server also requires a client certificate signed by a CA it trusts. Common in internal and enterprise APIs (and throughout Kubernetes), rare on public websites.
• The two keys of a pair are mathematically linked:
  • Confidentiality: encrypt with the recipient's public key. Only their private key decrypts it.
  • Authenticity/integrity: sign (a hash) with your private key. Anyone verifies with your public key. Signing ≠ encrypting the whole message with the private key. Never share private keys.
• PKI (Public Key Infrastructure) = the whole system of keys, certificates, CAs, issuance and validation.

File naming conventionCopy Anchor Link

[!WARNING] Exam traps / remember:

• Rule of thumb: "key" in the file name = private key. No "key" = certificate/public key.
• A stolen private key lets an attacker impersonate the server. Without forward secrecy, it can also decrypt previously captured sessions. Store keys securely and rotate them if compromised.

7.6 TLS in KubernetesCopy Anchor Link

Source: KodeKloud notes

Three kinds of certificateCopy Anchor Link

• Server certificates: on services, to secure their HTTPS endpoints.
• Root/CA certificates: the CA's own key pair (ca.crt, ca.key). Signs everything else. Distributing ca.crt makes components trust what the CA signs.
• Client certificates: presented by clients to prove identity (mTLS).
• Requirement: all component ↔ component and admin ↔ cluster communication is TLS-encrypted and authenticated.

Component → certificate mapCopy Anchor Link

• kube-apiserver is the only component that talks to etcd. To etcd it is a client. The apiserver can reuse its serving cert/key for this or get a dedicated client pair. Separate pairs make rotation and auditing easier.
• Grouping: client certs (users and components → apiserver) and server certs (apiserver, etcd, kubelet).

CAsCopy Anchor Link

• At least one CA per cluster.
• Single CA for everything: simpler, less isolation.
• Multiple CAs: e.g. a dedicated etcd CA that signs only etcd and its clients. Scoped trust, better isolation. kubeadm does this (§7.9).

Next steps / rulesCopy Anchor Link

• Generate the CA (ca.key/ca.crt) → CSRs per component → sign with the CA (§7.7).
• For hostname/IP verification modern clients only check SANs. Using CN for that is deprecated.
• Plan rotation, secure storage and revocation.

[!WARNING] Exam traps / remember:

• The apiserver and kubelet are each both server and client. etcd and kubelet are servers that the apiserver connects to.
• Private key files (*.key, *-key.pem) must never leak or be committed. If one is compromised, revoke and reissue every certificate that relies on it.

7.7 TLS in Kubernetes Certificate CreationCopy Anchor Link

Source: KodeKloud notes

Tools: OpenSSL (used here), Easy-RSA, CFSSL. Every certificate follows the same flow: genrsa (key) → req -new (CSR with subject) → x509 -req (sign).

1. CA (self-signed root)Copy Anchor Link

openssl genrsa -out ca.key 2048                                       # CA private key
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr    # CA CSR
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt -days 3650   # self-sign with its own key (10 y)

• -signkey = self-signed (signed by its own key). All later certs use -CA ca.crt -CAkey ca.key.
• The root cert should carry Basic Constraints CA:TRUE so that TLS clients accept it as a CA. Set this through an OpenSSL config v3_ca section or -extensions/-extfile. The plain command above does not explicitly add basicConstraints = CA:TRUE; check the generated certificate and the active OpenSSL configuration.
• Protect ca.key: if it is lost you can't sign new certs. If it leaks, everything it signed must be reissued. Use strict file permissions or a secret store.

[!NOTE] Beyond the course: two ways to get CA:TRUE, both checked with OpenSSL 3.x/4.x:

Copy Text

openssl req -x509 -new -key ca.key -subj "/CN=KUBERNETES-CA" -days 3650 -out ca.crt   # req -x509 adds CA:TRUE by default
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt -days 3650 \
  -extfile <(printf "basicConstraints=critical,CA:TRUE\nkeyUsage=critical,keyCertSign,cRLSign\n")

2. Client certificates (admin user, components)Copy Anchor Link

openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr   # CN = user name, O = group
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out admin.crt -days 365

• CN = the user name the apiserver sees (in audit logs and RBAC). Any name works, e.g. kube-admin / kubernetes-admin.
• O (organization) = group. system:masters is a built-in group with full cluster-admin rights. This is what separates the admin from an ordinary user.
• -CAcreateserial creates ca.srl (serial-number file) on first use.
• The same pattern (key → CSR → sign with the CA) applies to the other clients.

[!NOTE] Beyond the course: system components need the system: prefix in CN so that the built-in RBAC roles match: CN=system:kube-scheduler, CN=system:kube-controller-manager, CN=system:kube-proxy.

[!CAUTION] Correction: the original notes put the group in OU (/CN=kubernetes-admin/OU=system:masters, /CN=system:node:node01/OU=system:nodes). Kubernetes reads groups from the O (Organization) field only. An OU value is ignored, so the cert authenticates but gets no system:masters or system:nodes rights. Use /O=system:masters and /O=system:nodes.

[!NOTE] Beyond the course: since kubeadm v1.29, /etc/kubernetes/admin.conf uses O=kubeadm:cluster-admins (bound to the cluster-admin ClusterRole by a ClusterRoleBinding). The break-glass super-admin.conf keeps O=system:masters. Members of system:masters bypass RBAC entirely, and a cert can't be revoked short of rotating the CA.

Using the cert pair:

curl https://kube-apiserver:6443/api/v1/pods \
  --key admin.key --cert admin.crt --cacert ca.crt          # client key + cert, CA to verify the server

apiVersion: v1
kind: Config
clusters:
- name: kubernetes
  cluster:
    certificate-authority: ca.crt          # CA used to verify the apiserver
    server: https://kube-apiserver:6443
users:
- name: kubernetes-admin
  user:
    client-certificate: admin.crt
    client-key: admin.key
contexts:                                  # added so the file is usable as-is (kubeconfig details: §7.12)
- name: kubernetes-admin@kubernetes
  context:
    cluster: kubernetes
    user: kubernetes-admin
current-context: kubernetes-admin@kubernetes

• Every component and client needs a copy of ca.crt to verify the peer it talks to.

3. etcd server (and peer) certificatesCopy Anchor Link

• etcd uses TLS for client ↔ server and peer ↔ peer traffic. In HA clusters each member gets its own peer cert.
• Enable client-cert auth (mTLS) so that only clients signed by the trusted CA can connect.

# etcd flags (paths illustrative)
- --advertise-client-urls=https://127.0.0.1:2379
- --key-file=/path-to-certs/etcdserver.key
- --cert-file=/path-to-certs/etcdserver.crt
- --client-cert-auth=true
- --trusted-ca-file=/path-to-certs/ca.crt
- --peer-cert-file=/path-to-certs/etcdpeer1.crt
- --peer-key-file=/path-to-certs/etcdpeer1.key
- --peer-client-cert-auth=true
- --peer-trusted-ca-file=/path-to-certs/ca.crt
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://127.0.0.1:2380
- --initial-cluster=master=https://127.0.0.1:2380
- --listen-client-urls=https://127.0.0.1:2379
- --listen-peer-urls=https://127.0.0.1:2380
- --name=master
- --snapshot-count=10000

4. kube-apiserver serving cert (SANs)Copy Anchor Link

• Clients reach the apiserver by many names and IPs. All of them must be SANs:
  • DNS: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local
  • IPs: the kubernetes Service ClusterIP (first IP of the service CIDR, e.g. 10.96.0.1) and the host/advertise IP (e.g. 172.17.0.87). Add any LB name/IP too.
• CN = kube-apiserver.

# openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[v3_req]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 10.96.0.1
IP.2 = 172.17.0.87

openssl genrsa -out apiserver.key 2048
openssl req -new -key apiserver.key -subj "/CN=kube-apiserver" -out apiserver.csr -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out apiserver.crt -extensions v3_req -extfile openssl.cnf -days 1000   # SANs are applied at signing time

[!CAUTION] Correction: the original openssl.cnf fragment references distinguished_name = req_distinguished_name but has no [req_distinguished_name] section. OpenSSL then fails with "Unable to get 'req_distinguished_name' section". An empty section (as above) fixes it. Also, openssl x509 -req does not copy extensions from the CSR, so the -extensions v3_req -extfile openssl.cnf at signing time is what puts the SANs into the certificate.

5. kube-apiserver flags that reference certsCopy Anchor Link

ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=${INTERNAL_IP} \
  --allow-privileged=true \
  --apiserver-count=3 \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --enable-swagger-ui=true \
  --etcd-cafile=/var/lib/kubernetes/ca.crt \
  --etcd-certfile=/var/lib/kubernetes/apiserver-etcd-client.crt \
  --etcd-keyfile=/var/lib/kubernetes/apiserver-etcd-client.key \
  --etcd-servers=https://127.0.0.1:2379 \
  --event-ttl=1h \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt \
  --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.crt \
  --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client.key \
  --kubelet-https=true \
  --runtime-config=api/all \
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --client-ca-file=/var/lib/kubernetes/ca.crt \
  --tls-cert-file=/var/lib/kubernetes/apiserver.crt \
  --tls-private-key-file=/var/lib/kubernetes/apiserver.key \
  --v=2

[!CAUTION] Correction (outdated flags): --kubelet-https and --enable-swagger-ui have been removed from current kube-apiserver releases, and --apiserver-count is deprecated. Don't copy them into a modern manifest. Current apiservers also require --service-account-issuer and --service-account-signing-key-file (see the kubeadm manifest in §7.9).

6. kubelet certificates (per node)Copy Anchor Link

• Server cert (apiserver → kubelet HTTPS): one per node (node01, node02, node03), named after the node (CN or SAN). Pass it with kubelet flags.
• Client cert (kubelet → apiserver): CN = system:node:<nodeName>, O = system:nodes, e.g. /CN=system:node:node01/O=system:nodes. The apiserver needs this format to give the kubelet node permissions through the Node authorizer.
• Put the client cert/key and ca.crt into each kubelet's kubeconfig.

7. InspectCopy Anchor Link

openssl x509 -in apiserver.crt -text -noout      # cert: subject, issuer, validity, SANs
openssl req  -in apiserver.csr -text -noout      # CSR: check subject/SANs before signing

[!WARNING] Exam traps / remember:

• Self-sign the CA with -signkey ca.key. Sign everything else with -CA ca.crt -CAkey ca.key.
• Group = O, user = CN. Admin → O=system:masters. Nodes → CN=system:node:<name>, O=system:nodes. System components → CN=system:kube-scheduler and the like.
• A missing SAN (e.g. a new LB name) → "x509: certificate is valid for …, not …". Re-issue with the SAN added.
• Distribute ca.crt everywhere. Keep *.key private. Rotate before expiry and keep an inventory.

[!NOTE] Beyond the course: kubelet serving-certificate flags are --tls-cert-file / --tls-private-key-file; the KubeletConfiguration fields are tlsCertFile / tlsPrivateKeyFile.

7.8 View Certificate DetailsCopy Anchor Link

Source: KodeKloud notes

Step 1: how was the cluster built?Copy Anchor Link

cat /etc/systemd/system/kube-apiserver.service      # manual setup
cat /etc/kubernetes/manifests/kube-apiserver.yaml   # kubeadm

# manual setup: cert-related ExecStart flags (excerpt)
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=172.17.0.32 \
  --authorization-mode=Node,RBAC \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  -v=2

• In this manual setup a single pair (kubernetes.pem/kubernetes-key.pem) acts as the serving cert and the etcd/kubelet client cert: one cert reused for several roles.

# kubeadm: /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
spec:
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --advertise-address=172.17.0.32
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

Step 2: build a certificate inventory (spreadsheet)Copy Anchor Link

For every cert record: path, CN, SANs (alt names), Organization (O), Issuer, Expiration (UTC), purpose. This shows certs that expire soon or were issued by the wrong CA.

[!CAUTION] Correction: the original inventory’s last two rows misidentify the etcd CA and issuer: kubeadm uses CN etcd-ca, with a 10-year CA lifetime, and apiserver-etcd-client.crt is issued by that CA rather than self-signed. The 2017 expiry in the source is a typo. Inspect the actual certificates (§7.9).

• Reference for expected CN, parent CA, O, kind and SANs per cert, plus the recommended paths and the flag that consumes each one: Kubernetes PKI best practices.

Step 3: decode each certCopy Anchor Link

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3147495682089747350 (0x2bae26a58f090396)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes                          <- signing CA
        Validity
            Not Before: Feb 11 05:39:19 2019 GMT
            Not After : Feb 11 05:39:20 2020 GMT       <- expiry
        Subject: CN=kube-apiserver                     <- identity (and O= groups, if any)
    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        X509v3 Extended Key Usage:
            TLS Web Server Authentication              <- server vs client cert
    X509v3 Subject Alternative Name:
        DNS:master, DNS:kubernetes, DNS:kubernetes.default,
        DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local,
        IP Address:10.96.0.1, IP Address:172.17.0.27

Audit checklistCopy Anchor Link

• SANs cover every hostname/IP clients use: service ClusterIP, service DNS names, node IPs.
• Issuer is the expected CA (kubeadm CA or the organisation's CA).
• Validity: not expired, and the renewal policy fits your guidelines. An expired cert fails the TLS handshake immediately.
• Key usage / EKU fit the role (server vs client).
• Subject O fits the usage (e.g. system:masters only on privileged clients).

Step 4: logs for TLS failuresCopy Anchor Link

journalctl -u etcd.service -l                 # systemd-managed
kubectl logs etcd-controlplane -n kube-system # kubeadm static pod (pod name = etcd-<node>)
crictl ps -a                                  # apiserver/etcd down → kubectl fails → go to the runtime
crictl logs <container-id>

2019-02-13 02:53:28.185588 I | embed: ClientTLS: cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, trusted-ca = /etc/kubernetes/pki/etcd/old-ca.crt, client-cert-auth = true
WARNING: 2019/02/13 02:53:30 Failed to dial 127.0.0.1:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

• tls: bad certificate = the presented cert failed validation: SAN/CN mismatch, wrong issuer/CA (here trusted-ca points at old-ca.crt), or expired.

[!NOTE] Beyond the course (kubeadm cert lifecycle, exam-relevant):

Copy Text

kubeadm certs check-expiration                      # expiry + issuing CA of every kubeadm-managed cert/kubeconfig
kubeadm certs renew all                             # or: kubeadm certs renew apiserver
# then restart the control-plane static pods (e.g. move manifests out of /etc/kubernetes/manifests and back)
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -enddate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName

kubeadm leaf certs are valid for 1 year, CAs for 10 years. A kubeadm upgrade apply also renews them.

[!WARNING] Exam traps / remember:

• Identify the setup first (systemd vs static pods). The command and log location depend on it.
• Hostname checks use SANs, not CN.
• If kube-apiserver is down, kubectl logs doesn't work either. Use crictl ps -a / crictl logs, or journalctl -u kubelet.

7.9 Lab Solution View Certification DetailsCopy Anchor Link

Source: KodeKloud notes

Diagram (new): kubeadm's three CAs, which certificates each one signs, and which kube-apiserver/etcd flags point at each file.

Drill: find the files (answers come from the manifests)Copy Anchor Link

cat /etc/kubernetes/manifests/kube-apiserver.yaml   # or: grep -E 'crt|key|ca' on it
cat /etc/kubernetes/manifests/etcd.yaml
ls /etc/kubernetes/pki /etc/kubernetes/pki/etcd

kube-apiserver manifest from the lab (excerpt, kubeadm v1.20):

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.46.98.9
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.20.0

etcd manifest from the lab (excerpt):

spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.46.98.9:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.46.98.9:2380
    - --initial-cluster=controlplane=https://10.46.98.9:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://10.46.98.9:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://10.46.98.9:2380
    - --name=controlplane
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.4.13-0

• Ports: apiserver 6443. etcd clients 2379, peers 2380, metrics 2381 (plain http, localhost).
• /etc/kubernetes/pki/etcd/ holds: ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt server.key.

[!CAUTION] Correction (outdated lab output): --insecure-port was removed from kube-apiserver in v1.24, so it is absent from current manifests. Images now come from registry.k8s.io (the k8s.gcr.io registry is frozen).

Drill: read cert details (openssl x509 -in <crt> -text -noout)Copy Anchor Link

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout
openssl x509 -in /etc/kubernetes/pki/ca.crt -text -noout | grep -A2 Validity

Drill: troubleshooting scenario 1 (kubectl: "connection refused")Copy Anchor Link

The connection to the server was refused - did you specify the right host or port?

1. The apiserver is a static pod, so check it through the runtime:

   Copy Text

   crictl ps -a | grep kube-apiserver        # Exited / restarting?
   crictl logs <kube-apiserver-container-id>

2. apiserver log shows it can't reach etcd:

   Copy Text

   grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379 ...}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

   → etcd isn't listening. Check etcd: crictl ps -a | grep etcd, crictl logs <etcd-id>.
3. etcd log:

   Copy Text

   embed: ClientTLS: cert = /etc/kubernetes/pki/etcd/server-certificate.crt, key = /etc/kubernetes/pki/etcd/server.key, ...
   C | etcdmain: open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory

4. ls /etc/kubernetes/pki/etcd shows the file is really server.crt. Fix: in /etc/kubernetes/manifests/etcd.yaml set --cert-file=/etc/kubernetes/pki/etcd/server.crt and save. The kubelet notices the change and recreates the static pod. etcd binds 127.0.0.1:2379 and 10.46.98.9:2379, the apiserver reconnects, and kubectl works again.

Drill: troubleshooting scenario 2 ("TLS handshake timeout")Copy Anchor Link

Unable to connect to the server: net/http: TLS handshake timeout

1. etcd logs are full of rejections:

   Copy Text

   embed: rejected connection from "127.0.0.1:39626" (error "remote error: tls: bad certificate", ServerName "")

   → mTLS trust mismatch between the apiserver and etcd.
2. The apiserver manifest has --etcd-cafile=/etc/kubernetes/pki/ca.crt. That is the Kubernetes CA, not the CA that signed etcd's serving cert.
3. Fix in /etc/kubernetes/manifests/kube-apiserver.yaml:

   Copy Text

       - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
       - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
       - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
       - --etcd-servers=https://127.0.0.1:2379

   Save → the kubelet restarts the apiserver → handshakes succeed and kubectl comes back.

Debug checklist (apiserver ↔ etcd)Copy Anchor Link

• Check both manifests: /etc/kubernetes/manifests/kube-apiserver.yaml and etcd.yaml.
• Every referenced path must exist (ls /etc/kubernetes/pki, ls /etc/kubernetes/pki/etcd). A wrong path stops the static pod from starting, and the container log (or kubelet log) names the missing file.
• mTLS alignment: etcd serves etcd/server.crt and trusts etcd/ca.crt. The apiserver must use --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt and present apiserver-etcd-client.crt/.key (signed by the etcd CA).
• Compare the Issuer: of a cert with the CA file the other side is configured with. A mismatch is the classic cause.

[!CAUTION] Correction: the lab uses docker ps -a | grep kube-apiserver and docker logs <id>. Since dockershim was removed (v1.24), nodes run containerd/CRI-O, so use crictl ps -a and crictl logs <id>. The flags are the same.

[!NOTE] Beyond the course:

• After editing a static-pod manifest, wait. The kubelet recreates the pod by itself (watch with watch crictl ps). If nothing appears, check journalctl -u kubelet for YAML or path errors.
• Back up a manifest outside /etc/kubernetes/manifests before editing it. A copy inside the directory gets run as a second static pod.

• x509: certificate signed by unknown authority → compare the peer certificate’s Issuer: with the configured CA; for etcd use etcd/ca.crt, not the Kubernetes ca.crt.
• Inspect the etcd server cert with openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout. Course sample: Apr 17 2022 → Apr 17 2023, SANs controlplane, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, IPs 10.96.0.1, 10.46.98.9. Treat these as sample output, not required SANs for every etcd server.
• etcd static-Pod metadata: name etcd, namespace kube-system, labels component=etcd,tier=control-plane; find it with kubectl -n kube-system get pods -l component=etcd.

PKI map legend: blue marks serving certificates; red marks signing CAs/client certificates. This diagram uses certificate type, rather than component role, for colour.

7.10 Certificates APICopy Anchor Link

Source: KodeKloud notes

Why a Certificates APICopy Anchor Link

• The cluster signing CA (CA private key + root cert) is the crown jewel: whoever holds the key can mint certs for any identity, including privileged ones. Keep it on a dedicated CA server or in a key-management solution.
• kubeadm keeps the CA pair on the control-plane node (/etc/kubernetes/pki/ca.crt, ca.key), so that node effectively is the CA server.
• Signing CSRs by hand (openssl x509 -req -CA ..., see §7.7) does not scale. The Certificates API makes it an auditable workflow: CSRs become API objects that admins review, approve or deny, and the cluster signs approved ones automatically.

WorkflowCopy Anchor Link

1. User generates a private key and a CSR locally (the key never leaves the user).
2. User (or admin) submits the CSR as a CertificateSigningRequest object.
3. Admins list and review pending CSRs.
4. Admin approves (or denies) with kubectl.
5. If approved and the controller manager has the CA, the cluster signs it and stores the cert, base64-encoded, in status.certificate.
6. Extract the cert and hand it, together with the key, to the user (client auth / TLS).

Diagram (new): CSR lifecycle from key generation to extracted certificate, and where the CA key is used.

Commands: key, CSR, objectCopy Anchor Link

openssl genrsa -out jane.key 2048                                   # private key
openssl req -new -key jane.key -subj "/CN=jane/O=dev-team" -out jane.csr   # CN = username, O = group
cat jane.csr | base64 -w0                                           # single-line base64 for spec.request

# jane-csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane-csr
spec:
  signerName: kubernetes.io/kube-apiserver-client   # client cert for authenticating to the API server
  expirationSeconds: 600                            # optional: requested cert lifetime in seconds
  usages:
  - client auth
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K...   # base64 of the whole PEM CSR, one line

kubectl apply -f jane-csr.yaml
kubectl get csr                       # NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION -> Pending
kubectl describe csr jane-csr
kubectl certificate approve jane-csr  # or: kubectl certificate deny jane-csr
kubectl get csr jane-csr -o yaml      # status.conditions[type=Approved, reason=KubectlApprove] + status.certificate
kubectl get csr jane-csr -o jsonpath='{.status.certificate}' | base64 --decode > jane.crt

• After approval, status.conditions gets type: Approved, status: "True", reason: KubectlApprove; status.certificate is filled once signed (CONDITION Approved,Issued).
• Give jane.key + jane.crt to the user (use in kubeconfig, §7.12).

signerName and usagesCopy Anchor Link

[!CAUTION] Correction: the notes name kubernetes.io/kube-apiserver-serving as a serving signer. Kubernetes has no such built-in signer. The built-in signers are kubernetes.io/kube-apiserver-client, kubernetes.io/kube-apiserver-client-kubelet (kubelet client certs, can be auto-approved), kubernetes.io/kubelet-serving (kubelet serving certs, never auto-approved) and kubernetes.io/legacy-unknown (legacy v1beta1 only; cannot be requested using certificates.k8s.io/v1 and was never auto-approved). For users authenticating to the API, always use kubernetes.io/kube-apiserver-client with client auth.

Where signing happensCopy Anchor Link

• All Certificates API automation lives in kube-controller-manager controllers. They expose pending CSRs for review, can auto-approve some CSRs by configured rules, and sign approved CSRs.
• Signing requires the CA files on the controller manager:
  • --cluster-signing-cert-file (CA cert)
  • --cluster-signing-key-file (CA key)
• Anyone who can read those files or change controller-manager flags/startup files can mint identities, so lock down access to them.

[!WARNING] Exam traps / remember:

• spec.request = base64 of the entire PEM file (including the BEGIN/END lines), on one line (base64 -w0 or | tr -d '\n'). A multi-line value breaks the object.
• signerName is mandatory in certificates.k8s.io/v1. usages must match the signer (client auth for kube-apiserver-client).
• Approval alone does not mean issued: CONDITION goes Pending → Approved → Approved,Issued. status.certificate is base64, so decode it.
• CSRs are cluster-scoped (no -n).
• Signing is done by kube-controller-manager, using --cluster-signing-cert-file / --cluster-signing-key-file.

[!NOTE] Beyond the course:

• expirationSeconds minimum is 600 (10 min). Unset means the signer default, which kube-controller-manager caps with --cluster-signing-duration (default 1 year).
• On kubeadm the flags are in /etc/kubernetes/manifests/kube-controller-manager.yaml and point to /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key.
• Build the manifest in one go with a heredoc:

  Copy Text

  cat <<EOF | kubectl apply -f -
  apiVersion: certificates.k8s.io/v1
  kind: CertificateSigningRequest
  metadata:
    name: jane-csr
  spec:
    signerName: kubernetes.io/kube-apiserver-client
    usages: ["client auth"]
    request: $(base64 -w0 < jane.csr)
  EOF

7.11 Lab Solution Certificates APICopy Anchor Link

Source: KodeKloud notes

Scenario: new team member Akshay needs access. His CSR and key are on the control plane at /root/akshay.csr and /root/akshay.key. Theory: §7.10.

# akshay.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: akshay
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...   # output of base64 -w 0
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth

• Usages reflect the purpose: client auth, server auth, digital signature, etc.
• After cleanup kubectl get csr still lists csr-8mfn8, signer kubernetes.io/kube-apiserver-client-kubelet, requestor system:node:controlplane, Approved,Issued. That is the node's own kubelet client CSR, created and auto-approved when the node joined. Leave it alone.
• Useful fields to check before approving: spec.groups, spec.usages, spec.signerName, spec.username.

[!CAUTION] Correction: the lab text says approving makes "the API server" sign the certificate. The signer for kubernetes.io/kube-apiserver-client is the CSR signing controller in kube-controller-manager, using --cluster-signing-cert-file / --cluster-signing-key-file (§7.10).

[!WARNING] Exam traps / remember:

• Check spec.groups before approving. system:masters in a CSR is a red flag, so deny it.
• base64 -d (GNU) vs base64 -D (BSD/macOS). The exam terminal is Linux, so use -d / -w 0.

7.12 KubeConfigCopy Anchor Link

Source: KodeKloud notes

Without a kubeconfigCopy Anchor Link

# raw REST call with client cert auth
curl https://my-kube-playground:6443/api/v1/pods \
  --key admin.key --cert admin.crt --cacert ca.crt

# same credentials passed to kubectl as flags
kubectl get pods \
  --server=https://my-kube-playground:6443 \
  --client-key=admin.key \
  --client-certificate=admin.crt \
  --certificate-authority=ca.crt

• Typing these every time is tedious, so put them in a kubeconfig file.
• Default location: ~/.kube/config ($HOME/.kube/config). Override with --kubeconfig=<path> or the KUBECONFIG env var.

Structure: clusters, users, contextsCopy Anchor Link

• In the curl example: server + ca.crt → clusters; admin.key/admin.crt → users; a context my-kube-admin@my-kube-playground ties them together.
• A kubeconfig only tells kubectl which credentials to present to which server. Adding a user or context creates nothing in the cluster and grants no permissions.
• Context names are free text. user@cluster is just a convention.

apiVersion: v1
kind: Config
current-context: dev-user@google        # default context
clusters:
- name: my-kube-playground
  cluster:
    server: https://my-kube-playground:6443
    certificate-authority: /path/to/ca.crt
- name: development
  cluster:
    server: https://dev.example.com:6443
    certificate-authority: /path/to/dev-ca.crt
- name: production
  cluster:
    server: https://prod.example.com:6443
    certificate-authority: /path/to/prod-ca.crt
- name: google                            # added: the context below references it
  cluster:
    server: https://google.example.com:6443
    certificate-authority: /path/to/google-ca.crt
contexts:
- name: my-kube-admin@my-kube-playground
  context:
    cluster: my-kube-playground           # must match a clusters[].name
    user: my-kube-admin                   # must match a users[].name
- name: dev-user@google
  context:
    cluster: google
    user: dev-user
- name: prod-user@production
  context:
    cluster: production
    user: prod-user
users:
- name: my-kube-admin
  user:
    client-certificate: /path/to/admin.crt
    client-key: /path/to/admin.key
- name: dev-user
  user:
    client-certificate: /path/to/dev.crt
    client-key: /path/to/dev.key
- name: prod-user
  user:
    client-certificate: /path/to/prod.crt
    client-key: /path/to/prod.key

(The original example's dev-user@google context referenced a google cluster that was never defined. A cluster entry has been added here so the file is valid.)

View and switchCopy Anchor Link

kubectl config view                                  # current kubeconfig; *-data shown as REDACTED (DATA+OMITTED)
kubectl config view --kubeconfig=my-custom-config    # inspect another file
kubectl config use-context prod-user@production      # rewrites current-context in the file
kubectl config --help                                # set-cluster, set-credentials, set-context, delete-context, ...

Namespace in a contextCopy Anchor Link

apiVersion: v1
kind: Config
clusters:
- name: production
  cluster:
    server: https://172.17.0.51:6443
    certificate-authority: /etc/kubernetes/pki/ca.crt
contexts:
- name: admin@production
  context:
    cluster: production
    user: admin
    namespace: finance                 # kubectl without -n uses "finance" in this context
users:
- name: admin
  user:
    client-certificate: /etc/kubernetes/pki/admin.crt
    client-key: /etc/kubernetes/pki/admin.key

Certificate paths vs inline dataCopy Anchor Link

base64 -w0 /etc/kubernetes/pki/ca.crt > ca.crt.base64        # encode for *-data
echo 'LS0tLS1...rest_of_base64...' | base64 --decode > ca.crt # decode *-data back to PEM

• Inline data gives a portable, self-contained file, but a leaked file leaks the key too. Protect it with file permissions.
• Use straight quotes in shell commands. "Smart quotes" pasted from documents break them.

[!WARNING] Exam traps / remember:

• certificate-authority takes a path. certificate-authority-data takes base64 content. Mixing them up is a classic broken-kubeconfig bug.
• use-context changes the file, so it persists for later commands. Exam tasks often begin with kubectl config use-context <ctx>. Always run it.
• Context cluster/user values must exactly match clusters[].name / users[].name.

[!NOTE] Beyond the course:

Copy Text

kubectl config get-contexts                                   # list contexts, * = current
kubectl config current-context
kubectl config set-context --current --namespace=finance      # change default ns of current context
kubectl config set-cluster prod --server=https://172.17.0.51:6443 --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true
kubectl config set-credentials jane --client-certificate=jane.crt --client-key=jane.key --embed-certs=true
kubectl config set-context jane@prod --cluster=prod --user=jane --namespace=dev
kubectl config view --raw                                     # show real *-data values (not REDACTED)
kubectl config view --minify                                  # only the current context's entries

• KUBECONFIG=file1:file2 merges several files. The first file wins for current-context.

7.13 Lab Solution KubeConfigCopy Anchor Link

Source: KodeKloud notes

Theory: §7.12.

Relevant part of /root/my-kube-config:

apiVersion: v1
kind: Config
current-context: test-user@development
clusters:
- name: production
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443
- name: development
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443
- name: kubernetes-on-aws
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443
- name: test-cluster-1
  cluster:
    certificate-authority: /etc/kubernetes/pki/ca.crt
    server: https://controlplane:6443
contexts:
- name: test-user@development
  context: {cluster: development, user: test-user}
- name: aws-user@kubernetes-on-aws
  context: {cluster: kubernetes-on-aws, user: aws-user}
- name: test-user@production
  context: {cluster: production, user: test-user}
- name: research
  context: {cluster: test-cluster-1, user: dev-user}
users:
- name: test-user
  user:
    client-certificate: /etc/kubernetes/pki/users/test-user/test-user.crt
    client-key: /etc/kubernetes/pki/users/test-user/test-user.key
- name: dev-user
  user:
    client-certificate: /etc/kubernetes/pki/users/dev-user/developer-user.crt   # BUG: file is dev-user.crt
    client-key: /etc/kubernetes/pki/users/dev-user/dev-user.key
- name: aws-user
  user:
    client-certificate: /etc/kubernetes/pki/users/aws-user/aws-user.crt
    client-key: /etc/kubernetes/pki/users/aws-user/aws-user.key

[!TIP] When kubectl auth fails, check the users: entry first: do the client-certificate / client-key paths exist? (ls them.) Then check whether the cert is valid for this cluster's CA.

7.14 API GroupsCopy Anchor Link

Source: KodeKloud notes

The API surfaceCopy Anchor Link

• Every interaction (kubectl, client libraries, curl) goes to kube-apiserver. Resources are organised by API group and version.

curl -k https://kube-master:6443/version        # {"major":"1","minor":...,"gitVersion":"v1.x.y",...}
curl -k https://kube-master:6443/api/v1/pods    # PodList from the core group

• Top-level paths include /api, /apis, /healthz, /logs, /metrics, /openapi/v2, /version.

Core vs named groupsCopy Anchor Link

• Core resources: namespaces, pods, replicationcontrollers (rc), events, endpoints, nodes, bindings, persistentvolumes (PV), persistentvolumeclaims (PVC), configmaps, secrets, services.

• Named groups: apps, extensions, networking.k8s.io, storage.k8s.io, authentication.k8s.io, authorization.k8s.io, certificates.k8s.io, ...

• Hierarchy: API group → version → resource → verbs. Verbs (actions) include list, get, create, delete, update, watch. Authorization rules are written against exactly these (group, resource, verb). See §7.16.

DiscoveryCopy Anchor Link

curl -k https://localhost:6443          # lists available paths (if authorised)
curl -k https://localhost:6443/apis     # named groups and their versions

• Without credentials most paths return 403 Forbidden, User "system:anonymous" cannot get path "/".

kubectl proxyCopy Anchor Link

kubectl proxy                            # Starting to serve on 127.0.0.1:8001
curl http://localhost:8001               # no certs needed: proxy uses your kubeconfig credentials
curl http://localhost:8001/api/v1/namespaces/default/pods

[!WARNING] Exam traps / remember:

• Core group is the empty string "" in RBAC rules and plain v1 in apiVersion. Deployments are in apps, not core.
• Don't confuse kubectl proxy (local API access, port 8001) with kube-proxy (Service networking).

[!NOTE] Beyond the course:

Copy Text

kubectl api-resources                      # NAME, SHORTNAMES, APIVERSION, NAMESPACED, KIND
kubectl api-resources --api-group=apps     # resources of one group ("" = core)
kubectl api-resources --namespaced=false   # cluster-scoped resources
kubectl api-versions                       # all group/version pairs
kubectl get --raw /apis | head             # raw API call with kubeconfig creds, no proxy needed
kubectl explain deployment                 # shows GROUP/VERSION of a kind

• extensions (shown on the slide) no longer serves any resources: Deployments/DaemonSets/ReplicaSets moved to apps/v1, Ingress to networking.k8s.io/v1 (removed in v1.16 / v1.22).
• metadata.selfLink in the sample output was removed in v1.24.

Discovery paths returned: /api, /api/v1, /apis, /apis/, /healthz, /logs, /metrics, /openapi/v2, /swagger-2.0.0.json. /version is a separate endpoint.

7.15 AuthorizationCopy Anchor Link

Source: KodeKloud notes

PurposeCopy Anchor Link

• Authentication = who you are (§7.3). Authorization = what an authenticated user or service account may do.
• Goals: developers can deploy apps but not touch nodes, networking or storage config. CI/CD (e.g. Jenkins) and monitoring get the minimum needed. Users are restricted to their namespaces.

ModesCopy Anchor Link

• In production prefer fine-grained modes (RBAC, plus Webhook if needed) over global permissive ones.

Node authorizerCopy Anchor Link

• Kubelets authenticate with certs whose CN is system:node:<nodeName> and O is system:nodes (the system: prefix and system:nodes group are what the Node authorizer looks for).
• It grants kubelets only what they need:
  • Read: services, endpoints, nodes, pods
  • Write: node status, pod status, events
• It applies only to node-originated requests.

ABAC policy fileCopy Anchor Link

One JSON object per line, one line per user/group grant:

{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user": "dev-user", "namespace": "*", "resource": "pods", "apiGroup": "*"}}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user": "dev-user-2", "namespace": "*", "resource": "pods", "apiGroup": "*"}}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group": "dev-users", "namespace": "*", "resource": "pods", "apiGroup": "*"}}
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user": "security-1", "namespace": "*", "resource": "certificatesigningrequests", "apiGroup": "*"}}

Enabled with --authorization-mode=ABAC --authorization-policy-file=<file>.

[!CAUTION] Correction: the course's ABAC lines omit apiVersion and use "resource": "csr". Real policy lines need "apiVersion": "abac.authorization.kubernetes.io/v1beta1", and resource must be the plural resource name (certificatesigningrequests), not a kubectl short name. Corrected above.

Multiple modes: evaluation orderCopy Anchor Link

• Set on kube-apiserver with --authorization-mode, a comma-separated list. Default if unset: AlwaysAllow.
• Authorizers are consulted in the listed order. Each returns Allow (stop, permit), Deny (stop, reject) or No opinion (pass to the next).
• Example --authorization-mode=Node,RBAC,Webhook: Node handles kubelet requests first. Otherwise RBAC is tried, then Webhook. The first Allow/Deny wins.

Diagram (new): short-circuit evaluation of a comma-separated --authorization-mode list.

Configuring (systemd-managed API server)Copy Anchor Link

# kube-apiserver.service (excerpt) — insecure default shown
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=${INTERNAL_IP} \
  --allow-privileged=true \
  --apiserver-count=3 \
  --authorization-mode=AlwaysAllow \
  --bind-address=0.0.0.0 \
  --enable-swagger-ui=true \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/apiserver-etcd-client.crt \
  --etcd-keyfile=/var/lib/kubernetes/apiserver-etcd-client.key \
  --etcd-servers=https://127.0.0.1:2379 \
  --event-ttl=1h \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/apiserver-kubelet-client.crt \
  --kubelet-client-key=/var/lib/kubernetes/apiserver-kubelet-client.key \
  --service-node-port-range=30000-32767 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --tls-cert-file=/var/lib/kubernetes/apiserver.crt \
  --tls-private-key-file=/var/lib/kubernetes/apiserver.key \
  --v=2

Change to --authorization-mode=Node,RBAC,Webhook, then restart the API server (systemctl daemon-reload && systemctl restart kube-apiserver).

[!WARNING] Exam traps / remember:

• Know all six modes: Node, ABAC, RBAC, Webhook, AlwaysAllow, AlwaysDeny.
• Order matters: the first authorizer that returns Allow or Deny decides.
• To find the modes on a kubeadm cluster: grep authorization-mode /etc/kubernetes/manifests/kube-apiserver.yaml or ps -ef | grep kube-apiserver → typically Node,RBAC (§7.17).

[!NOTE] Beyond the course:

• If every authorizer has no opinion, the request is denied. RBAC never returns Deny, only Allow or no opinion, so you cannot write "deny" rules in RBAC.
• On kubeadm clusters, edit /etc/kubernetes/manifests/kube-apiserver.yaml. The kubelet restarts the static pod automatically, so no systemctl is needed.
• --enable-swagger-ui has been removed from current kube-apiserver. Newer clusters can use --authorization-config (structured authorization config file, GA in v1.32) instead of --authorization-mode. The two flags are mutually exclusive.

• Webhook: use it for policy managed centrally outside the cluster, or when policies are complex or change often (e.g. OPA).
• RBAC also suits CI/CD and service-account-based automation.

7.16 Role Based Access ControlsCopy Anchor Link

Source: KodeKloud notes

Role = namespaced set of rulesCopy Anchor Link

# developer-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: default            # Role lives in (and grants within) this namespace
rules:
- apiGroups: [""]               # core API group
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create"]

kubectl apply -f developer-role.yaml     # or: kubectl create -f developer-role.yaml

RoleBinding = attach Role to subjectsCopy Anchor Link

# devuser-developer-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: devuser-developer-binding
  namespace: default                    # same namespace as the Role
subjects:                               # who: User, Group or ServiceAccount
- kind: User
  name: dev-user
  apiGroup: rbac.authorization.k8s.io
roleRef:                                # what: exactly one Role (or ClusterRole)
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f devuser-developer-binding.yaml   # or kubectl create -f ...

• Roles and RoleBindings are namespace-scoped. The RoleBinding sits in the Role's namespace (it may instead reference a ClusterRole). For cluster-wide access use ClusterRole + ClusterRoleBinding (§7.18).

Diagram (new): how Role, RoleBinding and subjects relate inside a namespace.

View RBAC objectsCopy Anchor Link

$ kubectl describe role developer
PolicyRule:
  Resources   Non-Resource URLs  Resource Names  Verbs
  ---------   -----------------  --------------  -----
  configmaps  []                 []              [create]
  pods        []                 []              [get list create delete]

(The course output also shows watch for pods. That only appears if watch is in the Role's verbs.)

Test permissions: kubectl auth can-iCopy Anchor Link

kubectl auth can-i create deployments                     # yes/no for the current user
kubectl auth can-i delete nodes                           # no
kubectl auth can-i create deployments --as=dev-user       # impersonate a user (admin only)
kubectl auth can-i create pods --as=dev-user -n test      # namespace-specific check -> no if only granted in default

Restrict to named objectsCopy Anchor Link

rules:
- apiGroups: [""]
  resources: ["pods"]
  resourceNames: ["blue", "orange"]   # only these two pods; all other pods denied
  verbs: ["get"]

[!WARNING] Exam traps / remember:

• apiGroups: [""] for core resources. Deployments need apiGroups: ["apps"].
• Resources are plural and lowercase (pods, deployments).
• Subject kind is case-sensitive: User, Group, ServiceAccount. Users/Groups use apiGroup: rbac.authorization.k8s.io. A ServiceAccount subject needs namespace (and apiGroup: "").
• Role/RoleBinding work only in their namespace. can-i without -n checks the current namespace.

[!NOTE] Beyond the course: imperative forms (much faster in the exam):

Copy Text

kubectl create role developer --verb=get,list,create,delete --resource=pods -n default
kubectl create role cm-reader --verb=get --resource=configmaps --resource-name=kube-proxy -n kube-system
kubectl create rolebinding devuser-developer-binding --role=developer --user=dev-user -n default
kubectl create rolebinding sa-bind --role=developer --serviceaccount=default:my-sa   # ns:name
kubectl create rolebinding grp-bind --role=developer --group=dev-users
kubectl auth can-i --list --as=dev-user -n default        # everything dev-user may do
kubectl auth can-i get pods --as=system:serviceaccount:default:my-sa

• Standard resource verbs: get, list, watch, create, update, patch, delete, deletecollection (* = all). Special permissions include bind, escalate, impersonate, and CSR signer approve / sign.
• roleRef is immutable. To point a binding at another role, delete and recreate it.

7.17 Lab Role Based Access ControlsCopy Anchor Link

Source: KodeKloud notes

Theory: §7.15 (modes), §7.16 (Role/RoleBinding).

Final Role in blue:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: blue
rules:
- apiGroups: [""]
  resources: ["pods"]
  resourceNames: ["dark-blue-app"]     # only this pod
  verbs: ["get", "watch", "create", "delete"]
- apiGroups: ["apps"]                  # Deployments are NOT in the core group
  resources: ["deployments"]
  verbs: ["get", "watch", "create", "delete"]

$ kubectl describe role developer -n blue
  Resources         Non-Resource URLs  Resource Names   Verbs
  pods              []                 [dark-blue-app]  [get watch create delete]
  deployments.apps  []                 []               [get watch create delete]

• Omitting resourceNames = rule applies to all objects of that type in the namespace.
• The error message tells you exactly what's missing: verb, resource, API group and namespace. Turn those straight into a rule.

[!CAUTION] Correction: the lab says that with resourceNames, create still works if the new object's name matches the list. In Kubernetes, create (and deletecollection) cannot be restricted by resourceNames: the name isn't checked at authorization time, so a rule with resourceNames never grants create. To allow creating pods, put create in a separate rule without resourceNames.

[!NOTE] Beyond the course: the lab's manifest shows image k8s.gcr.io/kube-apiserver:v1.23.0. Current clusters pull from registry.k8s.io.

# /etc/kubernetes/manifests/kube-apiserver.yaml (excerpt)
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true

7.18 Cluster RolesCopy Anchor Link

Source: KodeKloud notes

Namespaced vs cluster-scoped resourcesCopy Anchor Link

• Role + RoleBinding (§7.16) are namespaced: created in a namespace (e.g. default), grant access only inside it.
• Cluster-scoped resources belong to no namespace — you can't say "node01 is in namespace dev". Examples: nodes, PersistentVolumes.
• Namespaced (normally managed with Roles): pods, replicasets, jobs, deployments, services, secrets, roles, rolebindings, configmaps, PVCs.
• Cluster-scoped: nodes, PVs, clusterroles, clusterrolebindings, certificatesigningrequests, namespaces.
• ClusterRoles and ClusterRoleBindings are themselves cluster-scoped objects.

kubectl api-resources -o wide                 # NAMESPACED column = true/false
kubectl api-resources --namespaced=false      # only cluster-scoped kinds (Beyond the course: filter flag)
kubectl api-resources --namespaced=true       # only namespaced kinds

ClusterRole + ClusterRoleBindingCopy Anchor Link

• ClusterRole = same structure as a Role (rules: apiGroups / resources / verbs), but effect is cluster-wide.
• Course examples: Cluster Admin → view/create/delete nodes; Storage Admin → view/create PVs, delete PVCs.
• ClusterRoleBinding links subjects (User / Group / ServiceAccount) to a ClusterRole. Both can go in one file separated by ---.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-administrator          # no namespace: cluster-scoped
rules:
- apiGroups: [""]                      # "" = core API group (nodes, pods, PVs...)
  resources: ["nodes"]
  verbs: ["list", "get", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admin-role-binding
subjects:
- kind: User
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-administrator
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f cluster-admin-rolebinding.yaml

ClusterRoles on namespaced resourcesCopy Anchor Link

• A ClusterRole is not limited to cluster-scoped resources; it can list namespaced ones (e.g. pods).
• Bound via ClusterRoleBinding → the subject gets that access in every namespace (vs a Role = one namespace only).
• Least privilege: be deliberate — a ClusterRole on pods + ClusterRoleBinding = pods everywhere.
• Kubernetes creates many default ClusterRoles at cluster setup (e.g. cluster-admin, system:node, plus admin/edit/view — see §7.19). ClusterRoleBindings can reference them; know them when designing RBAC for operators and ServiceAccounts.

[!CAUTION] Correction: the original lists system:masters as a built-in ClusterRole. It is a group: the default cluster-admin ClusterRoleBinding binds the cluster-admin ClusterRole to the group system:masters (see §7.19).

[!NOTE] Beyond the course: a RoleBinding may reference a ClusterRole. The rules then apply only inside the RoleBinding's namespace. This is the usual way to reuse one definition (e.g. the built-in edit or view) in many namespaces. Valid combinations:

Binding	roleRef	Effective scope
RoleBinding	Role (same ns)	that namespace
RoleBinding	ClusterRole	that namespace only
ClusterRoleBinding	ClusterRole	all namespaces + cluster-scoped resources
ClusterRoleBinding	Role	not allowed

roleRef is immutable: to change the referenced role, delete and recreate the binding.

[!WARNING] Exam traps / remember:

• Nodes, PVs, StorageClasses, namespaces and CSRs need a ClusterRole. A Role (even in default) cannot grant them.
• ClusterRole/ClusterRoleBinding manifests have no metadata.namespace.
• ClusterRoleBinding to a ClusterRole that covers pods = pods in all namespaces.

7.19 Lab Solution Cluster RolesCopy Anchor Link

Source: KodeKloud notes

Theory: §7.18 (ClusterRoles), §7.16 (RBAC).

[!NOTE] Beyond the course: view cannot read Secrets. edit cannot view or modify Roles/RoleBindings. admin can manage Roles/RoleBindings in the namespace but not the namespace itself or its ResourceQuota. admin, edit and view are meant to be granted per namespace with a RoleBinding.

Drill: give user michelle node accessCopy Anchor Link

kubectl get nodes --as michelle
# Forbidden: User "michelle" cannot list resource "nodes" in API group "" at the cluster scope

kubectl create clusterrole michelle-role --verb=get,list,watch --resource=nodes
kubectl create clusterrolebinding michelle-role-binding --clusterrole=michelle-role --user=michelle

kubectl describe clusterrole michelle-role             # nodes  [get list watch]
kubectl describe clusterrolebinding michelle-role-binding   # Role: ClusterRole/michelle-role, Subject: User michelle
kubectl get nodes --as michelle                         # now succeeds

Drill: give michelle storage access (PVs + StorageClasses)Copy Anchor Link

kubectl api-resources            # find exact names + API groups:
# persistentvolumes   pv  v1                 false  PersistentVolume
# persistentvolumeclaims pvc v1             true   PersistentVolumeClaim
# storageclasses      sc  storage.k8s.io/v1  false  StorageClass

kubectl create clusterrole storage-admin \
  --resource=persistentvolumes,storageclasses --verb=list,create,get,watch
kubectl get clusterrole storage-admin -o yaml   # verify the generated rules
kubectl create clusterrolebinding michelle-storage-binding --clusterrole=storage-admin --user=michelle
kubectl get storageclasses --as michelle

• kubectl create clusterrole automatically splits the rules by API group: persistentvolumes goes under apiGroups: [""], storageclasses under apiGroups: ["storage.k8s.io"]. If you write YAML by hand, you must set the groups yourself.

# generated rules (trimmed)
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["list", "create", "get", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["list", "create", "get", "watch"]

[!TIP] Verify with --as <user> (impersonation). Beyond the course: kubectl auth can-i list nodes --as michelle answers yes/no without running the real command. kubectl auth can-i --list --as michelle lists everything the user may do.

[!WARNING] Exam traps / remember:

• Do not bind cluster-admin when a small custom ClusterRole will do.
• Resource names in --resource are plural lowercase (persistentvolumes, storageclasses). Short names (pv, sc) are also accepted, but check the generated YAML.
• ClusterRoleBinding subject flags: --user=, --group=, --serviceaccount=<ns>:<name>.

7.20 Service AccountsCopy Anchor Link

Source: KodeKloud notes

Account typesCopy Anchor Link

• Scenario: Python app my-kubernetes-dashboard lists Pods through the API. Running in-cluster, it should authenticate with a ServiceAccount.
• ServiceAccount = namespaced identity object ("ID card"). Its token is presented as a Bearer token ("barcode" that is scanned to check the ID).

Diagram (new): the kubelet mounts a time-bound SA token into the Pod. The app or an external tool sends it as a Bearer token, and the API server authenticates and then checks RBAC.

Default SA and token mountingCopy Anchor Link

• Every namespace automatically has a default ServiceAccount. A Pod without a ServiceAccount set uses default.
• The token is mounted through a projected volume (kube-api-access-xxxxx) at /var/run/secrets/kubernetes.io/serviceaccount.

kubectl get serviceaccount                         # lists SAs in current ns (default always present)
kubectl describe pod my-kubernetes-dashboard       # "Service Account: default"; Mounts + Volumes below
kubectl exec -it my-kubernetes-dashboard -- ls /var/run/secrets/kubernetes.io/serviceaccount
# ca.crt  namespace  token

# describe pod excerpt
Volumes:
  kube-api-access-rtvkd:
    Type:                    Projected
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    DownwardAPI:             true

Create a custom SA and attach itCopy Anchor Link

• The default SA has minimal privileges. Create a dedicated SA and give it only the access it needs with RBAC: Role/ClusterRole + RoleBinding/ClusterRoleBinding (§7.16, §7.18).

kubectl create serviceaccount dashboard-sa      # imperative
kubectl describe serviceaccount dashboard-sa    # Image pull secrets / Mountable secrets / Tokens: <none>

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-sa
  namespace: default
---
apiVersion: v1
kind: Pod
metadata:
  name: my-kubernetes-dashboard
spec:
  serviceAccountName: dashboard-sa     # pod-level field; omitted → "default"
  containers:
  - name: my-kubernetes-dashboard
    image: my-kubernetes-dashboard

• Check: kubectl describe pod my-kubernetes-dashboard shows Service Account: dashboard-sa. The token is mounted at the same path.

Disable auto-mounting: automountServiceAccountToken: falseCopy Anchor Link

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dashboard-sa
  namespace: default
automountServiceAccountToken: false    # top-level field on the SA (not under spec)
---
apiVersion: v1
kind: Pod
metadata:
  name: my-kubernetes-dashboard
spec:
  automountServiceAccountToken: false  # per-Pod; overrides the SA setting
  serviceAccountName: dashboard-sa
  containers:
  - name: my-kubernetes-dashboard
    image: my-kubernetes-dashboard

• Set it to false on the Pod, or on the SA (unless the Pod overrides it), and no token is mounted.

Tokens for external useCopy Anchor Link

kubectl create token dashboard-sa                  # prints a short-lived JWT (default ~1h)
kubectl create token dashboard-sa --duration=2h    # request longer validity

# inspect JWT header + payload
jq -R 'split(".") | .[0], .[1] | @base64d | fromjson' <<< 'eyJhbGciOiJSUz...'
#   payload: "sub": "system:serviceaccount:default:dashboard-sa", "aud", "exp", "iat", "iss", "nbf",
#            "kubernetes.io": {namespace, serviceaccount{name, uid}}

curl --insecure --header "Authorization: Bearer eyJhbGciOiJSUz..." https://192.168.56.70:6443/api

[!NOTE] Beyond the course (v1.24+ behaviour): token Secrets are no longer created automatically for SAs. On new clusters the SECRETS column shows 0 (the course output shows 1 for an old default SA). Pods get projected, time-bound tokens that the kubelet refreshes. If you really need a long-lived token, create a Secret by hand:

Copy Text

apiVersion: v1
kind: Secret
metadata:
  name: dashboard-sa-token
  annotations:
    kubernetes.io/service-account.name: dashboard-sa   # token controller fills in token/ca.crt
type: kubernetes.io/service-account-token

[!WARNING] Exam traps / remember:

• The username of an SA in RBAC errors and JWT sub is system:serviceaccount:<namespace>:<sa-name>.
• RoleBinding subject for an SA: kind: ServiceAccount, name, namespace (no apiGroup). Imperative form: --serviceaccount=<ns>:<name>.
• The field is serviceAccountName. serviceAccount is the deprecated alias you may still see in -o yaml output.
• Beyond the course: you cannot change serviceAccountName of an existing Pod. Delete and recreate it, or edit the Deployment, which rolls out new Pods.
• Mount path to memorise: /var/run/secrets/kubernetes.io/serviceaccount.

7.21 Lab Solution Service AccountsCopy Anchor Link

Source: KodeKloud notes

Theory: §7.20. Scenario: the web-dashboard Deployment cannot list pods; fix it with a dedicated SA and RBAC.

# Step 6: if you must build the RBAC yourself (Beyond the course: imperative equivalent of /var/rbac files)
kubectl create role pod-reader --verb=get,watch,list --resource=pods -n default
kubectl create rolebinding read-pods --role=pod-reader --serviceaccount=default:dashboard-sa -n default
kubectl auth can-i list pods --as=system:serviceaccount:default:dashboard-sa -n default   # yes

# Step 8: course method
kubectl get deployment web-dashboard -n default -o yaml > dashboard.yaml
# edit: add serviceAccountName under spec.template.spec
kubectl apply -f dashboard.yaml

spec:
  template:
    spec:
      serviceAccountName: dashboard-sa   # pod template spec, NOT the Deployment's top-level spec

[!NOTE] Beyond the course: to set it in one command, kubectl set serviceaccount deployment web-dashboard dashboard-sa. Or use kubectl edit deploy web-dashboard. Both trigger a rollout.

[!WARNING] Exam traps / remember:

• serviceAccountName goes under spec.template.spec. At the Deployment's top-level spec it is wrong.
• kubectl apply on a Deployment originally made with kubectl create warns about a missing last-applied-configuration annotation. This is harmless.
• A pasted token only tests the permissions. The real fix is the Deployment SA, so new Pods mount the token automatically.

7.22 Image SecurityCopy Anchor Link

Source: KodeKloud notes

Image name formatCopy Anchor Link

[registry-host[:port]/][namespace/]repository[:tag | @sha256:<digest>]

• nginx = docker.io/library/nginx:latest
• myorg/webapp:1.0 = docker.io/myorg/webapp:1.0
• gcr.io/my-project/my-image:stable = pulled from Google Container Registry.
• If you leave parts out, the defaults apply (Docker Hub + library + latest). Writing full names makes pulls reproducible and safer.

• Use a private registry with access control for proprietary or internal images.

[!NOTE] Beyond the course: Google Container Registry has been replaced by Artifact Registry (*-docker.pkg.dev). Kubernetes images now come from registry.k8s.io (the old k8s.gcr.io is frozen).

Private registry: Docker vs KubernetesCopy Anchor Link

# Docker (local): credentials are stored in the local Docker config
docker login private-registry.io
docker run private-registry.io/apps/internal-app:latest

• In Kubernetes, the kubelet / container runtime on the node pulls the image, so it needs the credentials. Store them in a Secret of type kubernetes.io/dockerconfigjson and reference it in the Pod with imagePullSecrets.

kubectl create secret docker-registry regcred \
  --docker-server=private-registry.io \
  --docker-username=registry-user \
  --docker-password=registry-password \
  --docker-email=registry-user@org.com

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
  - name: nginx
    image: private-registry.io/apps/internal-app:latest
  imagePullSecrets:          # Pod-level (sibling of containers), list of Secret names
  - name: regcred

Best practicesCopy Anchor Link

• Use fully qualified names (registry + namespace + tag or digest), so nothing is pulled by accident from a public registry.
• In production, pin images by digest (@sha256:...) so the content cannot change.
• Restrict registry access with strong credentials or IAM (AWS IAM for ECR, GCP IAM for GCR).
• Scan images for vulnerabilities in CI/CD before deploying.
• Keep pull Secrets in the namespace that uses them, with least privilege.

[!NOTE] Beyond the course: add imagePullSecrets to a ServiceAccount and every Pod created afterwards using that SA inherits it (recreate existing Pods, e.g. kubectl rollout restart deploy/<name>): kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"regcred"}]}'

[!WARNING] Exam traps / remember:

• The Secret type in the command is docker-registry (the object's type becomes kubernetes.io/dockerconfigjson). An Opaque Secret will not work for pulls. generic --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=<config.json> also creates a correctly typed pull Secret.
• The Secret must be in the same namespace as the Pod.
• imagePullSecrets goes under the Pod spec, not inside a container.

7.23 Lab Solution Image SecurityCopy Anchor Link

Source: KodeKloud notes

Theory: §7.22. Steps: point Deployment web at the private registry, see the pull fail, create the Secret, add imagePullSecrets.

kubectl set image deploy/web nginx=myprivateregistry.com:5000/nginx:alpine   # Beyond the course: alternative to edit

kubectl create secret docker-registry private-reg-cred \
  --docker-server=myprivateregistry.com:5000 \
  --docker-username=dock_user \
  --docker-password=dock_password \
  --docker-email=dock_user@myprivateregistry.com

# kubectl edit deploy web
spec:
  template:
    spec:
      imagePullSecrets:              # pod spec level
      - name: private-reg-cred       # must match the Secret name exactly
      containers:
      - name: nginx
        image: myprivateregistry.com:5000/nginx:alpine
        imagePullPolicy: IfNotPresent

[!WARNING] Exam traps / remember:

• Put the registry host:port both in the image name and in --docker-server (myprivateregistry.com:5000).
• The path for Deployments is .spec.template.spec.imagePullSecrets. It is not under containers[].
• Secrets are namespaced, so create the Secret in the Deployment's namespace.
• Never commit registry Secrets or credentials to Git in plaintext. Use a secrets manager, encrypted manifests or a CI/CD secret store.

7.24 Prerequisite Security in DockerCopy Anchor Link

Source: KodeKloud notes

Isolation with namespacesCopy Anchor Link

• Containers share the host kernel; there is no guest kernel. Isolation comes from Linux namespaces: PID, mount, network, IPC, UTS. Docker normally creates these per container; Kubernetes shares some at Pod scope (e.g. network). User namespaces are opt-in.
• A container process is a real host process. It has a different PID in the host's PID namespace.

docker run -d --name sleeper ubuntu sleep 3600
docker exec -it sleeper ps aux     # inside: sleep 3600 is PID 1, user root
ps aux | grep sleep                # on host: same process, e.g. PID 3816, user root

Users inside containersCopy Anchor Link

• An image has its own /etc/passwd and /etc/group, but by default Docker runs the process as UID 0 (root).
• To run as non-root (recommended, limits the damage if a container is compromised):

docker run -d --name sleeper-uid --user=1001 ubuntu sleep 3600   # runtime override
docker exec -it sleeper-uid ps aux                               # USER shows 1001 (no passwd entry → numeric)

FROM ubuntu
USER 1000          # baked into the image

docker build -t my-ubuntu-image .
docker run -d --name my-sleeper my-ubuntu-image sleep 3600
docker exec -it my-sleeper ps aux          # runs as UID 1000

• If the UID has no entry in the image's /etc/passwd, tools show the number instead of a name. Add the entries if the app needs usernames.

Is container root = host root? No, it is restricted by:Copy Anchor Link

• Linux capabilities: root's power is split into fine-grained privileges (e.g. CAP_NET_BIND_SERVICE, CAP_SYS_TIME, CAP_SYS_ADMIN). Docker drops many high-risk capabilities by default. Full list: man 7 capabilities.
• seccomp: syscall filtering; dangerous syscalls are blocked by default.
• LSMs: AppArmor / SELinux profiles, where available.

docker run --cap-add=NET_ADMIN ubuntu sleep 3600
docker run --cap-drop=SYS_TIME ubuntu sleep 3600
docker run --privileged ubuntu sleep 3600          # avoid

• Kubernetes uses the same kernel primitives. Its securityContext (§7.25) maps to these Docker options and adds cluster-level policy on top.

[!NOTE] Beyond the course: current Kubernetes nodes run containerd or CRI-O (no dockershim since v1.24). The concepts are the same, but on nodes you inspect containers with crictl, not docker.

[!WARNING] Exam traps / remember: prefer specific --cap-add / --cap-drop over --privileged. The default user is root (UID 0) unless --user or USER sets another.

[!CAUTION] Correction: user-namespace isolation is opt-in (Docker userns-remap; Kubernetes spec.hostUsers: false). Without it, container UID 0 maps to host UID 0, with capabilities and other isolation still limiting access.

[!NOTE] Beyond the course: unlike Docker’s default, Kubernetes requires securityContext.seccompProfile.type: RuntimeDefault (or the kubelet’s seccomp-default setting) to use the runtime default seccomp profile. Pod Security Standards restricted requires an allowed profile. See seccomp.

7.25 Security ContextsCopy Anchor Link

Source: KodeKloud notes

Pod vs container levelCopy Anchor Link

• securityContext is the Kubernetes way to set the UID and capabilities that you set with Docker flags in §7.24.
• You can set it at the Pod level (applies to all containers) or at the container level. For the same field, the container value overrides the Pod value.

# 1) Pod-level: applies to every container
apiVersion: v1
kind: Pod
metadata:
  name: secctx-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["sleep", "3600"]