Kubernetes and Cloud Native Associate (KCNA)

KCNA is an entry-level certification program for anyone interested in cloud-native application development, runtime environments, and tooling. While the exam does cover Kubernetes, it does not expect you to actually solve problems in a practical manner. This exam is suitable for candidates interested in the topic with a broad exposure to the ecosystem.

Kubernetes and Cloud Native Security Associate (KCSA)

The certification focuses on basic knowledge of security concepts and their application in a Kubernetes cluster. The breadth and depth of the program is comparable to the KCNA, as it does not require solving problems hands-on.

Certified Kubernetes Application Developer (CKAD)

The CKAD exam focuses on verifying your ability to build, configure, and deploy a microservices-based application to Kubernetes. You are not expected to actually implement an application; however, the exam is suitable for developers familiar with topics like application architecture, runtimes, and programming languages.

Certified Kubernetes Administrator (CKA)

The target audience for the CKA exam are DevOps practitioners, system administrators, and site reliability engineers. This exam tests your ability to perform in the role of a Kubernetes administrator, which includes tasks like cluster, network, storage, and beginner-level security management, with a big emphasis on troubleshooting scenarios.

Certified Kubernetes Security Specialist (CKS)

The CKS exam expands on the topics verified by the CKA exam. Passing the CKA is a prerequisite before you can even sign up for the CKS exam. For this certification, you are expected to have a deeper knowledge of Kubernetes security aspects. The curriculum covers topics like applying best practices for building containerized applications and ensuring a secure Kubernetes runtime environment.

Cluster Setup

This section covers Kubernetes concepts that have already been covered by the CKA exam; however, they assume that you already understand the basics and expect you to be able to go deeper. Here, you will be tested on network policies and their effects on disallowing and granting network communication between Pods within the same namespace and across multiple namespaces. The main focus will be on restricting communication to minimize the attack surface. Furthermore, the domain “cluster setup” will verify your knowledge of setting up an Ingress object with Transport Layer Security (TLS) termination.

A big emphasis lies on identifying and fixing security vulnerabilities by inspecting the cluster setup. External tools like kube-bench can help with automating the process. As a result of executing the tool against your cluster, you will receive an actionable list of vulnerabilities. Changing the configuration settings of your cluster according to the recommendations can help with significantly reducing the security risk.

Last, locking down cluster node endpoints, ports, and graphical user interfaces (GUIs) can help with making it harder for attackers to gain control of the cluster. You need to be aware of the default cluster settings so that you can limit access to them as much as possible. Kubernetes binaries and executables like kubectl, kubeadm, and the kubelet need to be checked against their checksum to ensure they haven’t been tampered with by a third party. You need to understand how to retrieve the checksum file for binaries and how to use it verify the validity of the executable.

Cluster Hardening

Most organizations start out with a cluster that allows developers and administrators alike to manage the Kubernetes installation, configuration, and management of any objects. While this is a convenient approach for teams getting comfortable with Kubernetes, it is not a safe and sound situation, as it poses the potential of opening the floodgates for attackers. Once access has been gained to the cluster, any malicious operation can be performed.

Role-based access control (RBAC) maps permissions to users or processes. The exam requires deep knowledge of the API resources involved. The domain “cluster hardening” also focuses the topic of keeping the cluster version up-to-date to ensure the latest bug fixes are picked up. Kubernetes exposes the API server via endpoints. You should be aware of strategies for minimizing its exposure to the outside world.

System Hardening

This domain is all about understanding how to minimize access to the host system and external network to reduce the attack surface. This is where OS-level tools like AppArmor and seccomp come into play. You will need to demonstrate their use to fulfill the requirement. The domain also touches on the use of AWS IAM roles for clusters running in Amazon’s cloud environment specifically.

Minimize Microservice Vulnerabilities

Security contexts define privilege and access control for containers. Platform and security teams can govern and enforce desired security measures on the organizational level. The exam requires you to understand Pod security policies and the OPA Gatekeeper for that purpose. Moreover, you’ll be asked to demonstrate defining Secrets of different types and consuming them from Pods to inject sensitive runtime information.

Sometimes, you may want to experiment with container images from an unverified source or a potentially unsafe origin. Container runtime sandboxes like gVisor and Kata Containers can be configured in Kubernetes to execute a container image with very restricted permissions. Configuring and using such a container runtime sandbox is part of this domain. Further, you need to be aware of the benefits of mTLS Pod-to-Pod encryption and how to configure it.

Supply Chain Security

Container security starts with the base image. You need to be aware of the best practices for building container images that minimize the risk of introducing security vulnerabilities from the get-go. Optimally, you will only allow pulling trusted container images from an organization-internal container registry that has already scanned the image for vulnerabilities before it can be used. Allowing only those registries is paramount and will be one of the topics important to this domain. Tools like Trivy can help with the task of scanning images for vulnerabilities and are listed as a requirement to pass the exam.

Monitoring, Logging, and Runtime Security

One of the focus points of this domain is behavior analytics, the process of observing abnormal and malicious events. Falco is the primary tool to get familiar with in this section. A container should not be mutable after it has been started to avoid opening additional backdoors for attackers. You will need to be aware of best practices and demonstrate the ability to apply them in the configuration of a container.

Audit logging can be helpful for a real-time view on cluster events or for debugging purposes. Configuring audit logging for a Kubernetes cluster is part of the exam.

Scenario: Attacker Gains Access to a Pod

Say you are working for a company that operates a Kubernetes cluster with three worker nodes. Worker node 1 currently runs two Pods as part of a microservices architecture. Given Kubernetes default behavior for Pod-to-Pod network communication, Pod 1 can talk to Pod 2 unrestrictedly and vice versa.

As you can see in Figure 2-1, an attacker gained access to Pod 1. Without defining network policies, the attacker can simply talk to Pod 2 and cause additional damage. This vulnerability isn’t restricted to a single namespace. Pods 3 and 4 can be reached and compromised as well.

Figure 2-1. An attacker who gained access to Pod 1 has network access to other Pods

Observing the Default Behavior

We’ll set up three Pods to demonstrate the unrestricted Pod-to-Pod network communication in practice. As you can see in Example 2-1, the YAML manifest defines the Pods named backend and frontend in the namespace g04. The other Pod lives in the default namespace. Observe the label assignment for the namespace and Pods. We will reference them a little bit later in this chapter when defining network policies.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: orion
  name: g04
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    tier: backend
  name: backend
  namespace: g04
spec:
  containers:
  - image: bmuschko/nodejs-hello-world:1.0.0
    name: hello
    ports:
    - containerPort: 3000
  restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    tier: frontend
  name: frontend
  namespace: g04
spec:
  containers:
  - image: alpine
    name: frontend
    args:
    - /bin/sh
    - -c
    - while true; do sleep 5; done;
  restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    tier: outside
  name: other
spec:
  containers:
  - image: alpine
    name: other
    args:
    - /bin/sh
    - -c
    - while true; do sleep 5; done;
  restartPolicy: Never

Start by creating the objects from the existing YAML manifest using the declarative kubectl apply command:

$ kubectl apply -f setup.yaml
namespace/g04 created
pod/backend created
pod/frontend created
pod/other created

Let’s verify that the namespace g04 runs the correct Pods. Use the -o wide CLI option to determine the virtual IP addresses assigned to the Pods. The backend Pod uses the IP address 10.0.0.43, and the frontend Pod uses the IP address 10.0.0.193:

$ kubectl get pods -n g04 -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE     \
  NOMINATED NODE   READINESS GATES
backend    1/1     Running   0          15s   10.0.0.43    minikube \
  <none>           <none>
frontend   1/1     Running   0          15s   10.0.0.193   minikube \
  <none>           <none>

The default namespace handles a single Pod:

$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
other   1/1     Running   0          4h45m

The frontend Pod can talk to the backend Pod as no communication restrictions have been put in place:

$ kubectl exec frontend -it -n g04 -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
remote file exists
/ # exit

The other Pod residing in the default namespace can communicate with the backend Pod without problems:

$ kubectl exec other -it -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
remote file exists
/ # exit

In the next section, we’ll talk about restricting Pod-to-Pod network communication to a maximum level with the help of deny-all network policy rules. We’ll then open up ingress and/or egress communication only for the kind of network communication required for the microservices architecture to function properly.

Denying Directional Network Traffic

The best way to restrict Pod-to-Pod network traffic is with the principle of least privilege. Least privilege means that Pods should communicate with the lowest privilege for network communication. You’d usually start by disallowing traffic in any direction and then opening up the traffic needed by the application architecture.

The Kubernetes documentation provides a couple of helpful YAML manifest examples. Example 2-2 shows a network policy that denies ingress traffic to all Pods in the namespace g04.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: g04
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Selecting all Pods is denoted by the value {} assigned to the spec.podSelector attribute. The value attribute spec.policyTypes defines the denied direction of traffic. For incoming traffic, you can add Ingress to the array. Outgoing traffic can be specified by the value Egress. In this particular example, we disallow all ingress traffic. Egress traffic is still permitted.

The contents of the “deny-all” network policy have been saved in the file deny-all-ingress-network-policy.yaml. The following command creates the object from the file:

$ kubectl apply -f deny-all-ingress-network-policy.yaml
networkpolicy.networking.k8s.io/default-deny-ingress created

Let’s see how this changed the runtime behavior for Pod-to-Pod network communication. The frontend Pod cannot talk to the backend Pod anymore, as observed by running the same wget command we used earlier. The network call times out after one second, as defined by the CLI option --timeout:

$ kubectl exec frontend -it -n g04 -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
wget: download timed out
/ # exit

Furthermore, Pods running in a different namespace cannot connect to the backend Pod anymore either. The following wget command makes a call from the other Pod running in the default namespace to the IP address of the backend Pod:

$ kubectl exec other -it -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
wget: download timed out

This call times out as well.

Allowing Fine-Grained Incoming Traffic

Network policies are additive. To grant more permissions for network communication, simply create another network policy with more fine-grained rules. Say we wanted to allow ingress traffic to the backend Pod only from the frontend Pod that lives in the same namespace. Ingress traffic from all other Pods should be denied independently of the namespace they are running in.

Network policies heavily work with label selection to define rules. Identify the labels of the g04 namespace and the Pod objects running in the same namespace so we can use them in the network policy:

$ kubectl get ns g04 --show-labels
NAME   STATUS   AGE   LABELS
g04    Active   12m   app=orion,kubernetes.io/metadata.name=g04
$ kubectl get pods -n g04 --show-labels
NAME       READY   STATUS    RESTARTS   AGE     LABELS
backend    1/1     Running   0          9m46s   tier=backend
frontend   1/1     Running   0          9m46s   tier=frontend

The label assignment for the namespace g04 includes the key-value pair app=orion. The Pod backend label set includes the key-value pair tier=backend, and the frontend Pod the key-value pair tier=frontend.

Create a new network policy that allows the frontend Pod to talk to the backend Pod only on port 3000. No other communication should be allowed. The YAML manifest representation in Example 2-3 shows the full network policy definition.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-ingress
  namespace: g04
spec:
  podSelector:
    matchLabels:
      tier: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app: orion
      podSelector:
        matchLabels:
          tier: frontend
    ports:
    - protocol: TCP
      port: 3000

The definition of the network policy has been stored in the file backend-ingress-network-policy.yaml. Create the object from the file:

$ kubectl apply -f backend-ingress-network-policy.yaml
networkpolicy.networking.k8s.io/backend-ingress created

The frontend Pod can now talk to the backend Pod:

$ kubectl exec frontend -it -n g04 -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
remote file exists
/ # exit

Pods running outside of the g04 namespace still can’t connect to the backend Pod. The wget command times out:

$ kubectl exec other -it -- /bin/sh
/ # wget --spider --timeout=1 10.0.0.43:3000
Connecting to 10.0.0.43:3000 (10.0.0.43:3000)
wget: download timed out

Using kube-bench

You can use the tool kube-bench to check Kubernetes cluster components against the CIS Benchmark best practices in an automated fashion. Kube-bench can be executed in a variety of ways. For example, you can install it as a platform-specific binary in the form of an RPM or Debian file. The most convenient and direct way to run the verification process is by running kube-bench in a Pod directly on the Kubernetes cluster. For that purpose, create a Job object with the help of a YAML manifest checked into the GitHub repository of the tool.

Start by creating the Job from the file job-master.yaml, or job-node.yaml depending on whether you want to inspect a control plane node or a worker node. The following command runs the verification checks against the control plane node:

$ kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/\
main/job-master.yaml
job.batch/kube-bench-master created

Upon Job execution, the corresponding Pod running the verification process can be identified by its name in the default namespace. The Pod’s name starts with the prefix kube-bench, then appended with the type of the node plus a hash at the end. The following output uses the Pod named kube-bench-master-8f6qh:

$ kubectl get pods
NAME                      READY   STATUS      RESTARTS   AGE
kube-bench-master-8f6qh   0/1     Completed   0          45s

Wait until the Pod transitions into the “Completed” status to ensure that all verification checks have finished. You can have a look at the benchmark result by dumping the logs of the Pod:

$ kubectl logs kube-bench-master-8f6qh

Sometimes, it may be more convenient to write the verification results to a file. You can redirect the output of the kubectl logs command to a file, e.g., with the command kubectl logs kube-bench-master-8f6qh > control-plane-kube-bench-results.txt.

The kube-bench Verification Result

The produced verification result can be lengthy and detailed, but it consists of these key elements: the type of the inspected node, the inspected components, a list of passed checks, a list of failed checks, a list of warnings, and a high-level summary:

[INFO] 1 Control Plane Security Configuration 
[INFO] 1.1 Control Plane Node Configuration Files
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are \
set to 644 or more restrictive (Automated) 
...
[INFO] 1.2 API Server
[WARN] 1.2.1 Ensure that the --anonymous-auth argument is set to false \
(Manual) 
...
[FAIL] 1.2.6 Ensure that the --kubelet-certificate-authority argument is set \
as appropriate (Automated) 

== Remediations master ==
...
1.2.1 Edit the API server pod specification file /etc/kubernetes/manifests/ \
kube-apiserver.yaml on the control plane node and set the below parameter.
--anonymous-auth=false
...
1.2.6 Follow the Kubernetes documentation and setup the TLS connection between 
the apiserver and kubelets. Then, edit the API server pod specification file 
/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and \ 
set the --kubelet-certificate-authority parameter to the path to the cert \ 
file for the certificate authority. 
--kubelet-certificate-authority=<ca-string> 

...
== Summary total == 
42 checks PASS
9 checks FAIL
11 checks WARN
0 checks INFO

The inspected node, in this case the control plane node.

A passed check. Here, the file permissions of the API server configuration file.

A warning message that prompts you to manually check the value of an argument provided to the API server executable.

A failed check. For example, the flag --kubelet-certificate-authority should be set for the API server executable.

The remediation action to take to fix a problem. The number, e.g., 1.2.1, of the failure or warning corresponds to the number assigned to the remediation action.

The summary of all passed and failed checks plus warning and informational messages.

Fixing Detected Security Issues

The list of reported warnings and failures can be a bit overwhelming at first. Keep in mind that you do not have to fix them all at once. Some checks are merely guidelines or prompts to verify an assigned value for a configuration. The following steps walk you through the process of eliminating a warning message.

The configuration files of the control plane components can be found in the directory /etc/kubernetes/manifests on the host system of the control plane node. Say you wanted to fix the warning 1.2.12 reported by kube-bench:

[INFO] 1.2 API Server
...
[WARN] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is \
set (Manual)

== Remediations master ==
...
1.2.12 Edit the API server pod specification file /etc/kubernetes/manifests/ \
kube-apiserver.yaml
on the control plane node and set the --enable-admission-plugins parameter \
to include AlwaysPullImages.
--enable-admission-plugins=...,AlwaysPullImages,...

As proposed by the remediation action, you are supposed to edit the configuration file for the API server and add the value AlwaysPullImages to the list of admission plugins. Go ahead and edit the file kube-apiserver.yaml:

$ sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml

After appending the value AlwaysPullImages to the argument --enable-admission-plugins, the result could look as follows:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: \
    192.168.56.10:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.56.10
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,AlwaysPullImages
...

Save the changes to the file. The Pod running the API server in the kube-system namespace will be restarted automatically. The startup process can take a couple of seconds. Therefore, executing the following command may take a while to succeed:

$ kubectl get pods -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
...
kube-apiserver-control-plane   1/1     Running   0          71m
...

You will need to delete the existing Job object before you can verify the changed result:

$ kubectl delete job kube-bench-master
job.batch "kube-bench-master" deleted

The verification check 1.2.12 now reports a passed result:

$ kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/\
main/job-master.yaml
job.batch/kube-bench-master created
$ kubectl get pods
NAME                      READY   STATUS      RESTARTS   AGE
kube-bench-master-5gjdn   0/1     Completed   0          10s
$ kubectl logs kube-bench-master-5gjdn | grep 1.2.12
[PASS] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is \
set (Manual)

Setting Up the Ingress Backend

In the context of an Ingress, a backend is the combination of Service name and port. Before creating the Ingress, we’ll take care of the Service, a Deployment, and the Pods running nginx so we can later on demonstrate the routing of HTTPS traffic to an actual application. All of those objects are supposed to exist in the namespace t75. Example 2-4 defines all of those resources in a single YAML manifest file setup.yaml as a means to quickly create the Ingress backend.

apiVersion: v1
kind: Namespace
metadata:
  name: t75
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: t75
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: accounting-service
  namespace: t75
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Create the objects from the YAML file with the following command:

$ kubectl apply -f setup.yaml
namespace/t75 created
deployment.apps/nginx-deployment created
service/accounting-service created

Let’s quickly verify that the objects have been created properly, and the Pods have transitioned into the “Running” status. Upon executing the get all command, you should see a Deployment named nginx-deployment that controls three replicas, and a Service named accounting-service of type ClusterIP:

$ kubectl get all -n t75
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-6595874d85-5rdrh   1/1     Running   0          108s
pod/nginx-deployment-6595874d85-jmhvh   1/1     Running   0          108s
pod/nginx-deployment-6595874d85-vtwxp   1/1     Running   0          108s

NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S) \
  AGE
service/accounting-service   ClusterIP   10.97.101.228   <none>        80/TCP \
  108s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           108s

Calling the Service endpoint from another Pod running on the same node should result in a successful response from the nginx Pod. Here, we are using the wget command to verify the behavior:

$ kubectl run tmp --image=busybox --restart=Never -it --rm \
  -- wget 10.97.101.228:80
Connecting to 10.97.101.228:80 (10.97.101.228:80)
saving to 'index.html'
index.html           100% |**|   612  0:00:00 ETA
'index.html' saved
pod "tmp" deleted

With those objects in place and functioning as expected, we can now concentrate on creating an Ingress with TLS termination.

Creating the TLS Certificate and Key

We will need to generate a TLS certificate and key before we can create a TLS Secret. To do this, we will use the OpenSSL command. The resulting files are named accounting.crt and accounting.key:

$ openssl req -nodes -new -x509 -keyout accounting.key -out accounting.crt \
  -subj "/CN=accounting.tls"
Generating a 2048 bit RSA private key
...........................+
..........................+
writing new private key to 'accounting.key'
-----
$ ls
accounting.crt accounting.key

For use in production environments, you’d generate a key file and use it to obtain a TLS certificate from a certificate authority (CA). For more information on creating a TLS certification and key, see the OpenSSL documentation.

Creating the TLS-Typed Secret

The easiest way to create a Secret is with the help of an imperative command. This method of creation doesn’t require you to manually base64-encode the certificate and key values. The encoding happens automatically upon object creation. The following command uses the Secret option tls and assigns the certificate and key file name with the options --cert and --key:

$ kubectl create secret tls accounting-secret --cert=accounting.crt \
  --key=accounting.key -n t75
secret/accounting-secret created

Example 2-5 shows the YAML representation of a TLS Secret if you want to create the object declaratively.

apiVersion: v1
kind: Secret
metadata:
  name: accounting-secret
  namespace: t75
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk...
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk...

Make sure to assign the values for the attributes tls.crt and tls.key as single-line, base64-encoded values. To produce the base64-encoded value, simply point the base64 command to the file name you want to convert the contents for. The following example base64-encoded the contents of the file accounting.crt:

$ base64 accounting.crt
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNyakNDQ...

Creating the Ingress

You can use the imperative method to create the Ingress with the help of a one-liner command shown in the following snippet. Crafting the value of the --rule argument is hard to get right. You will likely have to refer to the --help option for the create ingress command as it requires a specific expression. The information relevant to creating the connection between Ingress object and the TLS Secret is the appended argument tls=accounting-secret:

$ kubectl create ingress accounting-ingress \
  --rule="accounting.internal.acme.com/*=accounting-service:80, \
  tls=accounting-secret" -n t75
ingress.networking.k8s.io/accounting-ingress created

Example 2-6 shows a YAML representation of an Ingress. The attribute for defining the TLS information is spec.tls[].

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: accounting-ingress
  namespace: t75
spec:
  tls:
  - hosts:
    - accounting.internal.acme.com
    secretName: accounting-secret
  rules:
  - host: accounting.internal.acme.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: accounting-service
            port:
              number: 80

After creating the Ingress object with the imperative or declarative approach, you should be able to find it in the namespace t75. As you can see in the following output, the port 443 is listed in the “PORT” column, indicating that TLS termination has been enabled:

$ kubectl get ingress -n t75
NAME                 CLASS   HOSTS                          ADDRESS       \
  PORTS     AGE
accounting-ingress   nginx   accounting.internal.acme.com   192.168.64.91 \
  80, 443   55s

Describing the Ingress object shows that the backend could be mapped to the path / and will route traffic to the Pod via the Service named accounting-service:

$ kubectl describe ingress accounting-ingress -n t75
Name:             accounting-ingress
Labels:           <none>
Namespace:        t75
Address:          192.168.64.91
Ingress Class:    nginx
Default backend:  <default>
TLS:
  accounting-secret terminates accounting.internal.acme.com
Rules:
  Host                          Path  Backends
  ----                          ----  --------
  accounting.internal.acme.com
                                /   accounting-service:80 \
                                (172.17.0.5:80,172.17.0.6:80,172.17.0.7:80)
Annotations:                    <none>
Events:
  Type    Reason  Age               From                      Message
  ----    ------  ----              ----                      -------
  Normal  Sync    1s (x2 over 31s)  nginx-ingress-controller  Scheduled for sync

Calling the Ingress

To test the behavior on a local Kubernetes cluster on your machine, you need to first find out the IP address of a node. The following command reveals the IP address in a minikube environment:

$ kubectl get nodes -o wide
NAME       STATUS   ROLES           AGE     VERSION   INTERNAL-IP   \
  EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    control-plane   3d19h   v1.24.1   192.168.64.91 \
  <none>        Buildroot 2021.02.12   5.10.57          docker://20.10.16

Next, you’ll need to add the IP address to the hostname mapping to your /etc/hosts file:

$ sudo vim /etc/hosts
...
192.168.64.91   accounting.internal.acme.com

You can now send HTTPS requests to the Ingress using the assigned domain name and receive an HTTP response code 200 in return:

$ wget -O- https://accounting.internal.acme.com --no-check-certificate
--2022-07-28 15:32:43--  https://accounting.internal.acme.com/
Resolving accounting.internal.acme.com (accounting.internal.acme.com)... \
192.168.64.91
Connecting to accounting.internal.acme.com (accounting.internal.acme.com) \
|192.168.64.91|:443... connected.
WARNING: cannot verify accounting.internal.acme.com's certificate, issued \
by ‘CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co’:
  Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
	requested host name ‘accounting.internal.acme.com’.
HTTP request sent, awaiting response... 200 OK

Scenario: A Compromised Pod Can Access the Metadata Server

Figure 2-3 shows an attacker who gained access to a Pod running on a node within a cloud provider Kubernetes cluster.

Figure 2-3. An attacker who gained access to the Pod has access to metadata server

Access to the metadata server has not been restricted in any form. The attacker can retrieve sensitive information, which could open other possibilities of intrusion.

Protecting Metadata Server Access with Network Policies

Let’s pick one of the cloud providers that exposes a metadata endpoint. In AWS, the metadata server can be reached with the IP address 169.254.169.254, as described in the AWS documentation. The endpoints exposed can provide access to EC2 instance metadata. For example, you can retrieve the local IP address of an instance to manage a connection to an external application or to contact the instance with the help of a script. See the corresponding documentation page for calls to those endpoints made with the curl command line tool.

To prevent any Pod in a namespace from reaching the IP address of the metadata server, set up a network policy that allows egress traffic to all IP addresses except 169.254.169.254. Example 2-7 demonstrates a YAML manifest with such a rule set.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress-metadata-server
  namespace: a12
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32

Once the network policy has been created, Pods in the namespace a12 should not be able to reach the metadata endpoints anymore. For detailed examples that use the endpoints via curl, see the relevant AWS documentation.

Scenario: An Attacker Gains Access to the Dashboard Functionality

The Kubernetes Dashboard runs as a Pod inside of the cluster. Installing the Dashboard also creates a Service of type ClusterIP that only allows access to the endpoint from within the cluster. To make the Dashboard accessible to end users, you’d have to expose the Service outside of the cluster. For example, you could switch to a NodePort Service type or stand up an Ingress. Figure 2-4 illustrates the high-level architecture of deploying and accessing the Dashboard.

Figure 2-4. An attacker who gained access to the Dashboard

As soon as you expose the Dashboard to the outside world, attackers can potentially gain access to it. Without the right security settings, objects can be deleted, modified, or used for malicious purposes. The most prominent victim of such an attack was Tesla, which in 2018 fell prey to hackers who gained access to its unprotected Dashboard to mine cryptocurrencies. Since then, newer versions of the Dashboard changed default settings to make it more secure from the get-go.

Installing the Kubernetes Dashboard

Installing the Kubernetes Dashboard is straightforward. You can create the relevant objects with the help of the YAML manifest available in the project’s GitHub repository. The following command installs all necessary objects:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/\
v2.6.0/aio/deploy/recommended.yaml

You can find the objects created by the manifest in the kubernetes-dashboard namespace. Among them are Deployments, Pods, and Services. The following command lists all of them:

$ kubectl get deployments,pods,services -n kubernetes-dashboard
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dashboard-metrics-scraper   1/1     1            1           11m
deployment.apps/kubernetes-dashboard        1/1     1            1           11m

NAME                                             READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-78dbd9dbf5-f8z4x   1/1     Running   0          11m
pod/kubernetes-dashboard-5fd5574d9f-ns7nl        1/1     Running   0          11m

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP \
  PORT(S)    AGE
service/dashboard-metrics-scraper   ClusterIP   10.98.6.37       <none>      \
  8000/TCP   11m
service/kubernetes-dashboard        ClusterIP   10.102.234.158   <none>      \
  80/TCP     11m

Accessing the Kubernetes Dashboard

The kubectl proxy command can help with temporarily creating a proxy that allows you to open the Dashboard in a browser. This functionality is only meant for troubleshooting purposes and is not geared toward production environments. You can find information about the proxy command in the documentation:

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

Open the browser with the URL http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy. The Dashboard will ask you to provide an authentication method and credentials. The recommended way to configure the Dashboard is through bearer tokens.

Creating a User with Administration Privileges

Before you can authenticate in the login screen, you need to create a ServiceAccount and ClusterRoleBinding object that grant admin permissions. Start by creating the file admin-user-serviceaccount.yaml and populate it with the contents shown in Example 2-8.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

Next, store the contents of Example 2-9 in the file admin-user-⁠clusterrole​bind⁠ing.yaml to map the ClusterRole named cluster-admin to the ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

Create both objects with the following declarative command:

$ kubectl create -f admin-user-serviceaccount.yaml
serviceaccount/admin-user created
$ kubectl create -f admin-user-clusterrolebinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin-user created

You can now create the bearer token of the admin user with the following command. The command will generate a token for the provided ServiceAccount object and render it on the console:

$ kubectl create token admin-user -n kubernetes-dashboard
eyJhbGciOiJSUzI1NiIsImtpZCI6...

Copy the output of the command and paste it into the “Enter token” field of the login screen, as shown in Figure 2-5.

Figure 2-5. The usage of the token in the Dashboard login screen

Pressing the “Sign in” button will bring you to the Dashboard shown in Figure 2-6.

Figure 2-6. The Dashboard view of Pods in a specific namespace

You can now manage end user and cluster objects without any restrictions.

Creating a User with Restricted Privileges

In the previous section, you learned how to create a user with cluster-wide administrative permissions. Most users of the Dashboard only need a restricted set of permissions, though. For example, developers implementing and operating cloud-native applications will likely only need a subset of administrative permissions to perform their tasks on a Kubernetes cluster. Creating a user for the Dashboard with restricted privileges consists of a three-step approach:

1. Create a ServiceAccount object.

2. Create a ClusterRole object that defines the permissions.

3. Create a ClusterRoleBinding that maps the ClusterRole to the ServiceAccount.

As you can see, the process is very similar to the one we went through for the admin user. Step 2 is new, as we need to be specific about which permissions we want to grant. The YAML manifests that follow will model a user working as a developer that should only be allowed read-only permissions (e.g., getting, listing, and watching resources).

Start by creating the file restricted-user-serviceaccount.yaml and populate it with the contents shown in Example 2-10.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: developer-user
  namespace: kubernetes-dashboard

The ClusterRole in Example 2-11 only allows getting, listing, and watching resources. All other operations are not permitted. Store the contents in the file restricted-user-clusterrole.yaml.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  name: cluster-developer
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - '*'
  verbs:
  - get
  - list
  - watch

Last, map the ServiceAccount to the ClusterRole in the file restricted-user-clusterrolebinding.yaml, as shown in Example 2-12.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: developer-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-developer
subjects:
- kind: ServiceAccount
  name: developer-user
  namespace: kubernetes-dashboard

Create all objects with the following declarative command:

$ kubectl create -f restricted-user-serviceaccount.yaml
serviceaccount/restricted-user created
$ kubectl create -f restricted-user-clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/cluster-developer created
$ kubectl create -f restricted-user-clusterrolebinding.yaml
clusterrolebinding.rbac.authorization.k8s.io/developer-user created

Generate the bearer token of the restricted user with the following command:

$ kubectl create token developer-user -n kubernetes-dashboard
eyJhbGciOiJSUzI1NiIsImtpZCI6...

Operations that are not allowed for the logged-in user will not be rendered as disabled options in the GUI. You can still select the option; however, an error message is rendered. Figure 2-7 illustrates the behavior of the Dashboard if you try to delete a Pod via the user that doesn’t have the permissions to perform the operation.

Figure 2-7. An error message rendered when trying to invoke a permitted operation

Avoiding Insecure Configuration Arguments

Securing the Dashboard in production environments involves the usage of execution arguments necessary for properly configuring authentication and authorization. By default, login functionality is enabled and the HTTPS endpoint will be exposed on port 8443. You can provide TLS certificates with the --tls-cert-file and --tls-cert-key command line options if you don’t want them to be auto-generated.

Avoid setting the command line arguments --insecure-port to expose an HTTP endpoint and --enable-insecure-login to enable serving the login page over HTTP instead of HTTPS. Furthermore, make sure you don’t use the option --enable-skip-login as it would allow circumventing an authentication method by simply clicking a Skip button in the login screen.

Scenario: An Attacker Injected Malicious Code into Binary

The executables kubectl and kubeadm are essential for interacting with Kubernetes. kubectl lets you run commands against the API server, e.g., for managing objects. kubeadm is necessary for upgrading cluster nodes from one version to another. Say you are in the process of upgrading the cluster version from 1.23 to 1.24. As part of the process, you will need to upgrade the kubeadm binary as well. The official upgrade documentation is very specific about what commands to use for upgrading the binary.

Say an attacker managed to modify the kubeadm executable for version 1.24 and coaxed you into thinking that you need to download that very binary from a location where the malicious binary was placed. As shown in Figure 2-8, you’d expose yourself to running malicious code every time you invoke the modified kubeadm executable. For example, you may be sending credentials to a server outside of your cluster, which would open new ways to infiltrate your Kubernetes environment.

Figure 2-8. An attacker who injected malicious code into a binary

Verifying a Binary Against Hash

You can verify the validity of a binary with the help of a hash code like MD5 or SHA. Kubernetes publishes SHA256 hash codes for each binary. You should run through a hash validation for individual binaries before using them for the first time. Should the generated hash code not match with the one you downloaded, then there’s something off with the binary. The binary may have been modified by a third party or you didn’t use the hash code for the correct binary type or version.

You can download the corresponding hash code for a binary from https://dl.k8s.io. The full URL for a hash code reflects the version, operating system, and architecture of the binary. The following list shows example URLs for platform binaries compatible with Linux AMD64:

• kubectl: https://dl.k8s.io/v1.26.1/bin/linux/amd64/kubectl.sha256

• kubeadm: https://dl.k8s.io/v1.26.1/bin/linux/amd64/kubeadm.sha256

• kubelet: https://dl.k8s.io/v1.26.1/bin/linux/amd64/kubelet.sha256

• kube-apiserver: https://dl.k8s.io/v1.26.1/bin/linux/amd64/kube-apiserver.sha256

You’ll have to use an operating system-specific hash code validation tool to check the validity of a binary. You may have to install the tool if you do not have it available on your machine yet. The following commands show the usage of the tool for different operating systems, as explained in the Kubernetes documentation:

• Linux: echo "$(cat kubectl.sha256) kubectl" | sha256sum --check

• MacOSX: echo "$(cat kubectl.sha256) kubectl" | shasum -a 256 --check

• Windows with Powershell: $($(CertUtil -hashfile .\kubectl.exe SHA256)[1] -replace " ", "") -eq $(type .\kubectl.exe.sha256)

The following commands demonstrate downloading the kubeadm binary for version 1.26.1 and its corresponding SHA256 hash file:

$ curl -LO "https://dl.k8s.io/v1.26.1/bin/linux/amd64/kubeadm"
$ curl -LO "https://dl.k8s.io/v1.26.1/bin/linux/amd64/kubeadm.sha256"

The validation tool shasum can verify if the checksum matches:

$ echo "$(cat kubeadm.sha256)  kubeadm" | shasum -a 256 --check
kubeadm: OK

The previous command returned with an “OK” message. The binary file wasn’t tampered with. Any other message indicates a potential security risk when executing the binary.

Processing a Request

Figure 3-1 illustrates the stages a request goes through when a call is made to the API server. For reference, you can find more information in the Kubernetes documentation.

Figure 3-1. API server request processing

The first stage of request processing is authentication. Authentication validates the identity of the caller by inspecting the client certificates or bearer tokens. If the bearer token is associated with a service account, then it will be verified here.

The second stage determines if the identity provided in the first stage can access the verb and HTTP path request. Therefore, stage two deals with authorization of the request, which is implemented with the standard Kubernetes RBAC model. Here, we’d ensure that the service account is allowed to list Pods or create a new Service object if that’s what has been requested.

The third stage of request processing deals with admission control. Admission control verifies if the request is well-formed and potentially needs to be modified before the request is processed. An admission control policy could, for example, ensure that the request for creating a Pod includes the definition of a specific label. If it doesn’t define the label, then the request is rejected.

The last stage of the process ensures that the resource included in the request is valid. Request validation can be implemented as part of admission control but doesn’t have to be. For example, this stage ensures that the name of a Service object sticks to the standard Kubernetes naming rules for provided DNS names.

Connecting to the API Server

It’s easy to determine the endpoint for the API server by running the following:

$ kubectl cluster-info
Kubernetes control plane is running at https://172.28.40.5:6443
...

For the given Kubernetes cluster, the API server has been exposed via the URL https://172.28.40.5:6443. Alternatively, you can also have a look at the command line options --advertise-address and --secure-port in the configuration file of the API server to determine the endpoint. You can find the API server configuration file at /etc/kubernetes/manifests/kube-apiserver.yaml.

$ kubectl get service kubernetes
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   32s

$ kubectl get endpoints kubernetes
NAME         ENDPOINTS          AGE
kubernetes   172.28.40.5:6443   4m3s

$ kubectl run kubernetes-envs --image=alpine:3.16.2 -it --rm --restart=Never \
  -- env
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443

$ curl https://172.28.40.5:6443/api/v1/namespaces -k
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "namespaces is forbidden: User \"system:anonymous\" cannot list \
              resource \"namespaces\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "namespaces"
  },
  "code": 403
}

$ kubectl config view --raw
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL... 
    server: https://172.28.132.5:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin 
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL... 
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktL... 

$ echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL... | base64 -d > ca
$ echo LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tL... | base64 -d > kubernetes-admin.crt
$ echo LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktL... | base64 - \
> kubernetes-admin.key

$ curl --cacert ca --cert kubernetes-admin.crt --key kubernetes-admin.key \
  https://172.28.132.5:6443/api/v1/namespaces
{
  "kind": "NamespaceList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "2387"
  },
  "items": [
    ...
  ]
}

Scenario: An Attacker Can Call the API Server from the Internet

Cloud providers sometimes expose the API server to the internet to simplify administrative access. An attacker can try to make an anonymous request to the API server endpoint by declining to provide a client certificate or bearer token. If the attacker is lucky enough to capture user credentials, then an authenticated call can be performed. Depending on the permissions assigned to the user, malicious operations can be executed. Figure 3-2 illustrates an attacker calling the API server from the internet.

Figure 3-2. An attacker calls the API server from the internet

In this chapter, we will have a look at how to restrict access to the API server and how to implement RBAC with limited permissions by example. “Understanding Open Policy Agent (OPA) and Gatekeeper” will review admission control with the help of OPA Gateway.

Restricting User Permissions

We’ve seen that we can use the credentials of the kubernetes-admin user to make calls to the Kubernetes API. This user should be used very sparingly, nor should the credentials be shared with a lot of humans. A lot of damage can be done if the credentials fall into the wrong hands. Reserve this user exclusively for humans in charge of cluster administration.

For other stakeholders of your Kubernetes cluster, you should set up a dedicated user with a limited set of permissions. You may have specific roles in your organization you can map to. For example, you may have a developer role that should be allowed to manage Deployments, Pods, ConfigMaps, Secrets, and Services, but nothing else. To create a new user and assign the relevant RBAC permissions, refer to the Kubernetes documentation. In a nutshell, there are four steps:

1. Create a private key.

2. Create and approve a CertificateSigningRequest.

3. Create a Role and a RoleBinding.

4. Add the user to the kubeconfig file.

We will cover the process in detail, but will come back to the RBAC concept in more detail for a service account in “Minimizing Permissions for a Service Account”.

$ openssl genrsa -out johndoe.key 2048
Generating RSA private key, 2048 bit long modulus
...+
......................................................................+
e is 65537 (0x10001)

$ openssl req -new -key johndoe.key -out johndoe.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:johndoe
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

$ cat johndoe.csr | base64 | tr -d "\n"
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tL...

$ cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: johndoe
spec:
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tL...
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
  - client auth
EOF
certificatesigningrequest.certificates.k8s.io/johndoe created

$ kubectl get csr johndoe
NAME        AGE   SIGNERNAME                            REQUESTOR     \
  REQUESTEDDURATION   CONDITION
johndoe     6s    kubernetes.io/kube-apiserver-client   minikube-user \
  24h                 Pending

$ kubectl certificate approve johndoe
certificatesigningrequest.certificates.k8s.io/johndoe approved
$ kubectl get csr johndoe
NAME        AGE   SIGNERNAME                            REQUESTOR     \
  REQUESTEDDURATION   CONDITION
johndoe     17s   kubernetes.io/kube-apiserver-client   minikube-user \
  24h                 Approved,Issued

$ kubectl get csr johndoe -o jsonpath={.status.certificate}| base64 \
  -d > johndoe.crt

$ kubectl create role developer --verb=create --verb=get --verb=list \
  --verb=update --verb=delete --resource=pods
role.rbac.authorization.k8s.io/developer created

$ kubectl create rolebinding developer-binding-johndoe --role=developer \
  --user=johndoe
rolebinding.rbac.authorization.k8s.io/developer-binding-johndoe created

$ kubectl config set-credentials johndoe --client-key=johndoe.key \
  --client-certificate=johndoe.crt --embed-certs=true
User "johndoe" set.
$ kubectl config set-context johndoe --cluster=minikube --user=johndoe
Context "johndoe" created.

$ kubectl config use-context johndoe
Switched to context "johndoe".

$ kubectl get pods
No resources found in default namespace.

$ kubectl get namespaces
Error from server (Forbidden): namespaces is forbidden: User "johndoe" cannot \
list resource "namespaces" in API group "" at the cluster scope

$ kubectl config use-context minikube
Switched to context "minikube".

Scenario: An Attacker Can Call the API Server from a Service Account

A user represents a real person who commonly interacts with the Kubernetes cluster using the kubectl executable or the UI dashboard. Under rare conditions, applications running inside of a Pod’s container need to interact with the Kubernetes API. A typical example for such a requirement is the package manager Helm. Helm manages Kubernetes resources based on the YAML manifests bundled in a Helm chart. Kubernetes uses a service account to authenticate the Helm service process with the API server through an authentication token. This service account can be assigned to a Pod and mapped to RBAC rules.

An attacker who gains access to the Pod will likely also be able to misuse the service account to make calls to the Kubernetes API, as shown in Figure 3-3.

Figure 3-3. An attacker uses a service account to call the API server

Minimizing Permissions for a Service Account

It’s important to limit the permissions to only those service accounts that are really necessary for the application to function. The next sections will explain how to achieve this to minimize the potential attack surface.

For this scenario to work, you’ll need to create a ServiceAccount object and assign it to the Pod. Service accounts can be tied in with RBAC and assigned a Role and RoleBinding to define what operations they should be allowed to perform.

apiVersion: v1
kind: Namespace
metadata:
  name: k97
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-api
  namespace: k97
---
apiVersion: v1
kind: Pod
metadata:
  name: list-objects
  namespace: k97
spec:
  serviceAccountName: sa-api
  containers:
  - name: pods
    image: alpine/curl:3.14
    command: ['sh', '-c', 'while true; do curl -s -k -m 5 -H \
              "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/ \
              serviceaccount/token)" https://kubernetes.default.svc.cluster. \
              local/api/v1/namespaces/k97/pods; sleep 10; done']
  - name: deployments
    image: alpine/curl:3.14
    command: ['sh', '-c', 'while true; do curl -s -k -m 5 -H \
              "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/ \
              serviceaccount/token)" https://kubernetes.default.svc.cluster. \
              local/apis/apps/v1/namespaces/k97/deployments; sleep 10; done']

$ kubectl apply -f setup.yaml
namespace/k97 created
serviceaccount/sa-api created
pod/list-objects created

$ kubectl logs list-objects -c pods -n k97
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:k97:sa-api\" \
              cannot list resource \"pods\" in API group \"\" in the \
              namespace \"k97\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}
$ kubectl logs list-objects -c deployments -n k97
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "deployments.apps is forbidden: User \
              \"system:serviceaccount:k97:sa-api\" cannot list resource \
              \"deployments\" in API group \"apps\" in the namespace \
              \"k97\"",
  "reason": "Forbidden",
  "details": {
    "group": "apps",
    "kind": "deployments"
  },
  "code": 403
}

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: list-pods-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

$ kubectl apply -f clusterrole.yaml
clusterrole.rbac.authorization.k8s.io/list-pods-clusterrole created

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: serviceaccount-pod-rolebinding
  namespace: k97
subjects:
- kind: ServiceAccount
  name: sa-api
roleRef:
  kind: ClusterRole
  name: list-pods-clusterrole
  apiGroup: rbac.authorization.k8s.io

$ kubectl apply -f rolebinding.yaml
rolebinding.rbac.authorization.k8s.io/serviceaccount-pod-rolebinding created

$ kubectl logs list-objects -c pods -n k97
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "628"
  },
  "items": [
      {
        "metadata": {
          "name": "list-objects",
          "namespace": "k97",
          ...
      }
  ]
}

$ kubectl logs list-objects -c deployments -n k97
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "deployments.apps is forbidden: User \
              \"system:serviceaccount:k97:sa-api\" cannot list resource \
              \"deployments\" in API group \"apps\" in the namespace \
              \"k97\"",
  "reason": "Forbidden",
  "details": {
    "group": "apps",
    "kind": "deployments"
  },
  "code": 403
}

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-api
  namespace: k97
automountServiceAccountToken: false

apiVersion: v1
kind: Pod
metadata:
  name: list-objects
  namespace: k97
spec:
  serviceAccountName: sa-api
  automountServiceAccountToken: false
  ...

$ kubectl create token sa-api
eyJhbGciOiJSUzI1NiIsImtpZCI6IjBtQkJzVWlsQjl...

$ kubectl create token sa-api --duration 10m
eyJhbGciOiJSUzI1NiIsImtpZCI6IjBtQkJzVWlsQjl...

$ kubectl get serviceaccount sa-api -n k97
NAME     SECRETS   AGE
sa-api   0         42m

apiVersion: v1
kind: Secret
metadata:
  name: sa-api-secret
  namespace: k97
  annotations:
    kubernetes.io/service-account.name: sa-api
type: kubernetes.io/service-account-token

$ kubectl create -f secret.yaml
secret/sa-api-secret created

$ kubectl describe secret sa-api-secret -n k97
...
Data
====
ca.crt:     1111 bytes
namespace:  3 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IjBtQkJzVWlsQjl...

Versioning Scheme

It’s up to the cluster administrator to update the Kubernetes version across all nodes on a regular basis. Kubernetes follows the semantic versioning scheme. A Semantic Version consists of a major version, minor version, and patch version. For example, for the Kubernetes version 1.24.3, the major version is 1, the minor version is 24, and the patch version is 3.

Each portion of the version carries a specific meaning. A change to the major version portion indicates a breaking change. Incrementing the minor version portion means that new functionality has been added in a backward-compatible manner. The patch version portion simply fixes a bug.

Release Cadence

You can expect a new minor version release of Kubernetes every three months. The release may include new features and additional bug fixes. Security fixes may be implemented as needed for the latest release of Kubernetes and will be backported to the two minor releases before that. Always staying on top of updating to the latest releases for your own cluster(s) takes quite a bit of people-power. You will need to reserve time for those activities accordingly.

Performing the Upgrade Process

It is recommended to upgrade from a minor version to the next higher one (e.g., from 1.23 to 1.24), or from a patch version to a more recent one (e.g., from 1.24.1 to 1.24.3). Abstain from jumping up multiple minor versions to avoid unexpected side effects.

You can find a full description of the upgrade steps in the official Kubernetes documentation. Figure 3-4 illustrates the upgrade process on a high level.

The cluster version upgrade process is already part of the CKA exam. Given that you have to pass the CKA as a prerequisite, I would assume that you already know how to perform the process. For a detailed description, refer to Certified Kubernetes Administrator (CKA) Study Guide.

Figure 3-4. Process for a cluster version upgrade

Scenario: An Attacker Exploits a Package Vulnerability

Figure 4-1 illustrates an attacker exploiting a vulnerability of a package installed on the system. For example, the application could be the package manager snapd. Assume that the attacker takes advantage of the known vulnerability USN-5292-1 that has the potential of exposing sensitive information to an attacker.

Figure 4-1. An attacker exploits an OS-level vulnerability

The following section will explain how to minimize security risks for services and packages that are not really needed for operating Kubernetes by simply disabling or removing them.

Disabling Services

On Linux, many applications run as services in the background. Services can be managed using the command line tool systemctl. The following systemctl command lists all running services:

$ systemctl | grep running
...
snapd.service   loaded active running   Snap Daemon

One of the services we will not need for operating a cluster node is the package manager snapd. For more details on the service, retrieve the status for it with the status subcommand:

$ systemctl status snapd
● snapd.service - Snap Daemon
     Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor \
     preset: enabled)
     Active: active (running) since Mon 2022-09-19 22:49:56 UTC; 30min ago
TriggeredBy: ● snapd.socket
   Main PID: 704 (snapd)
      Tasks: 12 (limit: 2339)
     Memory: 45.9M
     CGroup: /system.slice/snapd.service
             └─704 /usr/lib/snapd/snapd

You can stop service using the systemctl subcommand stop:

$ sudo systemctl stop snapd
Warning: Stopping snapd.service, but it can still be activated by:
  snapd.socket

Execute the disable subcommand to prevent the service from being started again upon a system restart:

$ sudo systemctl disable snapd
Removed /etc/systemd/system/multi-user.target.wants/snapd.service.

The service has now been stopped and disabled:

$ systemctl status snapd
● snapd.service - Snap Daemon
     Loaded: loaded (/lib/systemd/system/snapd.service; disabled; vendor \
     preset: enabled)
     Active: inactive (dead) since Mon 2022-09-19 23:22:22 UTC; 4min 4s ago
TriggeredBy: ● snapd.socket
   Main PID: 704 (code=exited, status=0/SUCCESS)

Removing Unwanted Packages

Now that the service has been disabled, there’s no more point in keeping the package around. You can remove the package to free up additional disk space and memory. You can use the apt purge command to remove a package and its transitive packages, as demonstrated in the following:

$ sudo apt purge --auto-remove snapd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  snapd* squashfs-tools*
0 upgraded, 0 newly installed, 2 to remove and 116 not upgraded.
After this operation, 147 MB disk space will be freed.
Do you want to continue? [Y/n] y
...

You can use the same command even if the package isn’t controlled by a service. Identify the packages you don’t need and simply remove them. You should end up with a much slimmer footprint of your system.

A potential attacker cannot use the snapd service anymore to exploit the system. You should repeat the process for any unwanted services. As a result, the snapd service ceases to exist on the system:

$ systemctl status snapd
Unit snapd.service could not be found.

Scenario: An Attacker Uses Credentials to Gain File Access

A security breach can lead to stolen user credentials. Gaining access to valid user credentials opens the door for additional attack vectors. Figure 4-2 shows an attacker who could log into a cluster node with stolen user credentials and can now interact with all files and directories with the permissions granted to the user.

Figure 4-2. An attacker uses stolen credentials to access files

It’s recommended to follow the principle of least privilege. Only grant administrative permissions to a limited group of users. All other users should only be allowed to perform operations necessary to perform their jobs.

Understanding User Management

Every user must authenticate to a system to use it. The authenticated user has access to resources based on the assigned permissions. This section will walk you through the primary operations required to manage users.

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
...

$ ps aux | grep bash
root         956  0.0  0.4  22512 19200 pts/0    Ss   17:57   0:00 -bash
root        7064  0.0  0.0   6608  2296 pts/0    S+   18:08   0:00 grep \
--color=auto bash

$ sudo adduser ben
Adding user ‘ben’ ...
Adding new group ‘ben’ (1001) ...
Adding new user ‘ben’ (1001) with group ‘ben’ ...
Creating home directory ‘/home/ben’ ...
Copying files from ‘/etc/skel’ ...
New password:
Retype new password:
...

$ cat /etc/passwd
...
ben:x:1001:1001:,,,:/home/ben:/bin/bash

$ su ben
Password:
ben@controlplane:/root$ pwd
/root

$ su - ben
ben@controlplane:~$ pwd
/home/ben

$ sudo -u ben pwd
/root

$ sudo userdel -r ben

Understanding Group Management

It’s more convenient for a system administrator to group users with similar access requirements to control permissions on an individual user level. Linux systems offer the concept of a group as a way to organize users based on teams, or specific organizational roles. We’ll briefly touch on the most important aspects of group management.

$ cat /etc/group
root:x:0:
plugdev:x:46:packer
nogroup:x:65534:
...

$ sudo groupadd kube-developers

$ cat /etc/group
...
kube-developers:x:1004:

$ sudo usermod -g kube-developers ben

$ cat /etc/passwd | grep ben
ben:x:1001:1004:,,,:/home/ben:/bin/bash

$ sudo groupdel kube-developers
groupdel: cannot remove the primary group of user ben

$ sudo usermod -g kube-admins ben
$ sudo groupdel kube-developers

Understanding File Permissions and Ownership

Assigning the file permissions with as minimal access as possible is crucial to maximizing security. This is where Linux file permissions and ownership come into play. I am only going to discuss the relevant operations on a high level. Refer to the Linux Foundation’s blog post about Linux file permissions for more details.

$ touch my-file

$ ls -l
total 0
-rw-r--r-- 1 root root 0 Sep 26 17:53 my-file

$ chown ben my-file
$ ls -l
total 0
-rw-r--r-- 1 ben  root 0 Sep 26 17:53 my-file

$ chmod -w file1
$ ls -l
total 0
-r--r--r-- 1 ben  root 0 Sep 26 17:53 my-file

Identifying and Disabling Open Ports

Applications like FTP servers, web servers, and file and print services such as Samba open ports as a means to expose a communication endpoint to clients. Running applications that open network communication can expose a security risk. You can eliminate the risk by simply disabling the service and deinstalling the application.

Let’s say we installed the Apache 2 HTTP web server on a control plane node with the following commands:

$ sudo apt update
$ sudo apt install apache2

We can inspect all open ports using the command line tool ss, a utility with similar functionality to netstat. The following command renders all of the open ports, including their processes. Among them is port 80, exposed by Apache 2:

$ sudo ss -ltpn
State    Recv-Q   Send-Q   Local Address:Port   Peer Address:Port   Process
...
LISTEN   0        511      *:80                 *:*                 users: \
(("apache2",pid=18435,fd=4),("apache2",pid=18434,fd=4),("apache2", ]\
pid=18432,fd=4))

You may have only needed the web server temporarily and may have simply forgotten about installing it. The process is currently managed by a server. You can review the status of a service with the systemctl status command:

$ sudo systemctl status apache2
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor \
     preset: enabled)
     Active: active (running) since Tue 2022-09-20 22:25:25 UTC; 39s ago
       Docs: https://httpd.apache.org/docs/2.4/
   Main PID: 18432 (apache2)
      Tasks: 55 (limit: 2339)
     Memory: 5.6M
     CGroup: /system.slice/apache2.service
             ├─18432 /usr/sbin/apache2 -k start
             ├─18434 /usr/sbin/apache2 -k start
             └─18435 /usr/sbin/apache2 -k start

Apache 2 is not needed by Kubernetes. We decide to shut down the service and deinstall the package:

$ sudo systemctl stop apache2
$ sudo systemctl disable apache2
Synchronizing state of apache2.service with SysV service script with \
/lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apache2
Removed /etc/systemd/system/multi-user.target.wants/apache2.service.
$ sudo apt purge --auto-remove apache2

Verify that the port isn’t used anymore. The ss command doesn’t find an application exposing port 80 anymore:

$ sudo ss -ltpn | grep :80

Setting Up Firewall Rules

Another way to control ports is with the help of an operating-system-level firewall. On Linux, you could use the Uncomplicated Firewall (UFW). This section will give you a very brief introduction on how to enable UFW and how to configure firewall rules.

Following the principle of least privilege, it’s a good idea to start by enabling the firewall and setting up deny rules for any incoming and outgoing network traffic. The following commands demonstrate the steps to achieve that:

$ sudo ufw allow ssh
Rules updated
Rules updated (v6)
$ sudo ufw default deny outgoing
Default outgoing policy changed to deny
(be sure to update your rules accordingly)
$ sudo ufw default deny incoming
Default incoming policy changed to deny
(be sure to update your rules accordingly)
$ sudo ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

You will want to allow external tools like kubectl to connect to the API server running on port 6443. On the control plane node, execute the following command to allow access to the API server port:

$ sudo ufw allow 6443
Rule added
Rule added (v6)

You will have to repeat the same process to open up other ports on control plane and worker nodes. Ensure that all other ports not needed to operate Kubernetes are blocked.

Using AppArmor

AppArmor provides access control to programs running on a Linux system. The tool implements an additional security layer between the applications invoked in the user space and the underlying system functionality. For example, we can restrict network calls or filesystem interaction. Many Linux distributions (e.g., Debian, Ubuntu, openSUSE) already ship with AppArmor. Therefore, AppArmor doesn’t have to be installed manually. Linux distributions that do not support AppArmor use Security-Enhanced Linux (SELinux) instead, which takes a similar approach to AppArmor. Understanding SELinux is out of scope for the CKS exam.

$ sudo aa-status
apparmor module is loaded.
31 profiles are loaded.
31 profiles are in enforce mode.
   /snap/snapd/15177/usr/lib/snapd/snap-confine
   ...
0 profiles are in complain mode.
14 processes have profiles defined.
14 processes are in enforce mode.
   /pause (11934) docker-default
   ...
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

#include <tunables/global>

profile k8s-deny-write flags=(attach_disconnected) { 
  #include <abstractions/base>

  file, 

  deny /** w, 
}

$ sudo apparmor_parser /etc/apparmor.d/k8s-deny-write

$ sudo aa-status
apparmor module is loaded.
32 profiles are loaded.
32 profiles are in enforce mode.
   k8s-deny-write
   ...

$ sudo apt-get update
$ sudo apt-get install apparmor-utils

apiVersion: v1
kind: Pod
metadata:
  name: hello-apparmor
  annotations:
    container.apparmor.security.beta.kubernetes.io/hello: \ 
    localhost/k8s-deny-write 
spec:
  containers:
  - name: hello 
    image: busybox:1.28
    command: ["sh", "-c", "echo 'Hello AppArmor!' && sleep 1h"]

$ kubectl apply -f pod.yaml
pod/hello-apparmor created
$ kubectl get pod hello-apparmor
NAME             READY   STATUS    RESTARTS   AGE
hello-apparmor   1/1     Running   0          4s

$ kubectl exec -it hello-apparmor -- /bin/sh
/ # touch test.txt
touch: test.txt: Permission denied

Using seccomp

Seccomp, short for “Secure Computing Mode,” is another Linux kernel feature that can restrict the calls made from the userspace into the kernel. A seccomp profile is the mechanism for defining the rules for restricting syscalls and their arguments. Using seccomp can reduce the risk of exploiting a Linux kernel vulnerability. For more information on seccomp on Kubernetes, see the documentation.

apiVersion: v1
kind: Pod
metadata:
  name: hello-seccomp
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault 
  containers:
  - name: hello
    image: busybox:1.28
    command: ["sh", "-c", "echo 'Hello seccomp!' && sleep 1h"]

$ kubectl apply -f pod.yaml
pod/hello-seccomp created
$ kubectl get pod hello-seccomp
NAME            READY   STATUS    RESTARTS   AGE
hello-seccomp   1/1     Running   0          4s

$ kubectl logs hello-seccomp
Hello seccomp!

$ sudo mkdir -p /var/lib/kubelet/seccomp/profiles

{
    "defaultAction": "SCMP_ACT_ALLOW", 
    "architectures": [ 
        "SCMP_ARCH_X86_64",
        "SCMP_ARCH_X86",
        "SCMP_ARCH_X32"
    ],
    "syscalls": [
        {
            "names": [
                "mkdir"
            ],
            "action": "SCMP_ACT_ERRNO" 
        }
    ]
}

apiVersion: v1
kind: Pod
metadata:
  name: hello-seccomp
spec:
  securityContext:
    seccompProfile:
      type: Localhost 
      localhostProfile: profiles/mkdir-violation.json 
  containers:
  - name: hello
    image: busybox:1.28
    command: ["sh", "-c", "echo 'Hello seccomp!' && sleep 1h"]
    securityContext:
      allowPrivilegeEscalation: false

$ kubectl apply -f pod.yaml
pod/hello-seccomp created
$ kubectl get pod hello-seccomp
NAME            READY   STATUS    RESTARTS   AGE
hello-seccomp   1/1     Running   0          4s

$ kubectl exec -it hello-seccomp -- /bin/sh
/ # mkdir test
mkdir: can't create directory test: Operation not permitted

Scenario: An Attacker Misuses root User Container Access

By default, containers run with root privileges. A vulnerability in the application could grant an attacker root access to the container. The container’s root user is the same as the root user on the host machine. Not only can the attacker then inspect or modify the application, but they can also potentially install additional tooling that allows the attacker to break out of the container and step into host namespace with root permissions. The attacker could also copy sensitive data from the host’s filesystem to the container. Figure 5-1 illustrates the scenario.

Figure 5-1. An attacker misuses root user container access

For that reason, running a container with the default root user is a bad idea. The next sections will explain how to declare a security context for the container that enforces the use of a non-root user or a specific user and/or group identifier. We’ll also discuss other security context settings relevant to shielding host access from the container.

Understanding Security Contexts

Kubernetes, as the container orchestration engine, can apply additional configuration to increase container security. You do so by defining a security context. A security context defines privilege and access control settings for a Pod or a container. The following list provides some examples:

• The user ID that should be used to run the Pod and/or container

• The group ID that should be used for filesystem access

• Granting a running process inside the container some privileges of the root user but not all of them

The security context is not a Kubernetes primitive. It is modeled as a set of attributes under the directive securityContext within the Pod specification. Security settings defined on the Pod level apply to all containers running in the Pod; however, container-level settings take precedence. For more information on Pod-level security attributes, see the PodSecurityContext API. Container-level security attributes can be found in the SecurityContext API.

Enforcing the Usage of a Non-Root User

We’ll have a look at a use case to make the functionality more transparent. Some images, like the one for the open source reverse-proxy server nginx, must be run with the root user. Say you wanted to enforce that containers cannot be run as the root user as a means to support a more sensible security strategy. The YAML manifest file container-non-root-user-error.yaml shown in Example 5-1 defines the security configuration specifically for a container. This security context only applies to this very container, but not others if you were to define more.

apiVersion: v1
kind: Pod
metadata:
  name: non-root-error
spec:
  containers:
  - image: nginx:1.23.1
    name: nginx
    securityContext:
      runAsNonRoot: true

The container fails during the startup process with the status CreateContainer​Confi⁠gError. A look at the Pod’s event log reveals that the image tries to run with the root user. The configured security context does not allow it:

$ kubectl apply -f container-non-root-user-error.yaml
pod/non-root-error created
$ kubectl get pod non-root-error
NAME             READY   STATUS                       RESTARTS   AGE
non-root-error   0/1     CreateContainerConfigError   0          9s
$ kubectl describe pod non-root-error
...
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  24s               default-scheduler  Successfully \
  assigned default/non-root to minikube
  Normal   Pulling    24s               kubelet            Pulling image \
  "nginx:1.23.1"
  Normal   Pulled     16s               kubelet            Successfully \
  pulled image "nginx:1.23.1" in 7.775950615s
  Warning  Failed     4s (x3 over 16s)  kubelet            Error: container \
  has runAsNonRoot and image will run as root (pod: "non-root-error_default \
  (6ed9ed71-1002-4dc2-8cb1-3423f86bd144)", container: secured-container)
  Normal   Pulled     4s (x2 over 16s)  kubelet            Container image \
  "nginx:1.23.1" already present on machine

There are alternative nginx container images available that are not required to run with the root user. An example is bitnami/nginx. Example 5-2 shows the contents of the file container-non-root-user-success.yaml. The major change in this file is the value assigned to the spec.containers[].image attribute.

apiVersion: v1
kind: Pod
metadata:
  name: non-root-success
spec:
  containers:
  - image: bitnami/nginx:1.23.1
    name: nginx
    securityContext:
      runAsNonRoot: true

Starting the container with the runAsNonRoot directive will work just fine. The container transitions into the “Running” status:

$ kubectl apply -f container-non-root-user-success.yaml
pod/non-root-success created
$ kubectl get pod non-root-success
NAME               READY   STATUS    RESTARTS   AGE
non-root-success   1/1     Running   0          7s

Let’s quickly check which user ID the container runs with. Shell into the container and run the id command. The output renders the user ID, the group ID, and the IDs of supplemental groups. The image bitnami/nginx sets the user ID to 1001 with the help of an instruction when the container image is built:

$ kubectl exec non-root-success -it -- /bin/sh
$ id
uid=1001 gid=0(root) groups=0(root)
$ exit

Setting a Specific User and Group ID

Many container images do not set an explicit user ID or group ID. Instead of running with the root default user, you can set the desired user ID and group ID to minimize potential security risks. The YAML manifest stored in the file container-user-id.yaml shown in Example 5-3 sets the user ID to 1000 and the group ID to 3000.

apiVersion: v1
kind: Pod
metadata:
  name: user-id
spec:
  containers:
  - image: busybox:1.35.0
    name: busybox
    command: ["sh", "-c", "sleep 1h"]
    securityContext:
      runAsUser: 1000
      runAsGroup: 3000

Creating the Pod will work without issues. The container transitions into the “Running” status:

$ kubectl apply -f container-user-id.yaml
pod/user-id created
$ kubectl get pods user-id
NAME      READY   STATUS    RESTARTS   AGE
user-id   1/1     Running   0          6s

You can inspect the user ID and group ID after shelling into the container. The current user is not allowed to create files in the / directory. Creating a file in the /tmp directory will work, as most users have the permissions to write to it:

$ kubectl exec user-id -it -- /bin/sh
/ $ id
uid=1000 gid=3000 groups=3000
/ $ touch test.txt
touch: test.txt: Permission denied
/ $ touch /tmp/test.txt
/ $ exit

Avoiding Privileged Containers

Kubernetes establishes a clear separation between the container namespace and the host namespace for processes, network, mounts, user ID, and more. You can configure the container’s security context to gain privileges to certain aspects of the host namespace. Assume the following implications when using a privileged container:

• Processes within a container almost have the same privileges as processes on the host.

• The container has access to all devices on the host.

• The root user in the container has similar privileges to the root user on the host.

• All directories on the host’s filesystem can be mounted in the container.

• Kernel settings can be changed, e.g., by using the sysctl command.

Let’s compare the behavior of a non-privileged container with one configured to run in privileged mode. First, we are going to set up a regular Pod, as shown in Example 5-4. No security context has been set on the Pod or container level.

apiVersion: v1
kind: Pod
metadata:
  name: non-privileged
spec:
  containers:
  - image: busybox:1.35.0
    name: busybox
    command: ["sh", "-c", "sleep 1h"]

Create the Pod and ensure that it comes up properly:

$ kubectl apply -f non-privileged.yaml
pod/non-privileged created
$ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
non-privileged   1/1     Running   0          6s

To demonstrate the isolation between the container namespace and host’s namespace, we’ll try to use the sysctl to change the hostname. As you can see in the output of the command, the container will clearly enforce the restricted privileges:

$ kubectl exec non-privileged -it -- /bin/sh
/ # sysctl kernel.hostname=test
sysctl: error setting key 'kernel.hostname': Read-only file system
/ # exit

To make a container privileged, simply assign the value true to the security context attribute privileged. The YAML manifest in Example 5-5 shows an example.

apiVersion: v1
kind: Pod
metadata:
  name: privileged
spec:
  containers:
  - image: busybox:1.35.0
    name: busybox
    command: ["sh", "-c", "sleep 1h"]
    securityContext:
      privileged: true

Create the Pod as usual. The Pod should transition into the “Running” status:

$ kubectl apply -f privileged.yaml
pod/privileged created
$ kubectl get pod privileged
NAME         READY   STATUS    RESTARTS   AGE
privileged   1/1     Running   0          6s

You can now see that the same sysctl will allow you to change the hostname:

$ kubectl exec privileged -it -- /bin/sh
/ # sysctl kernel.hostname=test
kernel.hostname = test
/ # exit

A container security context configuration related to privileged mode is the attribute allowPrivilegeEscalation. This attribute will allow a process running the container to gain more privileges than the parent process. The default value for the attribute is false, but if you see the attribute set to true, critically question its use. In most cases, you do not need the functionality.

Scenario: A Developer Doesn’t Follow Pod Security Best Practices

It is unfair to assume that developers of all trades and levels of seniority have extensive knowledge of Kubernetes features, especially the ones that apply to security best practices. In the previous section, we learned about the security context and which settings to avoid. Developers are probably unaware of those best practices without continued education and therefore may create Pods that use problematic security settings or none at all. Figure 5-2 shows a developer creating a Pod in privileged mode enabled by a copied manifest found on the internet. An attacker will gladly use this setup to their advantage.

Figure 5-2. A developer creates a Pod with enabled privileged mode

This is where you, as the Kubernetes security specialist, come in. The Kubernetes ecosystem provides core features and external tooling for enforcing security standards for Pods so that objects without the right configuration will be rejected, or at least audited. The next section explores the Kubernetes core feature named Pod Security Admission.

Understanding Pod Security Admission (PSA)

Older versions of Kubernetes shipped with a feature called Pod Security Policies (PSP). Pod Security Policies are a concept that help with enforcing security standards for Pod objects. Kubernetes 1.21 deprecated Pod Security Policies and introduced the replacement functionality Pod Security Admission. PSA determines which Pod Security Standard (PSS) to follow. A PSS defines a range of security policies from highly restrictive to highly permissive.

However, the Kubernetes release 1.25 completely removed Pod Security Policies. You may still see the feature listed in older versions of the CKS curriculum. We will only focus on Pod Security Admission in this book. PSA is enabled by default with Kubernetes 1.23; however, you will need to declare which Pods should adhere to the security standards. All you need to do to opt into the PSA feature is to add a label with a specific format to a namespace. All Pods in that namespace will have to follow the standards declared.

The label consists of three parts: a prefix, a mode, and a level. The prefix always uses the hard-coded value pod-security.kubernetes.io followed by a slash. The mode determines the handling of a violation. Finally, the level dictates the degree of security standards to adhere to. An example of such a label could look as follows:

metadata:
  labels:
    pod-security.kubernetes.io/enforce: restricted

The mode allows for setting three different options, shown in Table 5-1.

Table 5-2 illustrates the security policies determined by the level set for the PSA.

See the Kubernetes documentation for details on the PSA, including usage examples.

Enforcing Pod Security Standards for a Namespace

Let’s apply a PSA to a Pod in the namespace psa. Example 5-6 shows the definition of the namespace and the declaration of the relevant label. The label will enforce a PSS on the highest level of security standards.

apiVersion: v1
kind: Namespace
metadata:
  name: psa
  labels:
    pod-security.kubernetes.io/enforce: restricted

Make sure that the Pod is created in the namespace psa. Example 5-7 shows the YAML manifest for a simple Pod running the busybox image.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: psa
spec:
  containers:
  - image: busybox:1.35.0
    name: busybox
    command: ["sh", "-c", "sleep 1h"]

Violations will be rendered in the console upon running a command to create a Pod in the namespace. As you can see in the following, the Pod wasn’t allowed to be created:

$ kubectl create -f psa-namespace.yaml
namespace/psa created
$ kubectl apply -f psa-violating-pod.yaml
Error from server (Forbidden): error when creating "psa-pod.yaml": pods \
"busybox" is forbidden: violates PodSecurity "restricted:latest": \
allowPrivilegeEscalation != false (container "busybox" must set \
securityContext.allowPrivilegeEscalation=false), unrestricted \
capabilities (container "busybox" must set securityContext. \
capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container \
"busybox" must set securityContext.runAsNonRoot=true), seccompProfile \
(pod or container "busybox" must set securityContext.seccompProfile. \
type to "RuntimeDefault" or "Localhost")
$ kubectl get pod -n psa
No resources found in psa namespace.

You need to configure the Pod’s security context settings to follow the very restrictive standards. Example 5-8 shows an exemplary Pod definition that does not violate the Pod Security Standard.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: psa
spec:
  containers:
  - image: busybox:1.35.0
    name: busybox
    command: ["sh", "-c", "sleep 1h"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
      runAsNonRoot: true
      runAsUser: 2000
      runAsGroup: 3000
      seccompProfile:
        type: RuntimeDefault

Creating the Pod object now works as expected:

$ kubectl apply -f psa-non-violating-pod.yaml
pod/busybox created
$ kubectl get pod busybox -n psa
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          10s

PSA is a built-in, enabled-by-default feature in Kubernetes version 1.23 or higher. It’s easy to adopt, allows for picking and choosing a suitable policy standard, and can be configured to enforce or just log violations.

Unfortunately, PSA only applies to Pods with a predescribed set of policies. You cannot write your own custom rules, change the messaging, or mutate the Pod object should it not adhere to a PSS. In the next section, we are going to have a look at tooling that goes beyond the functionality of PSA.

Understanding Open Policy Agent (OPA) and Gatekeeper

Open Policy Agent (OPA) is an open source, general-purpose policy engine for enforcing rules. OPA is not specific to Kubernetes and can be used across other technology stacks. One of its benefits is the ability to define a policy in a very flexible fashion. You can write your own rules with the help of the query language named Rego. The validation logic written in Rego determines if the object is accepted or denied.

Gatekeeper is an extension to Kubernetes that uses OPA. Gatekeeper allows for defining and enforcing custom policies for any kind of Kubernetes API primitive. Therefore, it is far more versatile than PSA but requires more intricate knowledge on how to craft those rules. Gatekeeper gets involved in the admission control stage discussed in “Processing a Request”. The following list of policies tries to give you an impression on what’s possible with Gatekeeper:

• Ensuring that all Service objects need to define a label with the key team

• Ensuring that all container images defined by Pods need to be pulled from a company-internal registry

• Ensuring that Deployments need to control at least three replicas

At the time of writing, Gatekeeper allows for enforcing policies by rejecting object creation if requirements haven’t been met. Future versions of Gatekeeper might also provide a mechanism for mutating an object upon creation. For example, you may want to add specific label key-value pairs for any object created. The mutation would take care of adding those labels automatically.

Installing Gatekeeper

Installing Gatekeeper is relatively easy. All you need to do is to create a bunch of Kubernetes objects from a YAML manifest provided by the Gatekeeper project. You need to have cluster admin permissions to properly install Gatekeeper. The following command shows the kubectl command used to apply the latest release. For more information, see the installation manual:

$ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/\
gatekeeper/master/deploy/gatekeeper.yaml

Gatekeeper objects have been installed in the namespace gatekeeper-system. Make sure that all Pods in the namespace transition into the “Running” status before trying to use Gatekeeper:

$ kubectl get namespaces
NAME                STATUS   AGE
default             Active   29h
gatekeeper-system   Active   4s
...

Implementing an OPA Policy

We’ll use a specific use case as an example to demonstrate the moving parts required to define a custom OPA policy. “Using Network Policies to Restrict Pod-to-Pod Communication” explained how to assign a label to a namespace so that it can be selected from a network policy. At its core, our custom OPA policy will determine that namespaces need to define at least one label with the key app to signify the application hosted by the namespace.

Gatekeeper requires us to implement two components for custom policy, the constraint template and the constraint. In a nutshell, the constraint template defines the rules with Rego and describes the schema for the constraint. Example 5-9 shows a constraint template definition for enforcing a label assignment.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels 
      validation:
        openAPIV3Schema: 
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: | 
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Declares the kind to be used by the constraint.

Specifies the validation schema of the constraint. In this case, we allow to pass in a property named labels that captures the required label keys.

Uses Rego to check for the existence of labels and compares them to the list of required keys.

The constraint is essentially an implementation of the constraint template. It uses the kind defined by the constraint template and populates the data provided by the end user. In Example 5-10, the kind is K8sRequiredLabels, which we defined in the constraint template. We are matching on namespaces and expect them to define the label with the key app.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels 
metadata:
  name: ns-must-have-app-label-key
spec:
  match: 
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters: 
    labels: ["app"]

Uses the kind defined by the constraint template.

Defines the API resources the constraint template should apply to.

Declares that the labels property expects the key app to exist.

With the relevant YAML manifests in place, let’s create the objects for the constraint template and the constraint. Assume that the constraint template was written to the file constraint-template-labels.yaml and the constraint to the file constraint-ns-labels.yaml:

$ kubectl apply -f constraint-template-labels.yaml
constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
$ kubectl apply -f constraint-ns-labels.yaml
k8srequiredlabels.constraints.gatekeeper.sh/ns-must-have-app-label-key created

You can verify the validation behavior with a quick-to-run imperative command. The following command tries to create a new namespace without a label assignment. Gatekeeper will render an error message and prevent the creation of the object:

$ kubectl create ns governed-ns
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" \
denied the request: [ns-must-have-app-label-key] you must provide labels: {"app"}

Let’s make sure that we can actually create a namespace with the expected label assignment. Example 5-11 shows the YAML manifest of such a namespace.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app: orion
  name: governed-ns

The following command creates the object from the YAML manifest file named namespace-app-label.yaml:

$ kubectl apply -f namespace-app-label.yaml
namespace/governed-ns created

This simple example demonstrated the usage of OPA Gatekeeper. You can find a lot of other examples in the OPA Gatekeeper Library. Despite it not being spelled out explicitly in the CKS curriculum, you may also want to check out the project Kyverno, which recently gained a lot of traction with the Kubernetes community.

Scenario: An Attacker Gains Access to the Node Running etcd

Where etcd runs is dependent on the topology of your Kubernetes cluster. For the purpose of this scenario, we’ll assume that etcd runs on the control plane node. Any data stored in etcd exists in unencrypted form, so access to the control plane node allows for reading Secrets in plain text. Figure 5-3 shows an attacker gaining access to the control plane node and therefore the unencrypted Secrets in etcd.

Figure 5-3. An attacker gains access to etcd to read Secrets

One way to mitigate the situation is by encrypting the data stored in etcd. Access to etcd, either using etcdctl or by reading the etcd data from the filesystem, would not expose human-readable, sensitive information anymore.

Accessing etcd Data

We’ll start by showing how an attacker could read etcd data after being able to log into the control plane node. First, we need to create a Secret object to store in etcd. Use the following imperative command to create an entry:

$ kubectl create secret generic app-config --from-literal=password=passwd123
secret/app-config created

We created a Secret with the key-value pair password=passwd123. Shell into the control plane node using SSH. You can easily use the etcd client tool etcdctl to read an entry from etcd.

The following command uses the mandatory CLI options to read the contents from the Secret object named app-config. The following output displays the file contents in hexadecimal format. While not 100% obvious, you can still identify the key-value pair in plain text from the output:

$ sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/\
etcd/server.key get /registry/secrets/default/app-config | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 61 70 70 2d 63 6f  |s/default/app-co|
00000020  6e 66 69 67 0a 6b 38 73  00 0a 0c 0a 02 76 31 12  |nfig.k8s.....v1.|
00000030  06 53 65 63 72 65 74 12  d9 01 0a b7 01 0a 0a 61  |.Secret........a|
00000040  70 70 2d 63 6f 6e 66 69  67 12 00 1a 07 64 65 66  |pp-config....def|
00000050  61 75 6c 74 22 00 2a 24  36 38 64 65 65 34 34 38  |ault".*$68dee448|
00000060  2d 34 39 62 37 2d 34 34  32 66 2d 39 62 32 66 2d  |-49b7-442f-9b2f-|
00000070  33 66 39 62 39 62 32 61  66 66 36 64 32 00 38 00  |3f9b9b2aff6d2.8.|
00000080  42 08 08 97 f8 a4 9b 06  10 00 7a 00 8a 01 65 0a  |B.........z...e.|
00000090  0e 6b 75 62 65 63 74 6c  2d 63 72 65 61 74 65 12  |.kubectl-create.|
000000a0  06 55 70 64 61 74 65 1a  02 76 31 22 08 08 97 f8  |.Update..v1"....|
000000b0  a4 9b 06 10 00 32 08 46  69 65 6c 64 73 56 31 3a  |.....2.FieldsV1:|
000000c0  31 0a 2f 7b 22 66 3a 64  61 74 61 22 3a 7b 22 2e  |1./{"f:data":{".|
000000d0  22 3a 7b 7d 2c 22 66 3a  70 61 73 73 77 6f 72 64  |":{},"f:password|
000000e0  22 3a 7b 7d 7d 2c 22 66  3a 74 79 70 65 22 3a 7b  |":{}},"f:type":{|
000000f0  7d 7d 42 00 12 15 0a 08  70 61 73 73 77 6f 72 64  |}}B.....password|
00000100  12 09 70 61 73 73 77 64  31 32 33 1a 06 4f 70 61  |..passwd123..Opa|
00000110  71 75 65 1a 00 22 00 0a                           |que.."..|

In the next step, we’ll encrypt the Secret stored in etcd and then verify the existing entries with the same command.

Encrypting etcd Data

You can control how API data is encrypted in etcd with the help of the command line option --encryption-provider-config provided to the API server process. The value assigned to the parameter needs to point to a configuration file that defines an EncryptionConfiguration object. We’ll first create the configuration file and then configure the API server process to consume it.

Generate a 32-byte random key and base64-encode it. The value is needed to configure a so-called provider in the encryption configuration:

$ head -c 32 /dev/urandom | base64
W68xlPT/VXcOSEZJvWeIvkGJnGfQNFpvZYfT9e+ZYuY=

Next up, we’ll use the base64-encoded key and assign it to a provider in the encryption configuration, as shown in Example 5-12. Save the contents in the file /etc/kubernetes/enc/enc.yaml.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets 
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: W68xlPT/VXcOSEZJvWeIvkGJnGfQNFpvZYfT9e+ZYuY= 
      - identity: {}

Defines the API resource to be encrypted in etcd. We are only encrypting Secrets data here.

The base64-encoded key assigned to an AES-CBC encryption provider.

Edit the manifest at /etc/kubernetes/manifests/kube-apiserver.yaml, the YAML manifest that defines how to run an API server in a Pod. Add the parameter --encryption-provider-config, and define the Volume and its mountpath for the configuration file as the following shows:

$ sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: \
    192.168.56.10:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
    volumeMounts:
    ...
    - name: enc
      mountPath: /etc/kubernetes/enc
      readonly: true
  volumes:
  ...
  - name: enc
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate
...

The Pod running the API server should automatically restart. This process may take a couple of minutes. Once fully restarted, you should be able to query for it:

$ kubectl get pods -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
...
kube-apiserver-control-plane   1/1     Running   0          69s

New Secrets will be encrypted automatically. Existing Secrets need to be updated. You can run the following command to perform an update on Secrets across all namespaces. This includes the Secret named app-config in the default namespace:

$ kubectl get secrets --all-namespaces -o json | kubectl replace -f -
...
secret/app-config replaced

Running the etcdctl command we used before will reveal that the aescbc provider has been used to encrypt the data. The password value cannot be read in plain text anymore:

$ sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/\
etcd/server.key get /registry/secrets/default/app-config | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 61 70 70 2d 63 6f  |s/default/app-co|
00000020  6e 66 69 67 0a 6b 38 73  3a 65 6e 63 3a 61 65 73  |nfig.k8s:enc:aes|
00000030  63 62 63 3a 76 31 3a 6b  65 79 31 3a ae 26 e9 c2  |cbc:v1:key1:.&..|
00000040  7b fd a2 74 30 24 85 61  3c 18 1e 56 00 a1 24 65  |{..t0$.a<..V..$e|
00000050  52 3c 3f f1 24 43 9f 6d  de 5f b0 84 32 18 84 47  |R<?.$C.m._..2..G|
00000060  d5 30 e9 64 84 22 f5 d0  0b 6f 02 af db 1d 51 34  |.0.d."...o....Q4|
00000070  db 57 c8 17 93 ed 9e 00  ea 9a 7b ec 0e 75 0c 49  |.W........{..u.I|
00000080  6a e9 97 cd 54 d4 ae 6b  b6 cb 65 8a 5d 4c 3c 9c  |j...T..k..e.]L<.|
00000090  db 9b ed bc ce bf 3c ef  f6 2e cb 6d a2 53 25 49  |......<....m.S%I|
000000a0  d4 26 c5 4c 18 f3 65 bb  a8 4c 0f 8d 6e be 7b d3  |.&.L..e..L..n.{.|
000000b0  24 9b a8 09 9c bb a3 f9  53 49 78 86 f5 24 e7 10  |$.......SIx..$..|
000000c0  ad 05 45 b8 cb 31 bd 38  b6 5c 00 02 b2 a4 62 13  |..E..1.8.\....b.|
000000d0  d5 82 6b 73 79 97 7e fa  2f 5d 3b 91 a0 21 50 9d  |..ksy.~./];..!P.|
000000e0  77 1a 32 44 e1 93 9b 9c  be bf 49 d2 f9 dc 56 23  |w.2D......I...V#|
000000f0  07 a8 ca a5 e3 e7 d1 ae  9c 22 1f 98 b1 63 b8 73  |........."...c.s|
00000100  66 3f 9f a5 6a 45 60 a7  81 eb 32 e5 42 4d 2b fd  |f?..jE`...2.BM+.|
00000110  65 6c c2 c7 74 9f 1d 6a  1c 24 32 0e 7a 94 a2 60  |el..t..j.$2.z..`|
00000120  22 77 58 c9 69 c3 55 72  e8 fb 0b 63 9d 7f 04 31  |"wX.i.Ur...c...1|
00000130  00 a2 07 76 af 95 4e 03  0a 92 10 b8 bb 1e 89 94  |...v..N.........|
00000140  45 60 01 45 bf d7 95 df  ff 2e 9e 31 0a           |E`.E.......1.|
0000014d

For more details on encrypting etcd data, refer to the Kubernetes documentation. There, you will find additional information on other encryption providers, how to rotate the decryption key, and the process to consider for a high-availability (HA) cluster setup.

Scenario: An Attacker Gains Access to Another Container

In this scenario, we are confronted with a developer that pulls a container image from a public registry, as referenced by a Pod. The container has not been scanned for security vulnerabilities. An attacker can push a new tag of the container image executing malicious code. After instantiating a container from the image, the malicious code running in the kernel group of container 1 can access the process running in container 2. As you can see in Figure 5-4, both containers use the same kernel of the host system.

Figure 5-4. An attacker gains access to another container

Generally speaking, it’s not a good idea to blindly trust public container images. One way to ensure that such a container image runs with more isolation is the container runtime sandbox. The next section will introduce you to two implementations, both of which are explicitly mentioned by the curriculum.

Available Container Runtime Sandbox Implementations

In this book, we’ll only want to talk about two container runtime sandbox implementations, Kata Containers and gVisor. Kata containers achieves container isolation by running them in a lightweight virtual machine. gVisor takes a different approach. It effectively implements a Linux kernel that runs on the host system. Therefore, syscalls are not shared anymore across all containers on the host system.

A deeper discussion on the feature sets or specific use cases for those container runtime sandbox implementations goes beyond the scope of this book. We’ll simply learn how to use one solution as an example, gVisor, and how to tie it into Kubernetes. Have a look at the talk “Kata Containers and gVisor: a Quantitative Comparison” for an in-depth comparison.

Installing and Configuring gVisor

The following instructions describe the steps required to install gVisor on Linux using the apt package manager. You will want to repeat those steps on all host machines declared as worker nodes. For the exam, you will not be expected to install gVisor or Kata Containers. You can assume that the container runtime sandbox has already been installed and configured.

Start by installing the dependencies for gVisor with the following command:

$ sudo apt-get update && \
  sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg

Next, configure the key used to sign archives and the repository. As you can see in the following commands, gVisor is hosted on Google storage:

$ curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/\
keyrings/gvisor-archive-keyring.gpg
$ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/\
gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases \
release main" | sudo tee /etc/apt/sources.list.d/gvisor.list > /dev/null

gVisor includes an Open Container Initiative (OCI) runtime called runsc. The runsc runtime integrates with tools like Docker and Kubernetes to run container runtime sandboxes. The following command installs the executable from the repository:

$ sudo apt-get update && sudo apt-get install -y runsc

Let’s assume we are using containerd as the container runtime. You need to add some configuration to containerd to make it aware of runsc. You can find similar instructions for other container runtimes in the gVisor documentation:

$ cat <<EOF | sudo tee /etc/containerd/config.toml
version = 2
[plugins."io.containerd.runtime.v1.linux"]
  shim_debug = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"
EOF

Finally, restart containerd to let the changes take effect:

$ sudo systemctl restart containerd

We successfully installed gVisor and can now configure Pods to use it.

Creating and Using a Runtime Class

It’s a two-step approach to use a container runtime sandbox in a Pod. First, you need to create a runtime class. A RuntimeClass is a Kubernetes API resource that defines the configuration of the container runtime. Example 5-13 shows a YAML manifest of a container runtime that points to runsc as the handler we set in the containerd configuration file earlier.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

We can now reference the runtime class name, gvisor, in the configuration of a Pod. Example 5-14 shows a Pod definition that assigns the runtime class using the attribute spec.runtimeClassName.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  runtimeClassName: gvisor
  containers:
  - name: nginx
    image: nginx:1.23.2

Create the runtime class and Pod object using the apply command:

$ kubectl apply -f runtimeclass.yaml
runtimeclass.node.k8s.io/gvisor created
$ kubectl apply -f pod.yaml
pod/nginx created

You can verify that the container is running with the container runtime sandbox. Simply execute the dmesg command to examine the kernel ring buffer. The output from the command should mention gVisor, as shown in the following:

$ kubectl exec nginx -- dmesg
[    0.000000] Starting gVisor...
[    0.123202] Preparing for the zombie uprising...
[    0.415862] Rewriting operating system in Javascript...
[    0.593368] Reading process obituaries...
[    0.741642] Segmenting fault lines...
[    0.797360] Daemonizing children...
[    0.831010] Creating bureaucratic processes...
[    1.313731] Searching for needles in stacks...
[    1.455084] Constructing home...
[    1.834278] Gathering forks...
[    1.928142] Mounting deweydecimalfs...
[    2.109973] Setting up VFS...
[    2.157224] Ready!

Scenario: An Attacker Listens to the Communication Between Two Pods

An attacker can use the default, unencrypted Pod-to-Pod network communication behavior to their advantage. As you can see in Figure 5-5, an attacker doesn’t even need to break into a Pod. They can simply listen to the Pod-to-Pod communication by impersonating the sending or the receiving side, extract sensitive information, and then use it for more advanced attack vectors.

Figure 5-5. An attacker listens to Pod-to-Pod communication

You can mitigate the situation by setting up mTLS. The next section will briefly touch on the options for making that happen.

Adopting mTLS in Kubernetes

The tricky part about implementing mTLS in a Kubernetes cluster is the management of certificates. As you can imagine, we’ll have to deal with a lot of certificates when implementing a microservices architecture. Those certificates are usually generated by an official certificate authority (CA) to ensure that they can be trusted. Requesting a certificate involves sending a certificate signing request (CSR) to the CA. If the CA approves the request, it creates the certificate, then signs and returns it. It’s recommended to assign short lifespans to a certificate before it needs to be re-issued again. That process is called certificate rotation.

It’s somewhat unclear to what degree of detail the CKS exam requires you to understand mTLS. The general process of requesting and approving a certificate is described in the Kubernetes documentation.

In most cases, Kubernetes administrators rely on a Kubernetes service mesh to implement mTLS instead of implementing it manually. A Kubernetes service mesh, such as Linkerd or Istio, is a tool for adding cross-cutting functionality to your cluster, like observability and security.

Another option is to use transparent encryption to ensure that traffic doesn’t go on the wire unencrypted. Some of the popular CNI plugins, such as Calico and Cilium, have added support for WireGuard. WireGuard is an open source, lightweight, and secure Virtual Private Network (VPN) solution that doesn’t require the configuration or management of encryption keys or certificates. Many teams prefer WireGuard over a service mesh as it is easier to manage.

Services meshes and WireGuard are out of scope for the exam.

Scenario: An Attacker Exploits Container Vulnerabilities

One of the first decisions you have to make when defining a Dockerfile is the selection of a base image. The base image provides the operating system and additional dependencies, and it may expose shell access.

Some of the base images you can choose from on a public registry like Docker Hub are large in size and will likely contain functionality you don’t necessarily need to run your application inside of it. The operating system itself, as well as any dependencies available with the base image, can expose vulnerabilities.

In Figure 6-1, the attacker was able to figure out details about the container by gaining access to it. Those vulnerabilities can now be used as a launching pad for more advanced attacks.

Figure 6-1. An attacker exploits container image vulnerabilities

It is recommended to use a base image with a minimal set of functionality and dependencies. The next couple of sections will explain the methods for creating a more optimized base image that’s faster to build, quicker to download from a container registry, and that will ultimately lead to a smaller attack surface simply by reducing the bloat. The next sections will touch on the most important techniques. You can find a more detailed list of best practices for writing Dockerfiles in the Docker documentation.

Picking a Base Image Small in Size

Some container images can have a size of a gigabyte or even more. Do you really need all the functionality bundled with such a container image? Unlikely. Thankfully, many container producers upload a wide range of variations of their container images for the same release. One of those variations is an alpine image, a small, lightweight, and less vulnerable Linux distribution. As you can see in the following output, the downloaded alpine container image with the tag 3.17.0 only has a size of 7.05MB:

$ docker pull alpine:3.17.0
...
$ docker image ls alpine
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
alpine       3.17.0    49176f190c7e   3 weeks ago   7.05MB

The alpine container image comes with an sh shell you can use to troubleshoot the process running inside of the container. You can use the following command to open an interactive shell in a new container:

$ docker run -it alpine:3.17.0 /bin/sh
/ # exit

While runtime troubleshooting functionality can be useful, offering a shell as part of a container image increases the size of it and potentially opens the door for attackers. Additionally, the more software there is inside of a container image, the more vulnerabilities it will have.

You can further reduce the container image size and the attack surface by using a distroless image offered by Google. The following command downloads the latest tag of the container image gcr.io/distroless/static-debian11 and renders its details. The size of the container image is only 2.34MB:

$ docker pull gcr.io/distroless/static-debian11
...
$ docker image ls gcr.io/distroless/static-debian11:latest
REPOSITORY                          TAG       IMAGE ID       CREATED      \
  SIZE
gcr.io/distroless/static-debian11   latest    901590160d4d   53 years ago \
  2.34MB

A distroless container image does not ship with any shell, which you can observe by running the following command:

$ docker run -it gcr.io/distroless/static-debian11:latest /bin/sh
docker: Error response from daemon: failed to create shim task: OCI runtime \
create failed: runc create failed: unable to start container process: exec: \
"/bin/sh": stat /bin/sh: no such file or directory: unknown.

Kubernetes offers the concept of ephemeral containers for troubleshooting distroless containers. Those containers are meant to be disposable and can be deployed for troubleshooting minimal containers that would usually not allow opening a shell. Discussing ephemeral containers is out of scope of this book, but you can find more information about them in the Kubernetes documentation.

Using a Multi-Stage Approach for Building Container Images

As a developer, you can decide to build the application code as part of the instructions in a Dockerfile. This process may include compiling the code and building a binary that should become the entry point of the container image. Having all the necessarily tools and dependencies available to implement the process will automatically blow up the size of the container image, plus you won’t need those dependencies at runtime anymore.

The idea of a multi-stage build in Docker is that you separate the build stage from the runtime stage. As a result, all dependencies needed in the build stage will be discarded after the process has been performed and therefore do not end up in the final container image. This approach leads to a much smaller container image size by removing all the unnecessary cruft.

While we won’t go into the details of crafting and fully understanding multi-stage Dockerfile, I want to show you the differences on a high level. We’ll start by showing you a Dockerfile that builds and tests a simple program using the programming language Go, as shown in Example 6-1. In essence, we are using a base image that includes Go 1.19.4. The Go runtime provides the go executes, which we’ll invoke to execute the tests and build the binary of the application.

FROM golang:1.19.4-alpine 
WORKDIR /app

COPY go.mod .
COPY go.sum .
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 go test -v 
RUN go build -o /go-sample-app . 
CMD ["/go-sample-app"]

Uses a Go base image

Executes the tests against the application code

Builds the binary of the Go application

You can produce the image using the docker build command, as shown in the following:

$ docker build . -t go-sample-app:0.0.1
...

The size of the result container image is pretty big, 348MB, and there’s a good reason for it. It includes the Go runtime, even though we don’t actually need it anymore when starting the container. The go build command produced the binary that we can run as the container’s entry point:

$ docker images
REPOSITORY      TAG     IMAGE ID       CREATED          SIZE
go-sample-app   0.0.1   88175f3ab0d3   44 seconds ago   358MB

Next up, we’ll have a look at the multi-stage approach. In a multi-stage Dockerfile, you define at least two stages. In Example 6-2, we specify a stage aliased with build to run the tests and build the binary, similarly to what we’ve done earlier. A second stage copies the binary produced by the stage build into a dedicated directory; however, it uses the alpine base image to run it. When running the docker build command, the stage build will not be included in the final container image anymore.

FROM golang:1.19.4-alpine AS build 
RUN apk add --no-cache git
WORKDIR /tmp/go-sample-app
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .

RUN CGO_ENABLED=0 go test -v 
RUN go build -o ./out/go-sample-app . 

FROM alpine:3.17.0 
RUN apk add ca-certificates
COPY --from=build /tmp/go-sample-app/out/go-sample-app /app/go-sample-app 
CMD ["/app/go-sample-app"]

Uses a Go base image for building and testing the program in the stage named build.

Executes the tests against the application code.

Builds the binary of the Go application.

Uses a much smaller base image for running the application in a container.

Copies the application binary produced in the build stage and uses it as the command to run when the container is started.

The resulting container image size is significantly smaller when using the alpine base image, only 12MB. You can verify the outcome by running the docker build command again and inspecting the size of the container image by listing it:

$ docker build . -t go-sample-app:0.0.1
...
$ docker images
REPOSITORY      TAG     IMAGE ID       CREATED          SIZE
go-sample-app   0.0.1   88175f3ab0d3   44 seconds ago   12MB

Not only did we reduce the size, we also reduced the attack surface by including fewer dependencies. You can further reduce the size of the container image by incorporating a distroless base image instead of the alpine base image.

Reducing the Number of Layers

Every Dockerfile consists of an ordered list of instructions. Only some instructions create a read-only layer in the resulting container image. Those instructions are FROM, COPY, RUN, and CMD. All other instructions will not create a layer as they create temporary intermediate images. The more layers you add to the container image, the slower will be the build execution time and/or the bigger will be the size of the container image. Therefore, you need to be cautious about the instructions used in your Dockerfile to minimize the footprint of the container image.

It’s common practice to execute multiple commands in a row. You may define those commands using a list of RUN instructions on individual lines, as shown in Example 6-3.

FROM ubuntu:22.10
RUN apt-get update -y
RUN apt-get upgrade -y
RUN apt-get install -y curl

Each RUN instruction will create a layer, potentially adding to the size of the container image. It’s more efficient to string them together with && to ensure that only a single layer will be produced. Example 6-4 shows an example of this optimization technique.

FROM ubuntu:22.10
RUN apt-get update -y && apt-get upgrade -y && apt-get install -y curl

Using Container Image Optimization Tools

It’s easy to forget about the optimization practices mentioned previously. The open source community developed a couple of tools that can help with inspecting a produced container image. Their functionalities provide useful tips for pairing down on unnecessary layers, files, and folders:

DockerSlim

DockerSlim will optimize and secure your container image by analyzing your application and its dependencies. You can find more information in the tool’s GitHub repository.

Dive

Dive is a tool for exploring the layers baked into a container image. It makes it easy to identify unnecessary layers, which you can further optimize on. The code and documentation for Dive are available in a GitHub repository.

This is only the short list of container image optimization tools. In “Static Analysis of Workload”, we’ll have a look at other tools that focus on the static analysis of Dockerfiles and Kubernetes manifests.

Signing Container Images

You can sign a container image before pushing it to a container registry. Signing can be achieved with the docker trust sign command, which adds a signature to the container image, the so-called image digest. An image digest is derived from the contents of the container image and commonly represented in the form of SHA256. When consuming the container image, Kubernetes can compare the image digest with the contents of the image to ensure that it hasn’t been tampered with.

Scenario: An Attacker Injects Malicious Code into a Container Image

The Kubernetes component that verifies the image digest is the kubelet. If you configured the image pull policy to Always, the kubelet will query for the image digest from the container registry even though it may have downloaded and verified the container image before.

An attacker can try to modify the contents of an existing container image, inject malicious code, and upload it to the container registry with the same tag, as shown in Figure 6-2. The malicious code running in the container could then send sensitive information to a third-party server accessible by the attacker.

Figure 6-2. An attacker injects malicious code into a container image

Validating Container Images

In Kubernetes, you’re able to append the SHA256 image digest to the specification of a container. For example, this can be achieved with the attribute spec.containers[].image. The image digest is generally available in the container registry. For example, Figure 6-3 shows the image digest for the container image alpine:3.17.0 on Docker Hub. In this example, the image digest is sha256:c0d488a800e4127c334ad20d61d7bc21b4097540327217dfab52262adc02380c.

Figure 6-3. The image digest of the alpine:3.17.0 container image on Docker Hub

Let’s see the image digest in action. Instead of using the tag, Example 6-5 specifies the container image by appending the image digest.

apiVersion: v1
kind: Pod
metadata:
  name: alpine-valid
spec:
  containers:
  - name: alpine
    image: alpine@sha256:c0d488a800e4127c334ad20d61d7bc21b40 \
           97540327217dfab52262adc02380c
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10; done"]

Creating the Pod will work as expected. The image digest will be verified and the container transitions into the “Running” status:

$ kubectl apply -f pod-valid-image-digest.yaml
pod/alpine-valid created
$ kubectl get pod alpine-valid
NAME           READY   STATUS    RESTARTS   AGE
alpine-valid   1/1     Running   0          6s

Example 6-6 shows the same Pod definition; however, the image digest has been changed so that it does not match with the contents of the container image.

apiVersion: v1
kind: Pod
metadata:
  name: alpine-invalid
spec:
  containers:
  - name: alpine
    image: alpine@sha256:d006a643bccb6e9adbabaae668533c7f2e5 \
           111572fffb5c61cb7fcba7ef4150b
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo hello; sleep 10; done"]

You will see that Kubernetes can still create the Pod object but it can’t properly validate the hash of the container image. This results in the status “ErrImagePull.” As you can see from the event log, the container image couldn’t even be pulled from the container registry:

$ kubectl get pods
NAME             READY   STATUS         RESTARTS   AGE
alpine-invalid   0/1     ErrImagePull   0          29s
$ kubectl describe pod alpine-invalid
...
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  13s   default-scheduler  Successfully assigned default \
  /alpine-invalid to minikube
  Normal   Pulling    13s   kubelet            Pulling image "alpine@sha256: \
  d006a643bccb6e9adbabaae668533c7f2e5111572fffb5c61cb7fcba7ef4150b"
  Warning  Failed     11s   kubelet            Failed to pull image \
  "alpine@sha256:d006a643bccb6e9adbabaae668533c7f2e5111572fffb5c61cb7fcba7ef4 \
  150b": rpc error: code = Unknown desc = Error response from daemon: manifest \
  for alpine@sha256:d006a643bccb6e9adbabaae668533c7f2e5111572fffb5c61cb7fcba7e \
  f4150b not found: manifest unknown: manifest unknown
  Warning  Failed     11s   kubelet            Error: ErrImagePull
  Normal   BackOff    11s   kubelet            Back-off pulling image \
  "alpine@sha256:d006a643bccb6e9adbabaae668533c7f2e5111572fffb5c61cb7fcba7ef415 \
  0b"
  Warning  Failed     11s   kubelet            Error: ImagePullBackOff

Using Public Image Registries

Whenever a Pod is created, the container runtime engine will download the declared container image from a container registry if it isn’t available locally yet. This runtime behavior can be further tweaked using the image pull policy.

The prefix in the image name declares the domain name of the registry; e.g., gcr.io/google-containers/debian-base:v1.0.1 refers to the container image google-containers/debian-base:v1.0.1 in the Google Cloud container registry, denoted by gcr.io. The container runtime will try to resolve it from docker.io, the Docker Hub container registry if you leave off the domain name in the container image declaration.

Scenario: An Attacker Uploads a Malicious Container Image

While it is convenient to resolve container images from public container registries, it doesn’t come without risks. Anyone with a login to those container registries can upload new images. Consuming container images usually doesn’t even require an account.

As shown in Figure 6-4, an attacker can upload a container image containing malicious code to a public registry using stolen credentials. Any container referencing the container image from that registry will automatically run the malicious code.

Figure 6-4. An attacker uploads a malicious container image

On an enterprise level, you need to control which container registries you trust. It’s recommended to set up your own container registry within your company’s network, which you can fully control and govern. Alternatively, you can set up a private container registry in a cloud provider environment, not accessible by anyone else.

One of the prominent binary repository managers you can choose from is JFrog Artifactory. The product fully supports storing, scanning, and serving container images. Any consumer of container images should only be allowed to pull images from your whitelisted container registry. All other container registries should be denied.

Whitelisting Allowed Image Registries with OPA GateKeeper

One way to govern container registry usage is with OPA Gatekeeper. We discussed the installation process and functionality of OPA Gatekeeper in “Understanding Open Policy Agent (OPA) and Gatekeeper”. Here, we’ll’ touch on the constraint template and constraint for allowing trusted container registries.

Example 6-7 shows the constraint template I got directly from the OPA Gatekeeper library. As input properties, the constraint template defines an array of strings representing the prefixes of container registries. The Rego rules verify not only the assigned container images for the attribute spec.containers[] but also spec.initContainers[] and spec.ephemeralContainers[].

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedrepos
  annotations:
    metadata.gatekeeper.sh/title: "Allowed Repositories"
    metadata.gatekeeper.sh/version: 1.0.0
    description: >-
      Requires container images to begin with a string from the specified list.
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRepos
      validation:
        openAPIV3Schema:
          type: object
          properties:
            repos:
              description: The list of prefixes a container image is allowed to have.
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedrepos

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; \
          good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("container <%v> has an invalid image repo <%v>, allowed \
          repos are %v", [container.name, container.image, input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.initContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; \
          good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("initContainer <%v> has an invalid image repo <%v>, \
          allowed repos are %v", [container.name, container.image, \
          input.parameters.repos])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.ephemeralContainers[_]
          satisfied := [good | repo = input.parameters.repos[_] ; \
          good = startswith(container.image, repo)]
          not any(satisfied)
          msg := sprintf("ephemeralContainer <%v> has an invalid image repo <%v>, \
          allowed repos are %v", [container.name, container.image, \
          input.parameters.repos])
        }

The constraint shown in Example 6-8 is in charge of defining which container registries we want to allow. You’d usually go with a domain name of a server hosted in your company’s network. Here, we are going to use gcr.io/ for demonstration purposes.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: repo-is-gcr
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    repos:
      - "gcr.io/"

Let’s create both objects using the apply command:

$ kubectl apply -f allowed-repos-constraint-template.yaml
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
$ kubectl apply -f gcr-allowed-repos-constraint.yaml
k8sallowedrepos.constraints.gatekeeper.sh/repo-is-gcr created

In the constraint, we didn’t specify a namespace that the rules should apply to. Therefore, they’ll apply across all namespaces in the cluster. The following commands verify that the whitelisting rules work as expected. The following command tries to create a Pod using the nginx container image from Docker Hub. The creation of the Pod is denied with an appropriate error message:

$ kubectl run nginx --image=nginx:1.23.3
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" \
denied the request: [repo-is-gcr] container <nginx> has an invalid image \
repo <nginx:1.23.3>, allowed repos are ["gcr.io/"]

The next command creates a Pod with a container image from the Google Cloud container registry. The operation is permitted and the Pod object is created:

$ kubectl run busybox --image=gcr.io/google-containers/busybox:1.27.2
pod/busybox created
$ kubectl get pods
NAME      READY   STATUS      RESTARTS     AGE
busybox   0/1     Completed   1 (2s ago)   3s

Whitelisting Allowed Image Registries with the ImagePolicyWebhook Admission Controller Plugin

Another way to validate the use of allowed image registries is to intercept a call to the API server when a Pod is about to be created. This mechanism can be achieved by enabling an admission controller plugin. Once configured, the logic of an admission controller plugin is automatically invoked when the API server receives the request, but after it could authenticate and authorize the caller. We already touched on the phases an API call has to go through in “Processing a Request”.

The admission controller provides a way to approve, deny, or mutate a request before the request takes effect. For example, we can inspect the data sent with the API request to create a Pod, iterate over the assigned container images, and execute custom logic to validate the container image notation. If the container image doesn’t stick to the expected conventions, we can deny the creation of the Pod. Figure 6-5 illustrates the high-level workflow.

Figure 6-5. Intercepting a Pod-specific API call and handling it with a webhook

The ImagePolicyWebhook admission controller plugin is one of the plugins we can configure for intercepting Kubernetes API calls. You can probably derive the plugin’s functionality from its name. It defines a policy for all defined container images in a Pod. To compare container images with the defined policy, the plugin calls off to a service external to Kubernetes via HTTPS, a webhook. The external service then makes the decision on how to validate the data. In the context of the admission controller plugin, the external service is also referred to as backend.

Implementing the Backend Application

The backend application can be implemented with a programming language of your choice. There are only three requirements it must fulfill:

1. It’s a web application that can handle HTTPS requests.

2. It can accept and parse JSON request payloads.

3. It can send a JSON response payload.

Implementing the backend application is not part of the CKS exam, but you can find a sample Go-based implementation in the GitHub repository of this book. Be aware that the code is not considered to be production-ready.

The following commands demonstrate the runtime behavior of the application accessible at https://localhost:8080/validate. You can find an example request and response JSON body in the Kubernetes documentation.

The following curl command calls the validation logic for the container image nginx:1.19.0. As you can see from the JSON response, the image is denied:

$ curl -X POST -H "Content-Type: application/json" -k -d \'{"apiVersion": \
"imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "spec": \
{"containers": [{"image": "nginx:1.19.0"}]}}' https://localhost:8080/validate
{"apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", \
"status": {"allowed": false, "reason": "Denied request: [container 1 \
has an invalid image repo nginx:1.19.0, allowed repos are [gcr.io/]]"}}

The following curl command calls the validation logic for the container image gcr.io/nginx:1.19.0. As you can see from the JSON response, the image is allowed:

$ curl -X POST -H "Content-Type: application/json"  -k -d '{"apiVersion": \
"imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", "spec": {"containers": \
[{"image": "gcr.io/nginx:1.19.0"}]}}' https://localhost:8080/validate
{"apiVersion": "imagepolicy.k8s.io/v1alpha1", "kind": "ImageReview", \
"status": {"allowed": true, "reason": ""}}

Configuring the ImagePolicyWebhook Admission Controller Plugin

For the exam, you are expected to understand how to “wire” the ImagePolicyWebhook admission controller plugin. This section will walk you through the process. First, you’ll need to create a configuration file for the admission controller so it knows what plugins to use and how it should behave at runtime. Create the file /etc/kubernetes/admission-control/image-policy-webhook-admission-config.yaml and populate it with the content shown in Example 6-9.

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: ImagePolicyWebhook 
    configuration:
      imagePolicy:
        kubeConfigFile: /etc/kubernetes/admission-control/ \
                        imagepolicywebhook.kubeconfig 
        allowTTL: 50
        denyTTL: 50
        retryBackoff: 500
        defaultAllow: false 

Provides the configuration for the ImagePolicyWebhook plugin.

Points to the configuration file used to configure the backend.

Denies an API request if the backend cannot be reached. The default is true but setting it to false is far more sensible.

Next, create the file referenced by the plugins[].configuration.imagePolicy.kubeConfigFile attribute. The contents of the file point to the HTTPS URL of the external service. It also defines the client certificate and key file, as well as the CA file on disk. Example 6-10 shows the contents of the configuration file.

apiVersion: v1
kind: Config
preferences: {}
clusters:
  - name: image-validation-webhook
    cluster:
      certificate-authority: /etc/kubernetes/admission-control/ca.crt
      server: https://image-validation-webhook:8080/validate 
contexts:
- context:
    cluster: image-validation-webhook
    user: api-server-client
  name: image-validation-webhook
current-context: image-validation-webhook
users:
  - name: api-server-client
    user:
      client-certificate: /etc/kubernetes/admission-control/api-server-client.crt
      client-key: /etc/kubernetes/admission-control/api-server-client.key

The URL to the backend service. Must use the HTTPS protocol.

Enable the ImagePolicyWebhook admission controller plugin for the API server and point the admission controller to the configuration file. To achieve this, edit the configuration file of the API server, usually found at /etc/kubernetes/manifests/kube-apiserver.yaml.

Find the command line option --enable-admission-plugins and append the value ImagePolicyWebhook to the existing list of plugins, separated by a comma. Provide the command line option --admission-control-config-file if it doesn’t exist yet, and set the value to /etc/kubernetes/admission-control/image-policy-webhook-admission-configuration.yaml. Given that the configuration file lives in a new directory, you will have to define it as a Volume and mount it to the container. Example 6-11 shows the relevant options for the API server container.

...
spec:
  containers:
  - command:
    - kube-apiserver
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/etc/kubernetes/admission-control/ \
      image-policy-webhook-admission-configuration.yaml
    ...
    volumeMounts:
    ...
    - name: admission-control
      mountPath: /etc/kubernetes/admission-control
      readonly: true
  volumes:
  ...
  - name: admission-control
    hostPath:
      path: /etc/kubernetes/admission-control
      type: DirectoryOrCreate
...

The Pod running the API server should automatically restart. This process may take a couple of minutes. Restart the kubelet service with the command sudo systemctl restart kubelet should the API server not come up by itself. Once fully restarted, you should be able to query for it:

$ kubectl get pods -n kube-system
NAME                           READY   STATUS    RESTARTS   AGE
...
kube-apiserver-control-plane   1/1     Running   0          69s

Any API call that requests the creation of a Pod will now be routed to the backend. Based on the validation result, the creation of the object will be allowed or denied.

Using Hadolint for Analyzing Dockerfiles

Haskell Dockerfile Linter, also called hadolint in short, is a linter for Dockerfiles. The tool inspects a Dockerfile based on best practices listed on the Docker documentation page. Example 6-12 shows an unoptimized Dockerfile for building a container image running a Go-based application.

FROM golang
COPY main.go .
RUN go build main.go
CMD ["./main"]

The Haskell Dockerfile Linter supports a mode of operation that lets you point the hadolint executable to a Dockerfile on disk. You can see the command execution that follows, including the discovered warning messages produced by the analysis:

$ hadolint Dockerfile
Dockerfile:1 DL3006 warning: Always tag the version of an image explicitly
Dockerfile:2 DL3045 warning: `COPY` to a relative destination without \
`WORKDIR` set.

The output of the command suggests that you should assign a tag to the base image. The existing Dockerfile will pull the latest tag, which will resolve to the newest Go container image. This practice can result in an incompatibility between the Go runtime version and the application code. Defining a working directory for copying resources helps with keeping operating system-specific directories and files separate from application-specific directories and files. Example 6-13 fixes the warning messages found by the Haskell Dockerfile Linter.

FROM golang:1.19.4-alpine
WORKDIR /app
COPY main.go .
RUN go build main.go
CMD ["./main"]

Another execution against the modified Dockerfile leads to a successful exit code, and no additional warning messages will be rendered:

$ hadolint Dockerfile

The Dockerfile now follows best practices, as perceived by hadolint.