Kubernetes and Cloud Native Associate (KCNA)

KCNA is an entry-level certification program for anyone interested in cloud-native application development, runtime environments, and tooling. While the exam does cover Kubernetes, it does not expect you to interact with a cluster hands-on. This exam consists of multiple-choice questions and is suitable to candidates interested in the topic with a broad exposure to the ecosystem.

Kubernetes and Cloud Native Security Associate (KCSA)

This certification focuses on basic knowledge of security concepts and their application in a Kubernetes cluster. The breadth, depth, and format of the program is comparable to the KCNA.

Certified Kubernetes Application Developer (CKAD)

The CKAD exam focuses on verifying your ability to build, configure, and deploy a microservices-based application to Kubernetes. You are not expected to actually implement an application; however, the exam is suitable for developers familiar with topics like application architecture, runtimes, and programming languages.

Certified Kubernetes Administrator (CKA)

The target audience for the CKA exam are DevOps practitioners, system administrators, and site reliability engineers. This exam tests your ability to perform in the role of a Kubernetes administrator, which includes tasks like cluster, network, storage, and beginner-level security management, with emphasis on troubleshooting scenarios.

Certified Kubernetes Security Specialist (CKS)

The CKS exam expands on the topics verified by the CKA exam. Passing the CKA is a prerequisite before you can sign up for the CKS exam. For this certification, you are expected to have a deeper knowledge of Kubernetes security. The curriculum covers topics like applying best practices for building containerized applications and ensuring a secure Kubernetes runtime environment.

Application Design and Build

The first domain of the curriculum covers designing and building a containerized application and operating it in Kubernetes. You will need to be familiar with basic container concepts and how to define a container inside of a Pod. In addition, the domain covers more advanced use cases and Kubernetes concepts: the use of storage in Pods, the need for defining multiple containers inside of a Pod, and how to define and execute batch and periodic workloads.

Application Deployment

This domain primarily focuses on the Kubernetes primitive Deployment. A Deployment helps with scaling Pods with the same definition, so-called replicas, and managing the configuration across all replicas it controls. You need to understand managing deployments including strategies helpful for rolling out new versions of an application to replicas in a controlled fashion. Finally, you will need to be familiar with Helm, an open source tool for managing a set of manifests required to deploy and configure an application stack.

Application Observability and Maintenance

Deploying an application to Kubernetes is only the first step. You need to be able to monitor, inspect, and potentially debug Kubernetes objects. Probes are an important concept covered by this domain: they define health checks for applications running in a Pod. Furthermore, you need to be confident with identifying runtime issues for workload and how to fix them.

Application Environment, Configuration, and Security

Kubernetes offers security and resource management features configurable for a Pod. This includes the security context and resource requirements/constraints covered in this domain. Furthermore, you need to be able to demonstrate the use of ConfigMaps and Secrets to inject configuration data into a Pod to control its runtime behavior. The domain also touches on the rudimentary concepts and functionality of role-based access control (RBAC) and CustomResourceDefinitions (CRDs).

Services and Networking

The last domain of the curriculum deals with providing network access to your application from within and outside of the cluster. For that purpose, you’ll need to demonstrate knowledge of Services and Ingresses. Finally, you’ll need a rough understanding of network policies, which are essentially rules that deny or permit Pod-to-Pod communication.

Setting a Context and Namespace

The exam environment comes with multiple Kubernetes clusters already set up for you. Take a look at the instructions for a high-level, technical overview of those clusters. Each of the exam exercises needs to be solved on a designated cluster, as outlined in its description. Furthermore, the instructions will ask you to work in a namespace other than default. Make sure to set the context and namespace as the first course of action before working on a question. The following command sets the context and the namespace as a one-time action:

$ kubectl config set-context <context-of-question> \
  --namespace=<namespace-of-question>
$ kubectl config use-context <context-of-question>

You can find a more detailed discussion of the context concept and the corresponding kubectl commands in “Authentication with kubectl”.

Using the Alias for kubectl

In the course of the exam, you will have to execute the kubectl command tens or even hundreds of times. You might be an extremely fast typist; however, there’s no point in fully spelling out the executable over and over again. The exam environment already sets up the alias k for the kubectl command.

In preparation for the exam, you can set up the same behavior on your machine. The following alias command maps the letter k to the full kubectl command:

$ alias k=kubectl
$ k version

Using kubectl Command Auto-Completion

Memorizing kubectl commands and command-line options takes a lot of practice. The exam environment comes with auto-completion enabled by default. You can find instructions for setting up auto-completion for the shell on your machine in the Kubernetes documentation.

Internalize Resource Short Names

Many of the kubectl commands can be quite lengthy. For example, the command for managing Persistent volume claims is persistentvolumeclaims. Spelling out the full command can be error-prone and time-consuming. Thankfully, some of the longer commands come with a short-form usage. The command api-resources lists all available commands plus their short names:

$ kubectl api-resources
NAME                    SHORTNAMES  APIGROUP  NAMESPACED  KIND
...
persistentvolumeclaims  pvc                   true        PersistentVolumeClaim
...

Using pvc instead of persistentvolumeclaims results in a more concise and expressive command execution, as shown here:

$ kubectl describe pvc my-claim

Control Plane Node Components

The control plane node requires a specific set of components to perform its job. The following list of components will give you an overview:

API server

The API server exposes the API endpoints clients use to communicate with the Kubernetes cluster. For example, if you execute the tool kubectl, a command-line based Kubernetes client, you will make a RESTful API call to an endpoint exposed by the API server as part of its implementation. The API processing procedure inside of the API server will ensure aspects like authentication, authorization, and admission control. For more information on that topic, see Chapter 17.

Scheduler

The scheduler is a background process that watches for new Kubernetes Pods with no assigned nodes and assigns them to a worker node for execution.

Controller manager

The controller manager watches the state of your cluster and implements changes where needed. For example, if you make a configuration change to an existing object, the controller manager will try to bring the object into the desired state.

Etcd

Cluster state data needs to be persisted over time so it can be reconstructed upon a node or even a full cluster restart. That’s the responsibility of etcd, an open source software Kubernetes integrates with. At its core, etcd is a key-value store used to persist all data related to the Kubernetes cluster.

Common Node Components

Kubernetes employs components that are leveraged by all nodes independent of their specialized responsibility:

Kubelet

The kubelet runs on every node in the cluster; however, it makes the most sense to exist on a worker node. The reason is that the control plane node usually doesn’t execute workload, and the worker node’s primary responsibility is to run workload. The kubelet is an agent that makes sure that the necessary containers are running in a Pod. You could say that the kubelet is the glue between Kubernetes and the container runtime engine and ensures that containers are running and healthy. We’ll have a touch point with the kubelet in Chapter 14.

Kube proxy

The kube proxy is a network proxy that runs on each node in a cluster to maintain network rules and enable network communication. In part, this component is responsible for implementing the Service concept covered in Chapter 21.

Container runtime

As mentioned earlier, the container runtime is the software responsible for managing containers. Kubernetes can be configured to choose from a range of different container runtime engines. While you can install a container runtime engine on a control plane, it’s not necessary as the control plane node usually doesn’t handle workload. We’ll use a container runtime in Chapter 4 to create a container image and run a container with the produced image.

Imperative Object Management

Imperative object management does not require a manifest definition. You’ll use kubectl to drive the creation, modification, and deletion of objects with a single command and one or many command-line options. See the Kubernetes documentation for a more detailed description of imperative object management.

$ kubectl run frontend --image=nginx:1.24.0 --port=80
pod/frontend created

$ kubectl edit pod frontend

$ kubectl patch pod frontend -p '{"spec":{"containers":[{"name":"frontend",\
"image":"nginx:1.25.1"}]}}'
pod/frontend patched

$ kubectl delete pod frontend
pod "frontend" deleted

$ kubectl delete pod nginx --now

Declarative Object Management

Declarative object management requires one or several manifests in the format of YAML or JSON describing the desired state of an object. You create, update, and delete objects using this approach.

The benefit of using the declarative method is reproducibility and improved maintenance, as the file is checked into version control in most cases. The declarative approach is the recommended way to create objects in production environments.

More information on declarative object management can be found in the Kubernetes documentation.

.
├── app-stack
│   ├── mysql-pod.yaml
│   ├── mysql-service.yaml
│   ├── web-app-pod.yaml
│   └── web-app-service.yaml
├── nginx-deployment.yaml
└── web-app
    ├── config
    │   ├── db-configmap.yaml
    │   └── db-secret.yaml
    └── web-app-pod.yaml

$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created

$ kubectl apply -f app-stack/
pod/mysql-db created
service/mysql-service created
pod/web-app created
service/web-app-service created

$ kubectl apply -f web-app/ -R
configmap/db-config configured
secret/db-creds created
pod/web-app created

$ kubectl apply -f https://raw.githubusercontent.com/bmuschko/\
ckad-study-guide/master/ch03/object-management/nginx-deployment.yaml
deployment.apps/nginx-deployment created

$ kubectl get pod web-app -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{}, \
      "labels":{"app":"web-app"},"name":"web-app","namespace":"default"}, \
      "spec":{"containers":[{"envFrom":[{"configMapRef":{"name":"db-config"}}, \
      {"secretRef":{"name":"db-creds"}}],"image":"bmuschko/web-app:1.0.1", \
      "name":"web-app","ports":[{"containerPort":3000,"protocol":"TCP"}]}], \
      "restartPolicy":"Always"}}
...

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    team: red
spec:
  replicas: 5
...

$ kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment configured

$ kubectl get deployment nginx-deployment -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{}, \
      "labels":{"app":"nginx","team":"red"},"name":"nginx-deployment", \
      "namespace":"default"},"spec":{"replicas":5,"selector":{"matchLabels": \
      {"app":"nginx"}},"template":{"metadata":{"labels":{"app":"nginx"}}, \
      "spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx", \
      "ports":[{"containerPort":80}]}]}}}}
...

$ kubectl delete -f nginx-deployment.yaml
deployment.apps "nginx-deployment" deleted

Hybrid Approach

Sometimes you may want to go with a hybrid approach. You can start by using the imperative method to produce a manifest file without actually creating an object. You do so by executing the run or create command with the command-line options -o yaml and --dry-run=client:

$ kubectl run frontend --image=nginx:1.25.1 --port=80 \
  -o yaml --dry-run=client > pod.yaml

You can now use the generate YAML manifest as a starting point to make further modifications before creating the object. Simply open the file with an editor, change the content, and execute the declarative apply command:

$ vim pod.yaml
$ kubectl apply -f pod.yaml
pod/frontend created

Which Approach to Use?

During the exam, using imperative commands is the most efficient and quick way to manage objects. Not all configuration options are exposed through command-line flags, which may force you into using the declarative approach. The hybrid approach can help here.

While creating objects imperatively can optimize the turnaround time, in a real-world Kubernetes environment you’ll most certainly want to use the declarative approach. A YAML manifest file represents the ultimate source of truth of a Kubernetes object. Version-controlled files can be audited and shared, and they store a history of changes in case you need to revert to a previous revision.

Writing a Dockerfile

Before we can create the image, we have to write a Dockerfile. The Dockerfile can reside in any directory and is a plain-text file. The instructions that follow use the Azul JRE distribution of Java 21 as the base image. A base image contains the operating system and the necessary tooling, in this case Java.

Moreover, we include the binary file, an executable Java archive (JAR), into the directory /app of the image. Finally, we define the Java command that executes the program and expose the port 8080 to make the application accessible when run in a container. Example 4-1 outlines a sample Dockerfile.

FROM azul/zulu-openjdk:21-jre                                 
WORKDIR /app                                                  
COPY target/java-hello-world-0.0.1.jar java-hello-world.jar   
ENTRYPOINT ["java", "-jar", "/app/java-hello-world.jar"]      
EXPOSE 8080                                                   

Defines the base image.

Sets the working directory of a container. Any RUN, CMD, ADD, COPY, or ENTRYPOINT instruction will be executed in the specified working directory.

Copies the JAR containing the compiled application code into the working directory.

Sets the default command that executes when a container starts from an image.

Documents the network port(s) the container should listen on.

While writing Dockerfile looks straightforward to beginners, optimizing the container image for a small footprint and security aspects isn’t. You can find a more detailed list of best practices for writing Dockerfiles in the Docker documentation.

Building the Container Image

With the Dockerfile in place, we can create the image. The following command provides the name of the image and the tag. The last argument points to the context directory. A context directory contains the Dockerfile as well as any directories and files to be included in the image. Here, the context directory is the current directory we reside in referenced by “ . ”:

$ docker build -t java-hello-world:1.1.0 .
[+] Building 2.0s (9/9) FINISHED
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 284B
 => [internal] load metadata for docker.io/azul/zulu-openjdk:21-jre
 => [auth] azul/zulu-openjdk:pull token for registry-1.docker.io
 => [1/3] FROM docker.io/azul/zulu-openjdk:21-jre@sha256:d1e675cac0e5...
 => => resolve docker.io/azul/zulu-openjdk:21-jre@sha256:d1e675cac0e5...
 => => sha256:d1e675cac0e5ce9604283df2a6600d3b46328d32d83927320757ca7...
 => => sha256:67aa3090031eac26c946908c33959721730e42f9195f4f70409e4ce...
 => => sha256:ba408da684370e4d8448bec68b36fadf15c3819b282729df3bc8494...
 => [internal] load build context
 => => transferring context: 19.71MB
 => [2/3] WORKDIR /app
 => [3/3] COPY target/java-hello-world-0.0.1.jar java-hello-world.jar
 => exporting to image
 => => exporting layers
 => => writing image sha256:4b676060678b63de137536da24a889fc9d2d5fe0c...
 => => naming to docker.io/library/java-hello-world:1.1.0

What's Next?
  View a summary of image vulnerabilities and recommendations → ...

Listing Container Images

As indicated by the terminal output, the image has been created. You might have noticed that the base image has been downloaded as part of the process. The generated image can be found in your local Docker Engine cache by running the following command:

$ docker images
REPOSITORY         TAG       IMAGE ID       CREATED          SIZE
java-hello-world   1.1.0     4b676060678b   49 seconds ago   342MB

Running the Container

It’s time to run the application in a container. The run command points to an image and executes its logic in a container:

$ docker run -d -p 8080:8080 java-hello-world:1.1.0
b0ee04accf078ea7c73cfe3be0f9d1ac6a099ac4e0e903773bc6bf6258acbb66

We told the command to forward the port 8080 accessible on localhost to the container port 8080 using the -p CLI option. The -d CLI option runs the container in the background, which means it will detach from the container and return to the terminal prompt. This means we should now be able to resolve the application’s endpoint from the local machine. As the following command shows, a simple curl to the root context path renders the message “Hello World!”:

$ curl localhost:8080
Hello World!

Listing Containers

Any running containers can be listed to display their runtime properties. The following command renders the container started earlier. The output includes the container ID for later reference. Add the flag -a to render terminated containers as well:

$ docker container ls
CONTAINER ID   IMAGE                    COMMAND                  ...
b0ee04accf07   java-hello-world:1.1.0   "java -jar /app/java…"   ...

Interacting with the Container

Once the container has been started, you can interact with it. All you need is the container ID. Use the logs command to inspect log messages produced by the application. Inspecting logs can be helpful for troubleshooting. The following command renders the log messages produced by Spring Boot upon container startup:

$ docker logs b0ee04accf07
...
2023-06-19 21:06:27.757  INFO 1 --- [nio-8080-exec-1] \
o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing \
Spring DispatcherServlet 'dispatcherServlet'
2023-06-19 21:06:27.757  INFO 1 --- [nio-8080-exec-1] \
o.s.web.servlet.DispatcherServlet        : Initializing \
Servlet 'dispatcherServlet'
2023-06-19 21:06:27.764  INFO 1 --- [nio-8080-exec-1] \
o.s.web.servlet.DispatcherServlet        : Completed \
initialization in 7 ms

You can dig deeper into the internals of running containers if the container image is packaged with a command-line shell. For example, you may want to inspect files consumed or produced by the application. Use the exec command to run a command in the container. The flag -it allows for iterating with the container until you are ready to exit out of it. The following command opens an iterative bash shell to the running container:

$ docker exec -it b0ee04accf07 bash
root@b0ee04accf07:/app# pwd
/app
root@b0ee04accf07:/app# exit
exit

To leave the interactive bash shell, run the exit command. You’ll return to the terminal prompt on your host machine.

Publishing the Container Image

To publish an image to a registry, you’ll have to do some prework. Most registries require you to provide a prefix that signifies the username or hostname as part of the container image name, which you can achieve with the tag command.

For example, Docker Hub requires you to provide the username. My username is bmuschko and therefore I have to retag my image before pushing it:

$ docker tag java-hello-world:1.1.0 bmuschko/java-hello-world:1.1.0

The tag command does not create a copy of the container image. It simply adds another identifier pointing to the existing container image, as shown in the following output. The image ID and size of the container image is the same for both entries:

$ docker images
REPOSITORY                  TAG       IMAGE ID       CREATED         SIZE
bmuschko/java-hello-world   1.1.0     4b676060678b   6 minutes ago   342MB
java-hello-world            1.1.0     4b676060678b   6 minutes ago   342MB

If the registry is protected, you’ll also have to provide the credentials. For Docker Hub, we are logging in with username:

$ docker login --username=bmuschko
Password: *****
Login Succeeded

Finally, you can push the image to the registry using the push command:

$ docker push bmuschko/java-hello-world:1.1.0
The push refers to repository [docker.io/bmuschko/java-hello-world]
a7b86a39983a: Pushed
df1b2befe5f0: Pushed
e4db97f0e9ef: Mounted from azul/zulu-openjdk
8e87ff28f1b5: Mounted from azul/zulu-openjdk
1.1.0: digest: sha256:6a5069bd9396a7eded10bf8e24ab251df434c121f8f4293c2d3ef...

You can discover the published container image through the Docker Hub web page, as shown in Figure 4-2. The “Tags” tab lists all available tags for the image including their details and quick reference to the docker command for pulling the image.

Figure 4-2. Discovering container images on Docker Hub

Anyone with access to the registry can now consume the container image using the pull command.

Saving and Loading a Container Image

Instead of publishing a container image to a container registry, you may want to save it to a file. Files can be easily stored and backed up on a shared drive and don’t require a container registry. The save command saves one or many images to a tar archive. The resulting archive file contains all parent layers, and all tags + versions. The following command saves the container image to the file java-hello-world.tar:

$ docker save -o java-hello-world.tar java-hello-world:1.1.0

To load a container image from a tar archive, use the load command. The command restores both images and tags. The following command loads the container image from the file java-hello-world.tar:

$ docker load --input java-hello-world.tar
Loaded image: java-hello-world:1.1.0

The image is now available in the cache, as shown by running the images command:

$ docker images
REPOSITORY         TAG       IMAGE ID       CREATED         SIZE
java-hello-world   1.1.0     4b676060678b   7 minutes ago   342MB

Creating Pods

The Pod definition needs to state an image for every container. Upon creating the Pod object, imperatively or declaratively, the scheduler will assign the Pod to a node, and the container runtime engine will check if the container image already exists on that node. If the image doesn’t exist yet, the engine will download it from a container image registry. By default the registry is Docker Hub. As soon as the image exists on the node, the container is instantiated and will run. Figure 5-1 demonstrates the execution flow.

Figure 5-1. Container Runtime Interface interaction with container images

The run command is the central entry point for creating Pods imperatively. Let’s talk about its usage and the most important command line options you should memorize and practice. Say you wanted to run a Hazelcast instance inside of a Pod. The container should use the latest Hazelcast image, expose port 5701, and define an environment variable. In addition, we’ll also want to assign two labels to the Pod. The following imperative command combines this information and does not require any further editing of the live object:

$ kubectl run hazelcast --image=hazelcast/hazelcast:5.1.7 \
  --port=5701 --env="DNS_DOMAIN=cluster" --labels="app=hazelcast,env=prod"

The run command offers a wealth of command line options. Execute the kubectl run --help or refer to the Kubernetes documentation for a broad overview. For the exam, you’ll not need to understand every command. Table 5-1 lists the most commonly used options.

Some developers are more used to creating Pods from a YAML manifest. Probably you’re already accustomed to the declarative approach because you’re using it at work. You can express the same configuration for the Hazelcast Pod by opening the editor, copying a Pod YAML code snippet from the Kubernetes online documentation, and modifying it to your needs. Example 5-1 shows the Pod manifest saved in the file pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: hazelcast                      
  labels:                              
    app: hazelcast
    env: prod
spec:
  containers:
  - name: hazelcast
    image: hazelcast/hazelcast:5.1.7   
    env:                               
    - name: DNS_DOMAIN
      value: cluster
    ports:
    - containerPort: 5701              

Assigns the name of hazelcast to the Pod.

Specifies labels to the Pod.

Declares the container image to be executed in the container of the Pod.

Injects one or many environment variables to the container.

Number of port to expose on the Pod’s IP address.

Creating the Pod from the manifest is straightforward. Simply use the create or apply command, as shown here and explained in “Managing Objects”:

$ kubectl apply -f pod.yaml
pod/hazelcast created

Listing Pods

Now that you have created a Pod, you can further inspect its runtime information. The kubectl command offers a command for listing all Pods running in the cluster: get pods. The following command renders the Pod named hazelcast:

$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
hazelcast   1/1     Running   0          17s

Real-world Kubernetes clusters can run hundreds of Pods at the same time. If you know the name of the Pod of interest, it’s often easier to query by name. You would still see only a single Pod:

$ kubectl get pods hazelcast
NAME        READY   STATUS    RESTARTS   AGE
hazelcast   1/1     Running   0          17s

Pod Life Cycle Phases

Because Kubernetes is a state engine with asynchronous control loops, it’s possible that the status of the Pod doesn’t show a Running status right away when listing the Pods. It usually takes a couple of seconds to retrieve the image and start the container. Upon Pod creation, the object goes through several life cycle phases, as shown in Figure 5-2.

Figure 5-2. Pod life cycle phases

Understanding the implications of each phase is important as it gives you an idea about the operational status of a Pod. For example, during the exam you may be asked to identify a Pod with an issue and further debug the object. Table 5-2 describes all Pod life cycle phases.

The Pod life cycle phases should not be confused with container states within a Pod. Containers can have one of the three possible states: Waiting, Running, and Terminated. You can read more about container states in the Kubernetes documentation.

Rendering Pod Details

The rendered table produced by the get command provides high-level information about a Pod. But what if you needed a deeper look at the details? The describe command can help:

$ kubectl describe pods hazelcast
Name:               hazelcast
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               docker-desktop/192.168.65.3
Start Time:         Wed, 20 May 2020 19:35:47 -0600
Labels:             app=hazelcast
                    env=prod
Annotations:        <none>
Status:             Running
IP:                 10.1.0.41
Containers:
  ...
Events:
  ...

The terminal output contains the metadata information of a Pod, the containers it runs, and the event log, such as failures when the Pod was scheduled. The example output has been condensed to show just the metadata section. You can expect the output to be very lengthy.

There’s a way to be more specific about the information you want to render. You can combine the describe command with a Unix grep command if you want to identify the image for running in the container:

$ kubectl describe pods hazelcast | grep Image:
    Image:          hazelcast/hazelcast:5.1.7

Accessing Logs of a Pod

As application developers, we know very well what to expect in the log files produced by the application we implemented. Runtime failures may occur when operating an application in a container. The logs command downloads the log output of a container. The following output indicates that the Hazelcast server started up successfully:

$ kubectl logs hazelcast
...
May 25, 2020 3:36:26 PM com.hazelcast.core.LifecycleService
INFO: [10.1.0.46]:5701 [dev] [4.0.1] [10.1.0.46]:5701 is STARTED

It’s very likely that more log entries will be produced as soon as the container receives traffic from end users. You can stream the logs with the command line option -f. This option is helpful if you want to see logs in real time.

Kubernetes tries to restart a container under certain conditions, such as if the image cannot be resolved on the first try. Upon a container restart, you won’t have access to the logs of the previous container; the logs command renders the logs only for the current container. However, you can still get back to the logs of the previous container by adding the -p command line option. You may want to use the option to identify the root cause that triggered a container restart.

Executing a Command in Container

Some situations require you to get the shell to a running container and explore the filesystem. Maybe you want to inspect the configuration of your application or debug its current state. You can use the exec command to open a shell in the container to explore it interactively, as follows:

$ kubectl exec -it hazelcast -- /bin/sh
# ...

Notice that you do not have to provide the resource type. This command only works for a Pod. The two dashes (--) separate the exec command and its options from the command you want to run inside of the container.

It’s also possible to execute a single command inside of a container. Say you wanted to render the environment variables available to containers without having to be logged in. Just remove the interactive flag -it and provide the relevant command after the two dashes:

$ kubectl exec hazelcast -- env
...
DNS_DOMAIN=cluster

Creating a Temporary Pod

The command executed inside of a Pod—usually an application implementing business logic—is meant to run infinitely. Once the Pod has been created, it will stick around. Under certain conditions, you want to execute a command in a Pod just for troubleshooting. This use case doesn’t require a Pod object to run beyond the execution of the command. That’s where temporary Pods come into play.

The run command provides the flag --rm, which will automatically delete the Pod after the command running inside of it finishes. Say you want to render all environment variables using env to see what’s available inside of the container. The following command achieves exactly that:

$ kubectl run busybox --image=busybox:1.36.1 --rm -it --restart=Never -- env
...
HOSTNAME=busybox
pod "busybox" deleted

The last message rendered in the output clearly states that the Pod was deleted after command execution.

Using a Pod’s IP Address for Network Communication

Every Pod is assigned an IP address upon creation. You can inspect a Pod’s IP address by using the -o wide command-line option for the get pod command or by describing the Pod. The IP address of the Pod in the following console output is 10.244.0.5:

$ kubectl run nginx --image=nginx:1.25.1 --port=80
pod/nginx created
$ kubectl get pod nginx -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE       \
NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          37s   10.244.0.5   minikube   \
<none>           <none>
$ kubectl get pod nginx -o yaml
...
status:
  podIP: 10.244.0.5
...

The IP address assigned to a Pod is unique across all nodes and namespaces. This is achieved by assigning a dedicated subnet to each node when registering it. When creating a new Pod on a node, the IP address is leased from the assigned subnet. This is handled by the networking life cycle manager kube-proxy along with the Domain Name Service (DNS) and the Container Network Interface (CNI).

You can easily verify the behavior by creating a temporary Pod that calls the IP address of another Pod using the command-line tool curl or wget:

$ kubectl run busybox --image=busybox:1.36.1 --rm -it --restart=Never \
  -- wget 172.17.0.4:80
Connecting to 172.17.0.4:80 (172.17.0.4:80)
saving to 'index.html'
index.html           100% |********************************|   615  0:00:00 ETA
'index.html' saved
pod "busybox" deleted

It’s important to understand that the IP address is not considered stable over time. A Pod restart leases a new IP address. Therefore, this IP address is often referred to as virtual IP address. Building a microservices architecture—where each of the applications runs in its own Pod with the need to communicate between each other with a stable network interface—requires a different concept: the Service. Refer to Chapter 21 for more information.

Configuring Pods

The curriculum expects you to feel comfortable with editing YAML manifests either as files or as live object representations. This section shows you some typical configuration scenarios you may face during the exam. Later chapters will deepen your knowledge by touching on other configuration aspects.

apiVersion: v1
kind: Pod
metadata:
  name: spring-boot-app
spec:
  containers:
  - name: spring-boot-app
    image: bmuschko/spring-boot-app:1.5.3
    env:
    - name: SPRING_PROFILES_ACTIVE
      value: prod
    - name: VERSION
      value: '1.5.3'

$ kubectl run mypod --image=busybox:1.36.1 -o yaml --dry-run=client \
  > pod.yaml -- /bin/sh -c "while true; do date; sleep 10; done"

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: busybox:1.36.1
    args:
    - /bin/sh
    - -c
    - while true; do date; sleep 10; done

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: busybox:1.36.1
    command: ["/bin/sh"]
    args: ["-c", "while true; do date; sleep 10; done"]

$ kubectl apply -f pod.yaml
pod/mypod created
$ kubectl logs mypod -f
Fri May 29 00:49:06 UTC 2020
Fri May 29 00:49:16 UTC 2020
Fri May 29 00:49:26 UTC 2020
...

Deleting a Pod

Sooner or later you’ll want to delete a Pod. During the exam, you may be asked to remove a Pod. Or possibly, you made a configuration mistake and want to start the question from scratch:

$ kubectl delete pod hazelcast
pod "hazelcast" deleted

Keep in mind that Kubernetes tries to delete a Pod gracefully. This means that the Pod will try to finish active requests to the Pod to avoid unnecessary disruption to the end user. A graceful deletion operation can take anywhere from 5 to 30 seconds, time you don’t want to waste during the exam. See Chapter 1 for more information on how to speed up the process.

An alternative way to delete a Pod is to point the delete command to the YAML manifest you used to create it. The behavior is the same:

$ kubectl delete -f pod.yaml
pod "hazelcast" deleted

To save time during the exam, you can circumvent the grace period by adding the --now option to the delete command. Avoid using the --now flag in production Kubernetes environments.

Listing Namespaces

A Kubernetes cluster starts out with a couple of initial namespaces. You can list them with the following command:

$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   157d
kube-node-lease   Active   157d
kube-public       Active   157d
kube-system       Active   157d

The default namespace hosts objects that haven’t been assigned to an explicit namespace. Namespaces starting with the prefix kube- are not considered end user-namespaces. They have been created by the Kubernetes system. You will not have to interact with them as an application developer.

Creating and Using a Namespace

To create a new namespace, use the create namespace command. The following command uses the name code-red:

$ kubectl create namespace code-red
namespace/code-red created
$ kubectl get namespace code-red
NAME       STATUS   AGE
code-red   Active   16s

Example 5-5 shows the corresponding representation as a YAML manifest.

apiVersion: v1
kind: Namespace
metadata:
  name: code-red

Once the namespace is in place, you can create objects within it. You can do so with the command line option --namespace or its short-form -n. The following commands create a new Pod in the namespace code-red and then list the available Pods in the namespace:

$ kubectl run pod --image=nginx:1.25.1 -n code-red
pod/pod created
$ kubectl get pods -n code-red
NAME   READY   STATUS    RESTARTS   AGE
pod    1/1     Running   0          13s

Setting a Namespace Preference

Providing the --namespace or -n command line option for every command is tedious and error-prone. You can set a permanent namespace preference if you know that you need to interact with a specific namespace you are responsible for. The first command shown sets the permanent namespace code-red. The second command renders the currently set permanent namespace:

$ kubectl config set-context --current --namespace=code-red
Context "minikube" modified.
$ kubectl config view --minify | grep namespace:
    namespace: hello

Subsequent kubectl executions do not have to spell out the namespace code-red:

$ kubectl get pods
NAME   READY   STATUS    RESTARTS   AGE
pod    1/1     Running   0          13s

You can always switch back to the default namespace or another custom namespace using the config set-context command:

$ kubectl config set-context --current --namespace=default
Context "minikube" modified.

Deleting a Namespace

Deleting a namespace has a cascading effect on the object existing in it. Deleting a namespace will automatically delete its objects:

$ kubectl delete namespace code-red
namespace "code-red" deleted
$ kubectl get pods -n code-red
No resources found in code-red namespace.

Creating and Inspecting Jobs

Let’s first create a Job and observe its behavior in practice before delving into details. To create a Job imperatively, simply use the create job command. If the provided image doesn’t run any commands, you may want to append a command to be executed in the corresponding Pod.

The following command creates a Job that runs an iteration process. For each iteration of the loop, a variable named counter is incremented. The command execution finishes after reaching the counter value 3:

$ kubectl create job counter --image=nginx:1.25.1 \
  -- /bin/sh -c 'counter=0; while [ $counter -lt 3 ]; do \
  counter=$((counter+1)); echo "$counter"; sleep 3; done;'
job.batch/counter created

Example 6-1 shows the YAML manifest equivalent for the Job if you prefer the declarative approach.

apiVersion: batch/v1
kind: Job
metadata:
  name: counter
spec:
  template:   
    spec:
      containers:
      - name: counter
        image: nginx:1.25.1
        command:
        - /bin/sh
        - -c
        - counter=0; while [ $counter -lt 3 ]; do counter=$((counter+1)); \
          echo "$counter"; sleep 3; done;
      restartPolicy: Never

The Pod template uses the same attributes available in a Pod definition.

The output of listing the Job shows the current number of completions and the expected number of completions. The default number of completions is 1. This means if the Pod executing the command was successful, a Job is considered completed. As you can see in the following terminal output, a Job uses a single Pod by default to perform the work. The corresponding Pod can be identified by name—it uses the Job name as a prefix in its own name:

$ kubectl get jobs
NAME      COMPLETIONS   DURATION   AGE
counter   0/1           13s        13s
$ kubectl get jobs
NAME      COMPLETIONS   DURATION   AGE
counter   1/1           15s        19s
$ kubectl get pods
NAME            READY   STATUS      RESTARTS   AGE
counter-z6kdj   0/1     Completed   0          51s

To verify the correct behavior of the Job, you can download its logs. As expected, the output renders the counter for each iteration:

$ kubectl logs counter-z6kdj
1
2
3

You can further tweak the runtime behavior of a Job. The next two sections discuss configuring the Job operation types and restart behavior.

Job Operation Types

The default behavior of a Job is to run the workload in a single Pod and expect one successful completion. That’s what Kubernetes calls a non-parallel Job. Internally, those parameters are tracked by the attributes spec.completions and spec.parallelism, each with the assigned value 1. The following command renders the parameters of the Job we created earlier:

$ kubectl get jobs counter -o yaml | grep -C 1 "completions"
...
  completions: 1
  parallelism: 1
...

You can tweak any of those parameters to fit the needs of your use case. For example, if you expected the workload to complete successfully multiple times, then you’d increase the value of spec.completions to at least 2. Sometimes, you’ll want to execute the workload by multiple pods in parallel. In those cases, you’d bump up the value assigned to spec.parallelism. This is referred to as a parallel job. Remember that you can use any combination of assigned values for both attributes. Table 6-1 summarizes the different use cases.

Restart Behavior

The spec.backoffLimit attribute determines the number of retries a Job attempts to successfully complete the workload until the executed command finishes with an exit code 0. The default is 6, which means it will execute the workload 6 times before the Job is considered unsuccessful.

The Job manifest needs to explicitly declare the restart policy by using spec.​tem⁠plate.spec.restartPolicy. The default restart policy of a Pod is Always, which tells the Kubernetes scheduler to always restart the Pod even if the container exits with a 0 exit code. The restart policy of a Job can be only OnFailure or Never.

Restarting the container on failure

Figure 6-2 shows the behavior of a Job that uses the restart policy OnFailure. Upon a container failure, this policy will simply rerun the container.

Figure 6-2. Restart policy onFailure

Starting a new Pod on failure

Figure 6-3 shows the behavior of a Job that uses the restart policy Never. This policy does not restart the container upon a failure. It starts a new Pod instead.

Figure 6-3. Restart policy Never

Creating and Inspecting CronJobs

You can use the imperative create cronjob command to create a new CronJob. The following command schedules the CronJob to run every minute. The Pod created for every execution renders the current date to standard output using the Unix echo command:

$ kubectl create cronjob current-date --schedule="* * * * *" \
  --image=nginx:1.25.1 -- /bin/sh -c 'echo "Current date: $(date)"'
cronjob.batch/current-date created

To create a CronJob from the YAML manifest, use the definition shown in Example 6-2.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: current-date
spec:
  schedule: "* * * * *"   
  jobTemplate:            
    spec:
      template:
        spec:
          containers:
          - name: current-date
            image: nginx:1.25.1
            args:
            - /bin/sh
            - -c
            - 'echo "Current date: $(date)"'
          restartPolicy: OnFailure

Defines the cron expression that determines when a new Job object needs to be created.

The section that describes the Job template.

If you list the existing CronJob with the get cronjobs command, you will see the schedule, the last scheduled execution, and whether the CronJob is currently active:

$ kubectl get cronjobs
NAME           SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
current-date   * * * * *   False     0        <none>          28s
$ kubectl get cronjobs
NAME           SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
current-date   * * * * *   False     1        14s             53s

It’s easy to match Jobs and Pods created by a CronJob. You can simply identify them by the name prefix. In this case, the prefix is current-date-:

$ kubectl get jobs,pods
NAME                              COMPLETIONS   DURATION   AGE
job.batch/current-date-28473049   1/1           3s         2m23s
job.batch/current-date-28473050   1/1           3s         83s
job.batch/current-date-28473051   1/1           3s         23s

NAME                              READY   STATUS      RESTARTS   AGE
pod/current-date-28473049-l6hc7   0/1     Completed   0          2m23s
pod/current-date-28473050-csq7n   0/1     Completed   0          83s
pod/current-date-28473051-jg8st   0/1     Completed   0          23s

Configuring Retained Job History

Even after a task in a Pod controlled by a CronJob completes, it will not be deleted automatically. Keeping a historical record of Pods can be tremendously helpful for troubleshooting failed workloads or inspecting the logs. By default, a CronJob retains the last three successful Pods and the last failed Pod:

$ kubectl get cronjobs current-date -o yaml | grep successfulJobsHistoryLimit:
  successfulJobsHistoryLimit: 3
$ kubectl get cronjobs current-date -o yaml | grep failedJobsHistoryLimit:
  failedJobsHistoryLimit: 1

To reconfigure the job retention history limits, set new values for the spec.successfulJobsHistoryLimit and spec.failedJobsHistoryLimit attributes. Example 6-3 keeps the last five successful executions and the last three failed executions.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: current-date
spec:
  successfulJobsHistoryLimit: 5   
  failedJobsHistoryLimit: 3       
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: current-date
            image: nginx:1.25.1
            args:
            - /bin/sh
            - -c
            - 'echo "Current date: $(date)"'
          restartPolicy: OnFailure

Defines the number of successful Jobs kept in the history.

Defines the number of failed Jobs kept in the history.

Volume Types

Every Volume needs to define a type. The type determines the medium that backs the Volume and its runtime behavior.

Ephemeral Volumes

These exist for the lifespan of a Pod. Ephemeral Volumes are useful if you want to share data between multiple containers running in the Pod or if you can easily reconstruct the data stored on the Volume upon a Pod restart.

Persistent Volumes

These preserve data beyond the lifespan of a Pod. Persistent Volumes are a good option for applications that require data to exist longer, for example, in the form of storage for a database-driven application.

The Kubernetes documentation offers a long list of Volume types. Table 7-1 provides a select list of Volume types that I have found to be most relevant to the exam.

Ephemeral Volumes

Defining an ephemeral Volume for a Pod requires two steps. First, you need to declare the Volume itself using the attribute spec.volumes[]. As part of the definition, you provide the name and the type. Just declaring the Volume won’t be sufficient. Second, the Volume needs to be mounted to a path of the consuming container via spec.containers[].volumeMounts[]. The mapping between the Volume and the Volume mount occurs through the matching name.

apiVersion: v1
kind: Pod
metadata:
  name: business-app
spec:
  volumes:
  - name: logs-volume
    emptyDir: {}
  containers:
  - image: nginx:1.25.1
    name: nginx
    volumeMounts:
    - mountPath: /var/log/nginx
      name: logs-volume

$ kubectl create -f pod-with-volume.yaml
pod/business-app created
$ kubectl get pod business-app
NAME           READY   STATUS    RESTARTS   AGE
business-app   1/1     Running   0          43s
$ kubectl exec business-app -it -- /bin/sh
# cd /var/log/nginx
# pwd
/var/log/nginx
# ls
# touch app-logs.txt
# ls
app-logs.txt

Persistent Volumes

Data stored on Volumes outlives a container restart. In many applications, the data lives far beyond the life cycles of the applications, container, Pod, nodes, and even the clusters themselves. Data persistence ensures the life cycles of the data are decoupled from the life cycles of the cluster resources. A typical example would be data persisted by a database. That’s the responsibility of a persistent Volume. Kubernetes models persist data with the help of two primitives: the PersistentVolume and the PersistentVolumeClaim.

The PersistentVolume is the primitive representing a piece of storage in a Kubernetes cluster. It is completely decoupled from the Pod and therefore has its own life cycle. The object captures the source of the storage (e.g., storage made available by a cloud provider). A PersistentVolume is either provided by a Kubernetes administrator or assigned dynamically by mapping to a storage class.

The PersistentVolumeClaim requests the resources of a PersistentVolume—for example, the size of the storage and the access type. In the Pod, you will use the type persistentVolumeClaim to mount the abstracted PersistentVolume by using the PersistentVolumeClaim.

Figure 7-2 shows the relationship between the Pod, the PersistentVolumeClaim, and the PersistentVolume.

Figure 7-2. Claiming a PersistentVolume from a Pod

apiVersion: v1
kind: PersistentVolume
metadata:
  name: db-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/db

$ kubectl create -f db-pv.yaml
persistentvolume/db-pv created
$ kubectl get pv db-pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    \
  CLAIM   STORAGECLASS   REASON   AGE
db-pv   1Gi        RWO            Retain           Available \
                                  10s

$ kubectl get pv -o wide
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    \
CLAIM   STORAGECLASS   REASON   AGE   VOLUMEMODE
db-pv   1Gi        RWO            Retain           Available \
                                19m   Filesystem

$ kubectl get pv db-pv -o jsonpath='{.spec.accessModes}'
["ReadWriteOnce"]

$ kubectl get pv db-pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}'
Retain

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: db-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ""
  resources:
    requests:
      storage: 256Mi

$ kubectl create -f db-pvc.yaml
persistentvolumeclaim/db-pvc created
$ kubectl get pvc db-pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
db-pvc   Bound    db-pv    1Gi        RWO                           111s

$ kubectl describe pvc db-pvc
...
Used By:       <none>
...

apiVersion: v1
kind: Pod
metadata:
  name: app-consuming-pvc
spec:
  volumes:
  - name: app-storage
    persistentVolumeClaim:
      claimName: db-pvc
  containers:
  - image: alpine:3.18.2
    name: app
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 60; done;"]
    volumeMounts:
      - mountPath: "/mnt/data"
        name: app-storage

$ kubectl create -f app-consuming-pvc.yaml
pod/app-consuming-pvc created
$ kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
app-consuming-pvc   1/1     Running   0          3s
$ kubectl describe pod app-consuming-pvc
...
Volumes:
  app-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim \
                in the same namespace)
    ClaimName:  db-pvc
    ReadOnly:   false
...

$ kubectl describe pvc db-pvc
...
Used By:       app-consuming-pvc
...

$ kubectl exec app-consuming-pvc -it -- /bin/sh
/ # cd /mnt/data
/mnt/data # ls -l
total 0
/mnt/data # touch test.db
/mnt/data # ls -l
total 0
-rw-r--r--    1 root     root             0 Sep 29 23:59 test.db

Storage Classes

A storage class is a Kubernetes primitive that defines a specific type or “class” of storage. Typical storage characteristics can be the type (e.g., fast SSD storage versus remote cloud storage or the backup policy for storage). The storage class is used to provision a PersistentVolume dynamically based on its criteria. In practice, this means that you do not have to create the PersistentVolume object yourself. The provisioner assigned to the storage class takes care of it. Most Kubernetes cloud providers come with a list of existing provisioners. Minikube already creates a default storage class named standard, which you can query with the following command:

$ kubectl get storageclass
NAME                 PROVISIONER                RECLAIMPOLICY \
  VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete        \
  Immediate           false                  108d

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  replication-type: regional-pd

$ kubectl create -f fast-sc.yaml
storageclass.storage.k8s.io/fast created
$ kubectl get storageclass
NAME                 PROVISIONER                RECLAIMPOLICY \
  VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
fast                 kubernetes.io/gce-pd       Delete        \
  Immediate           false                  4s
...

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: db-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 512Mi
  storageClassName: standard

$ kubectl get pv,pvc
NAME                                                       CAPACITY \
  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM           STORAGECLASS \
  REASON  AGE
persistentvolume/pvc-b820b919-f7f7-4c74-9212-ef259d421734   512Mi \
    RWO           Delete          Bound   default/db-pvc  standard \
                  2s

NAME                          STATUS  VOLUME                                  \
CAPACITY  ACCESS MODES  STORAGECLASS  AGE
persistentvolumeclaim/db-pvc  Bound   pvc-b820b919-f7f7-4c74-9212-ef259d421734 \
512Mi     RWO           standard      2s

Init Containers

Init containers provide initialization logic concerns to be run before the main application starts. To draw an analogy, let’s look at a similar concept in programming languages. Many programming languages, especially object-oriented ones like Java or C++, come with a constructor or a static method block. Those language constructs initialize fields, validate data, and set the stage before a class can be created. Not all classes need a constructor, but they are equipped with the capability.

In Kubernetes, this functionality can be achieved with the help of init containers. Init containers are always started before the main application containers, which means they have their own life cycle. To split up the initialization logic, you can even distribute the work into multiple init containers that are run in the order of definition in the manifest. Of course, initialization logic can fail. If an init container produces an error, the whole Pod is restarted, causing all init containers to run again in sequential order. Thus, to prevent any side effects, making init container logic idempotent is a good practice. Figure 8-1 shows a Pod with two init containers and the main application.

Figure 8-1. Sequential and atomic life cycle of init containers in a Pod

In the past couple of chapters, we’ve explored how to define a container within a Pod: you simply specify its configuration under spec.containers. For init containers, Kubernetes provides a separate section: spec.initContainers. Init containers are always executed before the main application containers, regardless of the definition order in the manifest.

The manifest shown in Example 8-1 defines an init container and a main application container. For the most part, init containers use the same attributes as regular containers. There’s one big difference, however. They cannot define probes, discussed in Chapter 14. The init container sets up a configuration file in the directory /usr/shared/app. This directory has been shared through a Volume so that it can be referenced by a Node.js-based application running in the main container.

apiVersion: v1
kind: Pod
metadata:
  name: business-app
spec:
  initContainers:
  - name: configurer
    image: busybox:1.36.1
    command: ['sh', '-c', 'echo Configuring application... && \
              mkdir -p /usr/shared/app && echo -e "{\"dbConfig\": \
              {\"host\":\"localhost\",\"port\":5432,\"dbName\":\"customers\"}}" \
              > /usr/shared/app/config.json']
    volumeMounts:
    - name: configdir
      mountPath: "/usr/shared/app"
  containers:
  - image: bmuschko/nodejs-read-config:1.0.0
    name: web
    ports:
    - containerPort: 8080
    volumeMounts:
    - name: configdir
      mountPath: "/usr/shared/app"
  volumes:
  - name: configdir
    emptyDir: {}

When starting the Pod, you’ll see that the status column of the get command provides information on init containers as well. The prefix Init: signifies that an init container is in the process of being executed. The status portion after the colon character shows the number of init containers completed versus the overall number of init containers configured:

$ kubectl create -f init.yaml
pod/business-app created
$ kubectl get pod business-app
NAME           READY   STATUS    RESTARTS   AGE
business-app   0/1     Init:0/1  0          2s
$ kubectl get pod business-app
NAME           READY   STATUS    RESTARTS   AGE
business-app   1/1     Running   0          8s

Errors can occur during the execution of init containers. If any container fails in the sequential initialization chain, then the whole Pod will fail to start. You can always retrieve the logs of an init container by using the --container command-line option (or -c in its short form), as shown in Figure 8-2.

Figure 8-2. Targeting a specific container

The following command renders the logs of the configurer init container, which equates to the echo command we configured in the YAML manifest:

$ kubectl logs business-app -c configurer
Configuring application...

The Sidecar Pattern

The life cycle of an init container looks like this: it starts up, runs its logic, then terminates once the work has been done. Init containers are not meant to keep running over a longer period of time. But some scenarios call for a different usage pattern. For example, you may want to create a Pod that runs multiple containers continuously alongside one another.

Typically, there are two different categories of containers: the container that runs the application and another container that provides helper functionality to the primary application. In the Kubernetes space, the container providing helper functionality is called a sidecar. The most commonly used capabilities of a sidecar container include file synchronization, logging, and watcher capabilities. The sidecars are not part of the main traffic or API of the primary application. They usually operate asynchronously and are not involved in the public API.

To illustrate the behavior of a sidecar, consider the following use case. The main application container runs a web server—in this case, NGINX. Once started, the web server produces two standard logfiles. The file /var/log/nginx/access.log captures requests to the web server’s endpoint. The other file, /var/log/nginx/error.log, records failures while processing incoming requests.

As part of the Pod’s functionality, we want to implement a monitoring service. The sidecar container polls the file’s error.log periodically and checks if any failures have been discovered. More specifically, the service tries to find failures assigned to the error log level, indicated by [error] in the log file. If an error is found, the monitoring service will react to it. For example, it could send a notification to the system administrators. We want to keep the functionality as simple as possible. The monitoring service will simply render an error message to standard output. The file exchange between the main application container and the sidecar container happens through a Volume, as shown in Figure 8-3.

Figure 8-3. The sidecar pattern in action

The YAML manifest shown in Example 8-2 sets up the described scenario. The trickiest portion of the code is the lengthy bash command. This command runs an infinite loop. As part of each iteration, we inspect the contents of the file error.log, grep for an error and potentially act on it. The loop executes every 10 seconds.

apiVersion: v1
kind: Pod
metadata:
  name: webserver
spec:
  containers:
  - name: nginx
    image: nginx:1.25.1
    volumeMounts:
    - name: logs-vol
      mountPath: /var/log/nginx
  - name: sidecar
    image: busybox:1.36.1
    command: ["sh","-c","while true; do if [ \"$(cat /var/log/nginx/error.log \
              | grep 'error')\" != \"\" ]; then echo 'Error discovered!'; fi; \
              sleep 10; done"]
    volumeMounts:
    - name: logs-vol
      mountPath: /var/log/nginx
  volumes:
  - name: logs-vol
    emptyDir: {}

When starting up the Pod, you’ll notice that the overall number of containers will show 2. After all containers can be started, the Pod signals a Running status:

$ kubectl create -f sidecar.yaml
pod/webserver created
$ kubectl get pods webserver
NAME        READY   STATUS              RESTARTS   AGE
webserver   0/2     ContainerCreating   0          4s
$ kubectl get pods webserver
NAME        READY   STATUS    RESTARTS   AGE
webserver   2/2     Running   0          5s

You will find that error.log does not contain any failure to begin with. It starts out as an empty file. With the following commands, you’ll provoke an error on purpose. After waiting for at least 10 seconds, you’ll find the expected message on the terminal, which you can query for with the logs command:

$ kubectl logs webserver -c sidecar
$ kubectl exec webserver -it -c sidecar -- /bin/sh
/ # wget -O- localhost?unknown
Connecting to localhost (127.0.0.1:80)
wget: server returned error: HTTP/1.1 404 Not Found
/ # cat /var/log/nginx/error.log
2020/07/18 17:26:46 [error] 29#29: *2 open() "/usr/share/nginx/html/unknown" \
failed (2: No such file or directory), client: 127.0.0.1, server: localhost, \
request: "GET /unknown HTTP/1.1", host: "localhost"
/ # exit
$ kubectl logs webserver -c sidecar
Error discovered!

The Adapter Pattern

As application developers, we want to focus on implementing business logic. For example, as part of a two-week sprint, we’re tasked with adding a shopping cart feature. In addition to the functional requirements, we also have to think about operational aspects such as exposing administrative endpoints or crafting meaningful and properly formatted log output. It’s easy to fall into the habit of rolling all aspects into the application code, which makes it more complex and harder to maintain. Cross-cutting concerns in particular need to be replicated across multiple applications and are often copied and pasted from one code base to another.

In Kubernetes, we can avoid bundling cross-cutting concerns into the application code by running them in another container apart from the main application container. The adapter pattern transforms the output produced by the application to make it consumable in the format needed by another part of the system. Figure 8-4 illustrates a concrete example of the adapter pattern.

Figure 8-4. The adapter pattern in action

The business application running the main container produces timestamped information—in this case, the available disk space—and writes it to the file diskspace.txt. As part of the architecture, we want to consume the file from a third-party monitoring application. The problem is that the external application requires the information to exclude the timestamp. Of course, we could change the logging format to avoid writing the timestamp, but what if we actually want to know when the log entry has been written? This is where the adapter pattern can help. A sidecar container executes transformation logic that turns the log entries into the format needed by the external system without having to change application logic.

The YAML manifest in Example 8-3 illustrates what this implementation of the adapter pattern could look like. The app container produces a new log entry every five seconds. The transformer container consumes the contents of the file, removes the timestamp, and writes it to a new file. Both containers have access to the same mount path through a Volume.

apiVersion: v1
kind: Pod
metadata:
  name: adapter
spec:
  containers:
  - args:
    - /bin/sh
    - -c
    - 'while true; do echo "$(date) | $(du -sh ~)" >> /var/logs/diskspace.txt; \
       sleep 5; done;'
    image: busybox:1.36.1
    name: app
    volumeMounts:
      - name: config-volume
        mountPath: /var/logs
  - image: busybox:1.36.1
    name: transformer
    args:
    - /bin/sh
    - -c
    - 'sleep 20; while true; do while read LINE; do echo "$LINE" | cut -f2 -d"|" \
       >> $(date +%Y-%m-%d-%H-%M-%S)-transformed.txt; done < \
       /var/logs/diskspace.txt; sleep 20; done;'
    volumeMounts:
    - name: config-volume
      mountPath: /var/logs
  volumes:
  - name: config-volume
    emptyDir: {}

After creating the Pod, we’ll find two running containers. We should be able to locate the original file, /var/logs/diskspace.txt, after shelling into the transformer container. The transformed data exists in a separate file in the user home directory:

$ kubectl create -f adapter.yaml
pod/adapter created
$ kubectl get pods adapter
NAME      READY   STATUS    RESTARTS   AGE
adapter   2/2     Running   0          10s
$ kubectl exec adapter --container=transformer -it -- /bin/sh
/ # cat /var/logs/diskspace.txt
Sun Jul 19 20:28:07 UTC 2020 | 4.0K	/root
Sun Jul 19 20:28:12 UTC 2020 | 4.0K	/root
/ # ls -l
total 40
-rw-r--r--  1  root  root  60 Jul 19 20:28 2020-07-19-20-28-28-transformed.txt
...
/ # cat 2020-07-19-20-28-28-transformed.txt
 4.0K	/root
 4.0K	/root

The Ambassador Pattern

Another important design pattern covered by the CKAD is the ambassador pattern. The ambassador pattern provides a proxy for communicating with external services.

Many use cases can justify the introduction of the ambassador pattern. The overarching goal is to hide and/or abstract the complexity of interacting with other parts of the system. Typical responsibilities include retry logic upon a request failure, security concerns such as providing authentication or authorization, and monitoring latency or resource usage. Figure 8-5 illustrates this pattern.

Figure 8-5. The ambassador pattern in action

In this example, we’ll want to implement rate-limiting functionality for HTTP(S) calls to an external service. For example, the requirements for the rate limiter could say that an application can make a maximum of 5 calls every 15 minutes. Instead of strongly coupling the rate-limiting logic to the application code, it will be provided by an ambassador container. Any calls made from the business application need to be funneled through the ambassador container. Example 8-4 shows a Node.js-based rate limiter implementation that makes calls to the external service Postman.

const express = require('express');
const app = express();
const rateLimit = require('express-rate-limit');
const https = require('https');

const rateLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 5,
  message:
    'Too many requests have been made from this IP, please try again after an hour'
});

app.get('/test', rateLimiter, function (req, res) {
  console.log('Received request...');
  var id = req.query.id;
  var url = 'https://postman-echo.com/get?test=' + id;
  console.log("Calling URL %s", url);

  https.get(url, (resp) => {
    let data = '';

    resp.on('data', (chunk) => {
      data += chunk;
    });

    resp.on('end', () => {
      res.send(data);
    });

    }).on("error", (err) => {
      res.send(err.message);
    });
})

var server = app.listen(8081, function () {
  var port = server.address().port
  console.log("Ambassador listening on port %s...", port)
})

The corresponding Pod shown in Example 8-5 runs the main application container on a different port than the ambassador container. Every call to the HTTP endpoint of the container named business-app would delegate to the HTTP endpoint of the container named ambassador. It’s important to mention that containers running inside of the same Pod can communicate via localhost. No additional networking configuration is required.

apiVersion: v1
kind: Pod
metadata:
  name: rate-limiter
spec:
  containers:
  - name: business-app
    image: bmuschko/nodejs-business-app:1.0.0
    ports:
    - containerPort: 8080
  - name: ambassador
    image: bmuschko/nodejs-ambassador:1.0.0
    ports:
    - containerPort: 8081

Let’s test the functionality. First, we’ll create the Pod, shell into the container that runs the business application, and execute a series of curl commands. The first five calls will be allowed to the external service. On the sixth call, we’ll receive an error message, as the rate limit has been reached within the given time frame:

$ kubectl create -f ambassador.yaml
pod/rate-limiter created
$ kubectl get pods rate-limiter
NAME           READY   STATUS    RESTARTS   AGE
rate-limiter   2/2     Running   0          5s
$ kubectl exec rate-limiter -it -c business-app -- /bin/sh
# curl localhost:8080/test
{"args":{"test":"123"},"headers":{"x-forwarded-proto":"https", \
"x-forwarded-port":"443","host":"postman-echo.com", \
"x-amzn-trace-id":"Root=1-5f177dba-e736991e882d12fcffd23f34"}, \
"url":"https://postman-echo.com/get?test=123"}
...
# curl localhost:8080/test
Too many requests have been made from this IP, please try again after an hour

Declaring Labels

You can declare labels imperatively with the run command or declaratively in the metadata.labels section in the YAML manifest. The command-line option --labels (or -l in its short form) defines a comma-separated list of labels when creating a Pod. The following command creates a new Pod with two labels from the command line:

$ kubectl run labeled-pod --image=nginx:1.25.1 \
  --labels=tier=backend,env=dev
pod/labeled-pod created

Assigning labels to Kubernetes objects by editing the manifest requires a change to the metadata section. Example 9-1 shows the Pod definition from the previous command if we were to start with the YAML manifest.

apiVersion: v1
kind: Pod
metadata:
  name: labeled-pod
  labels:
    env: dev
    tier: backend
spec:
  containers:
  - image: nginx:1.25.1
    name: nginx

Inspecting Labels

You can inspect the labels assigned to a Kubernetes object from different angles. Here, we’ll want to look at the most common ways to identify the labels of a Pod. As with any other runtime information, you can use the describe or get commands to retrieve the labels:

$ kubectl describe pod labeled-pod | grep -C 2 Labels:
...
Labels:       env=dev
              tier=backend
...
$ kubectl get pod labeled-pod -o yaml | grep -C 1 labels:
metadata:
  labels:
    env: dev
    tier: backend
...

If you want to list the labels for all object types or a specific object type, use the --show-labels command-line option. This option is convenient if you need to sift through a longer list of objects. The output automatically adds a new column named LABELS:

$ kubectl get pods --show-labels
NAME          READY   STATUS    RESTARTS   AGE   LABELS
labeled-pod   1/1     Running   0          38m   env=dev,tier=backend

Modifying Labels for a Live Object

You can add or remove a label from an existing Kubernetes object, or simply modify an existing label at any time. One way to achieve this is by editing the live object and changing the label definition in the metadata.labels section. The other option, which offers a slightly faster turnaround, is the label command. The following commands add a new label, change the value of the label, and then remove the label with the minus character:

$ kubectl label pod labeled-pod region=eu
pod/labeled-pod labeled
$ kubectl get pod labeled-pod --show-labels
NAME          READY   STATUS    RESTARTS   AGE   LABELS
labeled-pod   1/1     Running   0          22h   env=dev,region=eu,tier=backend
$ kubectl label pod labeled-pod region=us --overwrite
pod/labeled-pod labeled
$ kubectl get pod labeled-pod --show-labels
NAME          READY   STATUS    RESTARTS   AGE   LABELS
labeled-pod   1/1     Running   0          22h   env=dev,region=us,tier=backend
$ kubectl label pod labeled-pod region-
pod/labeled-pod labeled
$ kubectl get pod labeled-pod --show-labels
NAME          READY   STATUS    RESTARTS   AGE   LABELS
labeled-pod   1/1     Running   0          22h   env=dev,tier=backend

Using Label Selectors

Labels become meaningful only when combined with the selection feature. A label selector uses a set of criteria to query for Kubernetes objects. For example, you could use a label selector to express “select all Pods with the label assignment env=dev, tier=frontend, and have a label with the key version independent of the assigned value,” as shown in Figure 9-2.

Figure 9-2. Selecting Pods by label criteria

Kubernetes offers two ways to select objects by labels: from the command line and within a manifest. Let’s talk about both options.

$ kubectl run frontend --image=nginx:1.25.1 --labels=env=prod,team=shiny
pod/frontend created
$ kubectl run backend --image=nginx:1.25.1 --labels=env=prod,team=legacy,\
app=v1.2.4
pod/backend created
$ kubectl run database --image=nginx:1.25.1 --labels=env=prod,team=storage
pod/database created
$ kubectl get pods --show-labels
NAME       READY   STATUS    RESTARTS   AGE   LABELS
backend    1/1     Running   0          37s   app=v1.2.4,env=prod,team=legacy
database   1/1     Running   0          32s   env=prod,team=storage
frontend   1/1     Running   0          42s   env=prod,team=shiny

$ kubectl get pods -l env=prod --show-labels
NAME       READY   STATUS    RESTARTS   AGE   LABELS
backend    1/1     Running   0          37s   app=v1.2.4,env=prod,team=legacy
database   1/1     Running   0          32s   env=prod,team=storage
frontend   1/1     Running   0          42s   env=prod,team=shiny

$ kubectl get pods -l 'team in (shiny, legacy)' --show-labels
NAME       READY   STATUS    RESTARTS   AGE   LABELS
backend    1/1     Running   0          19m   app=v1.2.4,env=prod,team=legacy
frontend   1/1     Running   0          20m   env=prod,team=shiny

$ kubectl get pods -l 'team in (shiny, legacy)',app=v1.2.4 --show-labels
NAME      READY   STATUS    RESTARTS   AGE   LABELS
backend   1/1     Running   0          29m   app=v1.2.4,env=prod,team=legacy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-network-policy
spec:
  podSelector:
    matchLabels:
      tier: frontend
...

Recommended Labels

As you continue working with labels, you will likely find common key-value pairs you want to assign to objects. Many of those labels evolve around metadata for an application, for example, the name of the component you are deploying with a Pod, or its version.

Kubernetes proposes a list of recommended labels, all of which start with the key prefix app.kubernetes.io. Example 9-3 shows the assignment of version and component labels in a Pod definition.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app.kubernetes.io/version: "1.25.1"
    app.kubernetes.io/component: server
spec:
  containers:
  - name: nginx
    image: nginx:1.25.1

Familiarize yourself with these recommended labels so you can use them across all objects you are managing. This provides the benefit of enabling tooling in the Kubernetes ecosystem to use the same terminology and enabling developers to use the same “language” when referring to application meta information.

Declaring Annotations

The kubectl run command does not provide a command-line option for defining annotations that’s similar to the one for labels. You will have to start by writing a YAML manifest and adding the desired annotations under metadata.annotations, as shown in Example 9-4.

apiVersion: v1
kind: Pod
metadata:
  name: annotated-pod
  annotations:
    commit: 866a8dc
    author: 'Benjamin Muschko'
    branch: 'bm/bugfix'
spec:
  containers:
  - image: nginx:1.25.1
    name: nginx

Inspecting Annotations

Similar to labels, you can use the describe or get commands to retrieve the assigned annotations:

$ kubectl describe pod annotated-pod | grep -C 2 Annotations:
...
Annotations:  author: Benjamin Muschko
              branch: bm/bugfix
              commit: 866a8dc
...
$ kubectl get pod annotated-pod -o yaml | grep -C 3 annotations:
metadata:
  annotations:
    author: Benjamin Muschko
    branch: bm/bugfix
    commit: 866a8dc
...

Modifying Annotations for a Live Object

The annotate command is the counterpart of the labels command but used for annotations. As you can see in the following examples, the usage pattern is the same:

$ kubectl annotate pod annotated-pod oncall='800-555-1212'
pod/annotated-pod annotated
$ kubectl annotate pod annotated-pod oncall='800-555-2000' --overwrite
pod/annotated-pod annotated
$ kubectl annotate pod annotated-pod oncall-
pod/annotated-pod annotated

Reserved Annotations

Kubernetes itself and extensions to Kubernetes use the concept of annotations to configure runtime behavior for an object. For example, you can assign the reserved annotation pod-security.kubernetes.io/enforce: "baseline" to a namespace object to enforce security standards for all Pods that live in the namespace. Example 9-5 shows a namespace definition that assigns the annotation.

apiVersion: v1
kind: Namespace
metadata:
  name: secured
  annotations:
    pod-security.kubernetes.io/enforce: "baseline"

See the Kubernetes documentation for a full list of reserved annotations in Kubernetes.

Creating Deployments

You can create a Deployment using the imperative command create deployment. The command offers a range of options, some of which are mandatory. At a minimum, you need to provide the name of the Deployment and the container image. The Deployment passes this information to the ReplicaSet, which uses it to manage the replicas. The default number of replicas created is 1; however, you can define a higher number of replicas using the option --replicas.

Let’s observe the command in action. The following command creates the Deployment named app-cache, which runs the object cache Memcached inside the container on four replicas:

$ kubectl create deployment app-cache --image=memcached:1.6.8 --replicas=4
deployment.apps/app-cache created

The mapping between the Deployment and the replicas it controls happens through label selection. When you run the imperative command, kubectl sets up the mapping for you. Example 10-1 shows the label selection in the YAML manifest. This YAML manifest can be used to create a Deployment declaratively or by inspecting the live object created by the previous imperative command.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-cache
  labels:
    app: app-cache
spec:
  replicas: 4
  selector:
    matchLabels:
      app: app-cache
  template:
    metadata:
      labels:
        app: app-cache
    spec:
      containers:
      - name: memcached
        image: memcached:1.6.8

When created by the imperative command, app is the label key the Deployment uses by default. You can find this key in three different places in the YAML output:

1. metadata.labels

2. spec.selector.matchLabels

3. spec.template.metadata.labels

For label selection to work properly, the assignment of spec.selector.matchLabels and spec.template.metadata needs to match, as shown in Figure 10-2.

The values of metadata.labels is irrelevant for mapping the Deployment to the Pod template. As you can see in the figure, the label assignment to metadata.labels has been changed deliberately to deploy: app-cache to underline that it is not important for the Deployment to Pod template selection.

Figure 10-2. Deployment label selection

Listing Deployments and Their Pods

You can inspect a Deployment after its creation by using the get deployments command. The output of the command renders the important details of its replicas, as shown here:

$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
app-cache   4/4     4            4           125m

The column titles relevant to the replicas controlled by the Deployment are shown in Table 10-1.

You can identify the Pods controlled by the Deployment by their naming prefix. In the case of the previously created Deployment, the Pods’ names start with app-cache-. The hash following the prefix is autogenerated and appended to the name upon creation:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
app-cache-596bc5586d-84dkv   1/1     Running   0          6h5m
app-cache-596bc5586d-8bzfs   1/1     Running   0          6h5m
app-cache-596bc5586d-rc257   1/1     Running   0          6h5m
app-cache-596bc5586d-tvm4d   1/1     Running   0          6h5m

Rendering Deployment Details

You can render the details of a Deployment. Those details include the label selection criteria, which can be extremely valuable when troubleshooting a misconfigured Deployment. The following output provides the full gist:

$ kubectl describe deployment app-cache
Name:                   app-cache
Namespace:              default
CreationTimestamp:      Sat, 07 Aug 2021 09:44:18 -0600
Labels:                 app=app-cache
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=app-cache
Replicas:               4 desired | 4 updated | 4 total | 4 available | \
                        0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=app-cache
  Containers:
   memcached:
    Image:        memcached:1.6.10
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   app-cache-596bc5586d (4/4 replicas created)
Events:          <none>

You might have noticed that the output contains a reference to a ReplicaSet. The purpose of a ReplicaSet is to replicate a set of identical Pods. You do not need to deeply understand the core functionality of a ReplicaSet for the exam. Just be aware that the Deployment automatically creates the ReplicaSet and uses the Deployment’s name as a prefix for the ReplicaSet, similar to the Pods it controls. In the case of the previous Deployment named app-cache, the name of the ReplicaSet is app-⁠cache-​596bc5586d.

Deleting a Deployment

A Deployment takes full charge of the creation and deletion of the objects it controls: Pods and ReplicaSets. When you delete a Deployment, the corresponding objects are deleted as well. Say you are dealing with the following set of objects shown in the output:

$ kubectl get deployments,pods,replicasets
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-cache   4/4     4            4           6h47m

NAME                             READY   STATUS    RESTARTS   AGE
pod/app-cache-596bc5586d-84dkv   1/1     Running   0          6h47m
pod/app-cache-596bc5586d-8bzfs   1/1     Running   0          6h47m
pod/app-cache-596bc5586d-rc257   1/1     Running   0          6h47m
pod/app-cache-596bc5586d-tvm4d   1/1     Running   0          6h47m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/app-cache-596bc5586d   4         4         4       6h47m

Run the delete deployment command for a cascading deletion of its managed objects:

$ kubectl delete deployment app-cache
deployment.apps "app-cache" deleted
$ kubectl get deployments,pods,replicasets
No resources found in default namespace.

Updating a Deployment’s Pod Template

You can choose from a range of options to update the definition of replicas controlled by a Deployment. Any of those options is valid, but they vary in ease of use and operational environment.

In real-world projects, you should check your manifest files into version control. Changes to the definition would then be made by directly editing the file. The kubectl apply can update a live object by pointing to the changed manifest:

$ kubectl apply -f deployment.yaml

The kubectl edit command lets you change the Pod template interactively by modifying the live object’s manifest in an editor. To edit the Deployment live object named web-server, use the following command:

$ kubectl edit deployment web-server

The imperative kubectl set image command changes only the container image assigned to a Pod template by selecting the name of the container. For example, you could use his command to assign the image nginx:1.25.2 to the container named nginx in the Deployment web-server:

$ kubectl set image deployment web-server nginx=nginx:1.25.2

The kubectl replace command lets you replace the existing Deployment with a new definition that contains your change to the manifest. The optional --force flag first deletes the existing object and then creates it from scratch. The following command assumes that you changed the container image assignment in deployment.yaml:

$ kubectl replace -f deployment.yaml

The command kubectl patch requires you to provide the merges as a patch to update a Deployment. The following command shows the operation in action. Here, you are sending the changes to be made in the form of a JSON structure:

$ kubectl patch deployment web-server -p '{"spec":{"template":{"spec":\
{"containers":[{"name":"nginx","image":"nginx:1.25.2"}]}}}}'

Rolling Out a New Revision

Deployments make it easy to roll out a new version of the application to all replicas it controls. Say you want to upgrade the version of Memcached from 1.6.8 to 1.6.10 to benefit from the latest features and bug fixes. All you need to do is change the desired state of the object by updating the Pod template. The Deployment updates all replicas to the new version one by one. This process is called the rolling update strategy.

The command set image offers a quick, convenient way to change the image of a Deployment, as shown in the following command:

$ kubectl set image deployment app-cache memcached=memcached:1.6.10
deployment.apps/app-cache image updated

You can check the current status of a rollout that’s in progress using the command rollout status. The output indicates the number of replicas that have already been updated since emitting the command:

$ kubectl rollout status deployment app-cache
Waiting for rollout to finish: 2 out of 4 new replicas have been updated...
deployment "app-cache" successfully rolled out

Kubernetes keeps track of the changes you make to a Deployment over time in the rollout history. Every change is represented by a revision. When changing the Pod template of a Deployment—for example, by updating the image—the Deployment triggers the creation of a new ReplicaSet. The Deployment will gradually migrate the Pods from the old ReplicaSet to the new one. You can check the rollout history by running the following command. You will see two revisions listed:

$ kubectl rollout history deployment app-cache
deployment.apps/app-cache
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

The first revision was recorded for the original state of the Deployment when you created the object. The second revision was added for changing the image tag.

To get a more detailed view of the revision, run the following command. You can see that the image uses the value memcached:1.6.10:

$ kubectl rollout history deployments app-cache --revision=2
deployment.apps/app-cache with revision #2
Pod Template:
  Labels:	app=app-cache
	pod-template-hash=596bc5586d
  Containers:
   memcached:
    Image:	memcached:1.6.10
    Port:	<none>
    Host Port:	<none>
    Environment:	<none>
    Mounts:	<none>
  Volumes:	<none>

The rolling update strategy ensures that the application is always available to end users. This approach implies that two versions of the same application are available during the update process. As an application developer, you have to be aware that convenience doesn’t come without potential side effects. If you happen to introduce a breaking change to the public API of your application, you might temporarily break consumers, as they could hit revision 1 or 2 of the application.

You can change the default update strategy of a Deployment by providing a different value to the attribute spec.strategy.type; however, consider the trade-offs. For example, the value Recreate kills all Pods first, then creates new Pods with the latest revision, causing potential downtime for consumers. See Chapter 11 for a more detailed description of common deployment strategies.

Adding a Change Cause for a Revision

The rollout history renders the column CHANGE-CAUSE. You can populate the information for a revision to document why you introduced a new change or which kubectl command you use to make the change.

By default, changing the Pod template does not automatically record a change cause. To add a change cause to the current revision, add an annotation with the reserved key kubernetes.io/change-cause to the Deployment object. The following imperative annotate command assigns the change cause “Image updated to 1.6.10”:

$ kubectl annotate deployment app-cache kubernetes.io/change-cause=\
"Image updated to 1.6.10"
deployment.apps/app-cache annotated

The rollout history now renders the change cause value for the current revision:

$ kubectl rollout history deployment app-cache
deployment.apps/app-cache
REVISION  CHANGE-CAUSE
1         <none>
2         Image updated to 1.6.10

Rolling Back to a Previous Revision

Problems can arise in production that require swift action. For example, the container image you just rolled out contains a crucial bug. Kubernetes gives you the option to roll back to one of the previous revisions in the rollout history. You can achieve this by using the rollout undo command. To pick a specific revision, provide the command-line option --to-revision. The command rolls back to the previous revision if you do not provide the option. Here, we are rolling back to revision 1:

$ kubectl rollout undo deployment app-cache --to-revision=1
deployment.apps/app-cache rolled back

As a result, Kubernetes performs a rolling update to all replicas with the revision 1.

The rollout history now lists revision 3. Given that we rolled back to revision 1, there’s no more need to keep that entry as a duplicate. Kubernetes simply turns revision 1 into 3 and removes 1 from the list:

$ kubectl rollout history deployment app-cache
deployment.apps/app-cache
REVISION  CHANGE-CAUSE
2         Image updated to 1.16.10
3         <none>

Manually Scaling a Deployment

Scaling (up or down) the number of replicas controlled by a Deployment is a straightforward process. You can either manually edit the live object using the edit deployment command and change the value of the attribute spec.replicas, or you can use the imperative scale deployment command. In real-world production environments, you want to edit the Deployment YAML manifest, check it into version control, and apply the changes. The following command increases the number of replicas from four to six:

$ kubectl scale deployment app-cache --replicas=6
deployment.apps/app-cache scaled

You can observe the creation of replicas in real time using the -w command line flag. You’ll see a change of status for the newly created Pods turning from Container​Creat⁠ing to Running:

$ kubectl get pods -w
NAME                         READY   STATUS              RESTARTS   AGE
app-cache-5d6748d8b9-6cc4j   1/1     ContainerCreating   0          11s
app-cache-5d6748d8b9-6rmlj   1/1     Running             0          28m
app-cache-5d6748d8b9-6z7g5   1/1     ContainerCreating   0          11s
app-cache-5d6748d8b9-96dzf   1/1     Running             0          28m
app-cache-5d6748d8b9-jkjsv   1/1     Running             0          28m
app-cache-5d6748d8b9-svrxw   1/1     Running             0          28m

Manually scaling the number of replicas takes a bit of guesswork. You will still have to monitor the load on your system to see if your number of replicas is sufficient to handle the incoming traffic.

Autoscaling a Deployment

Another way to scale a Deployment is with the help of a Horizontal Pod Autoscaler (HPA). The HPA is an API primitive that defines rules for automatically scaling the number of replicas under certain conditions. Common scaling conditions include a target value, an average value, or an average utilization of a specific metric (e.g., for CPU and/or memory). Refer to the MetricTarget API for more information.

Let’s say you want to define average CPU utilization of CPU as the scaling condition. At runtime, the HPA checks the metrics collected by the metrics server to determine if the average maximum CPU or memory usage across all replicas of a Deployment is less than or greater than the defined threshold. Make sure that you have the metrics server installed in the cluster. Collecting metrics may take a couple of minutes initially after installing the component. See “Inspecting Resource Metrics” for more information.

Figure 10-3 shows an overview architecture diagram involving an HPA.

Figure 10-3. Autoscaling a Deployment

Creating Horizontal Pod Autoscalers

Figure 10-4 shows the use of an HPA that will scale up the number of replicas if an average of 80% CPU utilization is reached across all available Pods controlled by the Deployment.

Figure 10-4. Autoscaling a Deployment horizontally

You can use the autoscale deployment command to create an HPA for an existing Deployment. The option --cpu-percent defines the average maximum CPU usage threshold. At the time of writing, the imperative command doesn’t offer an option for defining the average maximum memory utilization threshold. The options --min and --max provide the minimum number of replicas to scale down to and the maximum number of replicas the HPA can create to handle the increased load, respectively:

$ kubectl autoscale deployment app-cache --cpu-percent=80 --min=3 --max=5
horizontalpodautoscaler.autoscaling/app-cache autoscaled

This command is a great shortcut for creating an HPA for a Deployment. The YAML manifest representation of the HPA object looks like Example 10-2.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-cache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-cache
  minReplicas: 3
  maxReplicas: 5
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource

Listing Horizontal Pod Autoscalers

The short-form command for a Horizontal Pod Autoscaler is hpa. Listing all of the HPA objects transparently describes their current state: the CPU utilization and the number of replicas at this time:

$ kubectl get hpa
NAME        REFERENCE              TARGETS         MINPODS   MAXPODS   REPLICAS \
  AGE
app-cache   Deployment/app-cache   <unknown>/80%   3         5         4        \
  58s

If the Pod template of the Deployment does not define CPU resource requirements or if the CPU metrics cannot be retrieved from the metrics server, the left-side value of the column TARGETS says <unknown>. Example 10-3 sets the resource requirements for the Pod template so that the HPA can work properly. You can learn more about defining resource requirements in “Working with Resource Requirements”.

# ...
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - name: memcached
        # ...
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m

Once traffic hits the replicas, the current CPU usage is shown as a percentage. Here the average maximum CPU utilization is 15%:

$ kubectl get hpa
NAME        REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
app-cache   Deployment/app-cache   15%/80%   3         5         4          58s

Rendering Horizontal Pod Autoscaler Details

The event log of an HPA can provide additional insight into the rescaling activities. Rendering the HPA details can be a great tool for overseeing when the number of replicas was scaled up or down, as well as their scaling conditions:

$ kubectl describe hpa app-cache
Name:                                                  app-cache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 15 Aug 2021 \
                                                       15:54:11 -0600
Reference:                                             Deployment/app-cache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 80%
Min replicas:                                          3
Max replicas:                                          5
Deployment pods:                                       3 current / 3 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully \
  calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less \
  than the minimum replica count
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  13m   horizontal-pod-autoscaler  New size: 3; \
  reason: All metrics below target

Defining Multiple Scaling Metrics

You can define more than a single resource type as a scaling metric. As you can see in Example 10-4, we are inspecting CPU and memory utilization to determine if the replicas of a Deployment need to be scaled up or down.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-cache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-cache
  minReplicas: 3
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 500Mi

To ensure that the HPA determines the currently used resources, we’ll set the memory resource requirements for the Pod template as well, as shown in Example 10-5.

...
spec:
  ...
  template:
    ...
    spec:
      containers:
      - name: memcached
        ...
        resources:
          requests:
            cpu: 250m
            memory: 100Mi
          limits:
            cpu: 500m
            memory: 500Mi

Listing the HPA renders both metrics in the TARGETS column, as in the output of the get command shown here:

$ kubectl get hpa
NAME        REFERENCE              TARGETS                 MINPODS   MAXPODS \
  REPLICAS   AGE
app-cache   Deployment/app-cache   1994752/500Mi, 0%/80%   3         5       \
  3          2m14s

Implementation

A Deployment uses the rolling deployment strategy by default. The runtime value for the attribute spec.strategy.type is RollingUpdate. Users can fine-tune this strategy. You can use the attributes spec.strategy.rollingUpdate.maxUnavailable and spec.strategy.rollingUpdate.maxSurge to change the rollout rate. Both attributes can use a fixed integer (for example, 3) or assign a percentage of the total required number of Pods (for example, 33%). The default value for maxUnavailable and maxSurge is 25%.

The attribute maxUnavailable specifies the maximum number of Pods that can be unavailable during the update process. For example, if you set the value to 40%, then the old ReplicaSet can scale down to 60% immediately when the rolling update starts.

The attribute maxSurge specifies the maximum number of Pods that can be created over the desired number of Pods. For example, if you set the value to 10%, the total number of new and old Pods cannot exceed a total of 110% after the new ReplicaSet has been created.

Independent of the values assigned to the attributes maxUnavailable and maxSurge, all replicas controlled by the old ReplicaSet will be ramped down to 0 over time until all replicas controlled by the new ReplicaSet equals the value of spec.replicas.

It’s recommended to define a readiness probe for the Pod template to ensure that a replica is ready to handle incoming requests. The attribute spec.minReadySeconds specifies the number of seconds a replica needs to be available for before it is made available to incoming requests.

Example 11-1 shows the usage of those attributes in the context of a full Deployment YAML manifest stored in the file deployment-rolling-update.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 40%   
      maxSurge: 10%         
  minReadySeconds: 60       
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.4.23-alpine
        ports:
        - containerPort: 80
          protocol: TCP
        readinessProbe:     
          httpGet:
            path: /
            port: 80

The percentage of Pods that can be unavailable during the update.

The percentage of Pods that can temporarily exceed the total number of replicas.

The number of seconds for which the readiness probe in a Pod needs to be healthy until the rollout process can continue.

The readiness probe for all replicas referred to by spec.minReadySeconds.

The combination of assigned values to maxUnavailable and maxSurge determines the runtime behavior and speed of a rollout. You will adjust those parameters to find the most suitable combination for your application.

Use Cases and Trade-Offs

The rolling deployment is a fitting deployment strategy for rolling out a new application version with zero downtime. Depending on the number of replicas the Deployment manages, this process can be relatively slow, as old versions of the applications are ramped down and new versions of the application are ramped up in batches.

It’s important to mention that this deployment strategy comes with a potential risk. Old and new versions of the application run in parallel. Breaking changes introduced with the new version can lead to unexpected and hard-to-debug errors for consumers if they haven’t adapted their client software to the latest changes. It’s a good idea to roll out a new application version in a backward-compatible fashion, for example by using a versioned API, to avoid running into this situation.

Implementation

To configure the fixed deployment strategy for a Deployment, set the attribute spec.strategy.type to Recreate. Internally, this strategy type will automatically assign the total number of replicas to the attribute maxUnavailable. No other configuration options need to be provided.

Example 11-2 shows the Recreate strategy type in the context of a full Deployment YAML manifest stored in the file deployment-fixed.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
spec:
  replicas: 4
  strategy:
    type: Recreate   
  selector:
    matchLabels:
      app: httpd
  template:
    metadata:
      labels:
        app: httpd
    spec:
      containers:
      - name: httpd
        image: httpd:2.4.23-alpine
        ports:
        - containerPort: 80
          protocol: TCP

The strategy type for configuring fixed deployment.

Assigning a readiness probe to containers defined by the Pod template isn’t strictly necessary because all replicas with the old application version will be shut down at once. Nevertheless, it still makes sense to verify that the application is up and running by defining a readiness probe before incoming traffic can reach the container.

Use Cases and Trade-Offs

The fixed deployment strategy is suitable for situations where application downtime is acceptable. For example, it’s great if you want to roll out a new application to a developer environment for testing purposes. For production environments, this deployment strategy may work if you announce an outage time window to customers.

Implementation

Blue-green deployment is not a built-in strategy you can configure within the Deployment resource. You will have to create a Deployment object for both application versions. The Service routes traffic to replicas managed by either the blue or the green Deployment.

Example 11-3 shows a blue Deployment YAML manifest stored in the file deployment-blue.yaml specifying the container image httpd:2.4.23-alpine in the Pod template.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server-blue
spec:
  replicas: 4
  selector:
    matchLabels:
      type: blue
  template:
    metadata:
      labels:
        type: blue                   
    spec:
      containers:
      - name: httpd
        image: httpd:2.4.23-alpine   
        ports:
        - containerPort: 80
          protocol: TCP

Uses the label assignment type: blue to any replica managed by the corresponding ReplicaSet.

The old application version 2.4.23-alpine.

To set up a green deployment that runs the newer container image httpd:2.4.57-alpine, simply create another Deployment object. Note that the label used for the Pod template is different than for the blue deployment. Example 11-4 shows the green Deployment definition in the file deployment-green.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server-green
spec:
  replicas: 4
  selector:
    matchLabels:
      type: green
  template:
    metadata:
      labels:
        type: green                  
    spec:
      containers:
      - name: httpd
        image: httpd:2.4.57-alpine   
        ports:
        - containerPort: 80
          protocol: TCP

Uses the label assignment type: green to any replica managed by the corresponding ReplicaSet.

The new application version 2.4.57-alpine.

As mentioned earlier, the Service is the Kubernetes object responsible for routing network traffic to the old or new application version. Example 11-5 shows a Service object.

apiVersion: v1
kind: Service
metadata:
  name: web-server
spec:
  selector:
    type: blue   
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

The label selector pointing to replicas managed by the blue deployment.

The resource declaration currently points to the blue deployment; to switch to green, simply change the label selection from type: blue to type: green. At that point, you can delete the blue Deployment.

Use Cases and Trade-Offs

The blue-green deployment strategy is suitable for deployment scenarios where complex upgrades need to be performed without downtime to consumers. This situation may arise if a rollout requires a data migration or if multiple, dependent software components need to be changed at once. Should a rollback to the old application version be required, a simple change of the label selection in the Service will do.

On the downside, it’s worth mentioning that you will need more hardware resources than for other deployment strategies. If you need five replicas to run the old application version, then you will need the same amount of resources for the new application version, assuming the resource requirements won’t differ.

Implementation

In a Kubernetes cluster, you represent each application version with the help of a Deployment object. You want to roll out the new application version with fewer replicas than the current application version by assigning a smaller value to the attribute spec.replicas.

The following code snippet shows the truncated definition of the Deployment controlling the old application version:

kind: Deployment
spec:
  replicas: 4
  selector:
    matchLabels:
      app: httpd

For the new application version, assign a smaller number of replicas:

kind: Deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpd

To ensure that replicas for old and new application versions receive requests from consumers, the assigned Pod template label(s) in both Deployment objects need to be the same. Ensure that the Service selects those label(s), as shown in this truncated Service definition:

kind: Service
spec:
  selector:
    app: httpd   

Selects the label assigned to both Deployments.

Use Cases and Trade-Offs

Organization typically use the canary deployment strategy to roll out experimental features or changes with potential impact on system performance. You can evaluate your success criteria while making the new feature available to only a subset of consumers. Implementing a canary deployment usually requires fewer hardware resources than the blue-green deployment as the number of replicas with the new application version is much lower.

Identifying a Chart

Over the years, the Kubernetes community implemented and published thousands of Helm charts. Artifact Hub provides a web-based search capability for discovering charts by keyword.

Say you wanted to find a chart that installs the Continuous Integration solution Jenkins. All you’d need to do is to enter the term “jenkins” into the search box and press the enter key. Figure 12-1 shows the list of results in Artifact Hub.

Figure 12-1. Searching for a Jenkins chart on Artifact Hub

At the time of writing, there are 141 matches for the search term. You will be able to inspect details about the chart by clicking on one of the search results, which includes a high-level description and the repository that the chart file resides in. Moreover, you can inspect the templates bundled with the chart file, indicating the objects that will be created upon installation and their configuration options. Figure 12-2 shows the page for the official Jenkins chart.

Figure 12-2. Jenkins chart details

You cannot install a chart directly from Artifact Hub. You must install it from the repository hosting the chart file.

Adding a Chart Repository

The chart description may mention the repository that hosts the chart file. Alternatively, you can click on the “Install” button to render repository details and the command for adding it. Figure 12-3 shows the contextual pop-up that appears after clicking the “Install” button.

By default, a Helm installation defines no external repositories. The following command shows how to list all registered repositories. No repositories have been registered yet:

$ helm repo list
Error: no repositories to show

Figure 12-3. Jenkins chart installation instructions

As you can see from the screenshot, the chart file lives in the repository with the URL https://charts.jenkins.io. We will need to add this repository. This is an one-time operation. You can install other charts from that repository or you can update a chart that originated from that repository with commands we’ll discuss in a later section.

You need to provide a name for the repository when registering one. Make the repository name as descriptive as possible. The following command registers the repository with the name jenkinsci:

$ helm repo add jenkinsci https://charts.jenkins.io/
"jenkinsci" has been added to your repositories

Listing the repositories now shows the mapping between name and URL:

$ helm repo list
NAME        URL
jenkinsci   https://charts.jenkins.io/

You permanently added the repository to the Helm installation.

Searching for a Chart in a Repository

The “Install” pop-up window already provided the command to install the chart. You can also search the repository for available charts in case you do not know their names or latest versions. Add the --versions flag to list all available versions:

$ helm search repo jenkinsci
NAME                CHART VERSION   APP VERSION   DESCRIPTION
jenkinsci/jenkins   4.6.5           2.414.2       ...

The latest version available is 4.6.5. This may be different if you run the command on your machine, given that the Jenkins project may have released a newer version.

Installing a Chart

Let’s assume that the latest version of the Helm chart contains a security vulnerability. Therefore, we decide to install the Jenkins chart with the previous version, 4.6.4. You need to assign a name to be able to identify an installed chart. The name we’ll use here is my-jenkins:

$ helm install my-jenkins jenkinsci/jenkins --version 4.6.4
NAME: my-jenkins
LAST DEPLOYED: Thu Sep 28 09:47:21 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
...

The chart automatically created the Kubernetes objects in the default namespace. You can use the following command to discover the most important resource types:

$ kubectl get all
NAME               READY   STATUS    RESTARTS   AGE
pod/my-jenkins-0   2/2     Running   0          12m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP    ...
service/my-jenkins         ClusterIP   10.99.166.189    <none>         ...
service/my-jenkins-agent   ClusterIP   10.110.246.141   <none>         ...

NAME                          READY   AGE
statefulset.apps/my-jenkins   1/1     12m

The chart has been installed with the default configuration options. You can inspect those default values by clicking on the “Default Values” button on the chart page, as shown in Figure 12-4.

Figure 12-4. Jenkins chart default values

You can also discover those configuration options using the following command. The output shown renders only a subset of values, the admin username and its password, represented by controller.adminUser and controller.adminPassword:

$ helm show values jenkinsci/jenkins
...
controller:
  # When enabling LDAP or another non-Jenkins identity source, the built-in \
  # admin account will no longer exist.
  # If you disable the non-Jenkins identity store and instead use the Jenkins \
  # internal one,
  # you should revert controller.adminUser to your preferred admin user:
  adminUser: "admin"
  # adminPassword: <defaults to random>
...

You can customize any configuration value when installing the chart. To pass configuration data during the install processing use one of the following flags:

• --values: Specifies the overrides in the form of a pointer to a YAML manifest file.

• --set: Specifies the overrides directly from the command line.

For more information, see “Customizing the Chart Before Installing” in the Helm documentation.

You can decide to install the chart into a custom namespace. Use the -n flag to provide the name of an existing namespace. Add the flag --create-namespace to automatically create the namespace if it doesn’t exist yet.

The following command shows how to customize some of the values and the namespace used during the installation process:

$ helm install my-jenkins jenkinsci/jenkins --version 4.6.4 \
--set controller.adminUser=boss --set controller.adminPassword=password \
-n jenkins --create-namespace

We specifically set the username and the password for the admin user. Helm created the objects controlled by the chart into the jenkins namespace.

Listing Installed Charts

Charts can live in the default namespace or a custom namespace. You can inspect the list of installed charts using the helm list command. If you do not know which namespace, simply add the --all-namespaces flag to the command:

$ helm list --all-namespaces
NAME         NAMESPACE   REVISION   UPDATED         STATUS     CHART
my-jenkins   default     1          2023-09-28...   deployed   jenkins-4.6.4

The output of the command includes the column NAMESPACE that shows the namespace used by a particular chart. Similar to the use of kubectl, the helm list command provides the option -n for spelling out a namespace. Providing no flag(s) with the command will return the result for the default namespace.

Upgrading an Installed Chart

Upgrading an installed chart usually means moving to a new chart version. You can poll for new versions available in the repository by running this command:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jenkinsci" chart repository
Update Complete. ⎈Happy Helming!⎈

What if you want to upgrade your existing chart installation to a newer chart version? Run the following command to upgrade the chart to that specific version with the default configuration:

$ helm upgrade my-jenkins jenkinsci/jenkins --version 4.6.5
Release "my-jenkins" has been upgraded. Happy Helming!
...

As with the install command, you will have to provide custom configuration values if you want to tweak the chart’s runtime behavior when upgrading a chart.

Uninstalling a Chart

Sometimes you no longer need to run a chart. The command for uninstalling a chart is straightforward, as shown here. It will delete all objects controlled by the chart. Don’t forgot to provide the -n flag if you previously installed the chart into a namespace other than default:

$ helm uninstall my-jenkins
release "my-jenkins" uninstalled

Executing the command may take up to 30 seconds, as Kubernetes needs to wait for the workload grace period to end.

Probe Types

Kubernetes provides a concept called health probing to automate the detection and correction of such issues. You can configure a container to execute a periodic mini-process that checks for certain conditions. These processes are defined as follows:

Readiness probe

Even after an application has started up, it may still need to execute configuration procedures—for example, connecting to a database and preparing data. This probe checks if the application is ready to serve incoming requests. Figure 14-1 shows the readiness probe.

Figure 14-1. A readiness probe checks if the application is ready to accept traffic

Liveness probe

Once the application is running, you want to make sure that it still works as expected without issues. This probe periodically checks for the application’s responsiveness. Kubernetes restarts the container automatically if the probe considers the application be in an unhealthy state, as shown in Figure 14-2.

Figure 14-2. A liveness probe checks if the application is healthy

Startup probe

Legacy applications in particular can take a long time to start up—possibly several minutes. A startup probe can be instantiated to wait for a predefined amount of time before a liveness probe is allowed to start probing. By setting up a startup probe, you can prevent the application process from being overwhelmed with probing requests. Startup probes kill the container if the application can’t start within the set time frame. Figure 14-3 illustrates the behavior of a startup probe.

Figure 14-3. A startup probe holds off on starting the liveness probe

From an operational perspective, the most important probe to implement is the readiness probe. Without defining liveness and startup probes, the Kubernetes control plane components will handle the majority of the default behavior.

Each probe offers distinct methods to verify the health of a container, discussed in the next section.

Health Verification Methods

You can define one or many of the health verification methods for a container. Table 14-1 describes the available health verification methods, their corresponding YAML attribute, and their runtime behavior.

Remember that you can combine any probe with any health check method. The health verification method you choose highly depends on the type of application you are running in the container. For example, the obvious choice for a web-based application is to use the HTTP GET request verification method.

Health Check Attributes

Every probe offers a set of attributes that can further configure the runtime behavior, as shown in Table 14-2. For more information, see the API of the Probe v1 core object.

The following sections will demonstrate the usage of most verification methods for different probe types.

Retrieving High-Level Information

To retrieve the information, run either the kubectl get pods command for just the Pods running in the namespace or the kubectl get all command to retrieve the most prominent object types in the namespace (which includes Deployments). You will want to look at the columns READY, STATUS, and RESTARTS. In the optimal case, the number of ready containers matches the number of containers you defined in the spec. For a single-container Pod, the READY column would say 1/1.

The status should say Running to indicate that the Pod entered the proper life cycle state. Be aware that it’s totally possible that a Pod renders a Running state, but the application isn’t actually working properly. If the number of restarts is greater than 0, then you might want to check the logic of the liveness probe (if defined) and identify the reason a restart was necessary.

The following Pod observes the status ErrImagePull and makes 0/1 containers available to incoming traffic. In short, this Pod has a problem:

$ kubectl get pods
NAME              READY   STATUS         RESTARTS   AGE
misbehaving-pod   0/1     ErrImagePull   0          2s

After working with Kubernetes for a while, you’ll automatically recognize common error conditions. Table 15-1 lists some of those error statuses and explains how to fix them.