Leveraging Kustomize for Kubernetes Manifests

Brent Laster

Leveraging Kustomize for Kubernetes Manifests

by Brent Laster

Copyright © 2022 O’Reilly Media Inc.. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

Acquisitions Editor: John Devins

Development Editor: Angela Rufino

Production Editor: Katherine Tozer

Copyeditor: Piper Editorial Consulting, LLC

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Kate Dullea

January 2021: First Edition

Revision History for the First Edition

2022-01-10: First Release

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Leveraging Kustomize for Kubernetes Manifests, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-11706-1

[LSI]

Chapter 1. Kubernetes Application Deployment and Reuse

Across the enterprise and cloud environments, Kubernetes has achieved unparalleled dominance as the platform for running containerized workloads. It has done this, in part, by automating and simplifying the key functions needed to ensure reliable hosting, scheduling, and management of workloads. And it maintains this throughout an application’s life cycle.

Yet, ironically, until the introduction of some of the tools we will discuss here, the process for working with the specifications for Kubernetes workloads (also known as manifest) was not automated or simplified. For early Kubernetes users (and some today), managing these requirements was a high-cost activity up front, and a potential high cost any time that reuse was needed.

An approach to help with the situation was introduced with a tool called Helm. Helm provided a way to make Kubernetes manifests into templates and insert values for the template at deployment time. It also made simple changes at execution time possible, such as changing case and inserting text based on some condition. And it enabled managing sets of manifests as a group when deployed to Kubernetes.

Helm represented a significant improvement over basic tooling like the kubectl command-line interface to manage manifests. But the overall effort and knowledge to convert manifests to the Helm format, along with the corresponding effort to maintain them, could be complex.

Manifests that were changed to work with Helm were unusable without it. They could no longer be applied with just kubectl, but had to be processed through Helm to produce specifications that were ready for Kubernetes. In previous versions, Helm experienced significant security concerns due to how part of it (a piece called Tiller) was deployed into Kubernetes in order to drive it. (Those security concerns were addressed with the release of version three, which removed the need for Tiller.)

All of this prompted users to search for a better way to enable reuse of Kubernetes manifests. Enter Kustomize. The developers of Kustomize took a fundamentally different approach from Helm. Rather than modifying the original files, Kustomize provided a way to create custom versions of manifests via layering on additional changes and producing a separate version with the changes. Users could save or apply this separate, changed version to Kubernetes directly. This ability to layer changes on top of existing manifests meant that users could maintain the original versions as they were while still enabling reuse and custom changes.

In this report, we’ll look more broadly at some of these challenges to application deployment and reuse that Kubernetes users typically encounter. We’ll also see how Kustomize helps simplify and alleviate them. By the end of the report, you’ll have an overall understanding of what Kustomize is, how it functions, how it compares to similar tools like Helm, and how it fits in to an overall strategy for managing Kubernetes manifests.

The Kubernetes Value Proposition

For any software platform to be widely accepted, the value proposition must include ease of defining and executing workloads. In Kubernetes, executing workloads is largely automatic once valid specifications that describe the workload are applied. This model is known as a desired state approach. The type of objects an application needs in order to run and their characteristics are clearly defined. Then the platform takes care of ensuring the objects are created and maintained according to those definitions for as long as needed.

In reality it is not always this simple. But the basic premise of declaring the object definitions, and relying on the platform to ensure they are instantiated and maintained as corresponding objects over time, is extremely useful. For example, we may declare that we want a Kubernetes service of a particular type with specific ports available in a manifest. Then when we provide the manifest as input to Kubernetes, Kubernetes is responsible for ensuring a service object is created that matches our specifications.

Much like a data center ensures that infrastructure and systems are available for running applications, Kubernetes helps ensure that resources are available to serve the needs of the containerized workloads running in it. If a situation comes up where what’s actually running in the platform (the observed state) doesn’t match what was specified as needed (the desired state), Kubernetes, like a human operator in a data center, can take actions to ensure the desired state and the observed state are reconciled—meaning we get back to what was asked for.

This leads to a key benefit of using the Kubernetes approach: application developers, and those responsible for deploying the applications, can worry less about how the applications are instantiated and run over time. This allows them to focus more on delivering functionality and customer value.

For this approach to work, the specifications or manifests still need to get defined and running in Kubernetes up front. The specifications are usually collections of manifests for different types of objects. These collections include everything Kubernetes needs to execute, track, and manage to run the application. The needs can be significant. A simple microservice typically requires one specification for a deployment, one for a service, and more for various supporting pieces. Multiply this by the number of microservices that support a typical customer-facing application, and the number of specifications quickly rises.

Another Alternative

For completeness, another alternative is to leverage a higher-level abstraction, like a Knative Service that combines all of these application parts into a single manifest type. Visit the Knative documentation for more information.

In short, a significant investment can be required up front for application development teams and Kubernetes administrators to get to the set of manifests that will successfully deploy with the correct configuration into Kubernetes. If reuse is also needed, additional work has to be done to allow for targeted changes to the original. If not done carefully, such changes can introduce significant overhead. These areas are where Kustomize provides value.

With that context, let’s look more broadly at the challenges that Kustomize is designed to help solve.

A Brief Survey of Application Deployment and Reuse Options

We have mentioned that Kubernetes is an example of a desired-state system, also known as a declarative approach. Stating this in practical terms, a user or process provides a set of text files (typically in a format called YAML) that specify how the various objects should be structured and maintained while they exist in the cluster.

About YAML

YAML (usually defined as “Yet Another Markup Language”) is a structured syntax for specifying input into Helm and Kustomize. By structured, we mean it is divided into different sections and subsections; indenting topics under other topics indicates ownership.

Kubernetes, in turn, uses these text files as a blueprint to instantiate instances of the deployments, services, and other pieces needed to run the user’s apps. The declarative YAML files for Kubernetes (the manifests) with the definitions for the deployments, services, configmaps, etc. are usually required for even a simple microservice, so we need a collection of such files to get an app running in Kubernetes.

Creating collections of manifests that work for a specific application in its intended environment can require a nontrivial investment of time and effort. Once working, these files often become key business assets because they are critical to defining the business functions to be carried out in Kubernetes. So these files need to be managed efficiently and easily for application deployment and reuse.

Application Deployment

In simple terms, application deployment refers to providing the manifests to Kubernetes. You might think of an order at a restaurant as a kind of manifest: you place an order with your server, who is then responsible for ensuring you get what you asked for.

There are multiple ways to initiate the application deployment, but unless we call the Kubernetes APIs directly, all will involve providing text specifications to Kubernetes. The most basic approach is simply using the Kubernetes command-line utility, kubectl, and issuing its apply command against the YAML files.

At the other extreme, as we discussed earlier, Helm allows users to turn the manifests into templates with placeholders in them. At deployment time, Helm can fill in the placeholders automatically with values passed through the command line or from a data file. Simple text manipulations can also be done with Helm. So, after multiple layers of processing, we end up with a file that can be applied directly or managed via Helm as a release (a versioned grouping of manifests) in Kubernetes.

Kustomize falls between the basic option of kubectl and the advanced options of Helm. It provides a way to produce variations through a set of explicit and separate transformations to produce different versions. This offers flexibility with less complexity than Helm. And it does not need to modify the original files. Being able to keep the original manifests intact, while still rendering different variations for Kubernetes, provides a solid foundation for reuse. And it is much less of a pain point for those who have to deal with the manifests.

Kustomize Integration with kubectl

Since we’re mentioning both, it’s worth noting that the kubectl command-line interface for Kubernetes now includes an apply -k option to run a built-in version of Kustomize. Note, however, that the built-in version is not always the most recent one. More on that later.

Typical Approaches for Reusing Manifests

You may be wondering if you really need to worry that much about reuse. You do. A key value-add for manifests in Kubernetes is being able to repurpose them for different instances of similar workloads; for example, being able to change the storage location for a database easily, or the port number associated with a service. Reusability can also help with creating alternatives for different environments; for example, deployments on different clouds or different variants in a dev, test, or prod scenario.

At a high level, there are a couple of different approaches currently in use to accomplish reusability.

The copy approach

One approach to reuse that is less than ideal is to create one-offs by making a copy of the manifests, modifying the copies as needed, and storing them in a different location or with a different version or name. This leads to a dual maintenance situation if the two versions need to be kept in sync, which works for the immediate use case, but differences can multiply quickly, making it difficult to keep the necessary portions in sync.

As an example of the potential problems, suppose that you have made a copy of a manifest and later you want to change the original to add a new resource or update a value. A person or an automated process will need to examine the original, figure out the differences, and merge them into the copy.

The problem becomes even worse if other people make their own copies and change them to suit their particular use cases. Very quickly, the content diverges. As a result, users may miss important or significant updates to the original manifests. Or they might end up using confusing or conflicting variations of similar files.

Over time, the situation can worsen, and people can spend a significant amount of time just trying to keep things up to date. If copies of the copies are made, you can end up losing track of what was in the original and which one is most correct. This can dramatically affect usability and maintainability.

The parameterization approach

Another approach is to create parameterized templates from the files. The manifests are made into templates by replacing static, hardcoded values with placeholders that can be filled in with any appropriate data. Values are usually supplied at deployment time, with the data passed in from a command line or read in from a file. The resulting templates with the values filled in are then rendered as valid manifests for Kubernetes and applied to the cluster.

This is Helm’s approach, and it has challenges. Removing values and using placeholders fundamentally changes manifests and increases their complexity. And, as we have said, the specifications are no longer usable on their own; they require an application or process like Helm to find or derive the needed values and render something Kubernetes can use. Additionally, as templates, anyone who just wants to look at the original files can no longer easily parse them.

The templates are also susceptible to the same issues as the copy approach. In fact, the problems can be compounded when copies of templates are modified to have more placeholders with assortments of data values for the various copies stored elsewhere. Users may also add functions and logic to modify the flow and options used in the templates. At some level, this can turn the templates into a sort of program coded with YAML. At the extreme, it may make the templates unusable and unreadable. In such cases, the only way to make sense of them is to use Helm to render them, meaning plug in the data values and produce a completed version that people (and Kubernetes) can understand and use.

Introducing Kustomize

Ideally, you would be able to keep using your existing files in their original format and make variations without making permanent changes or copies. And the differences between versions would be kept small and simple.

These are the basic tenets of Kustomize’s approach. It’s an Apache License 2.0 tool that generates custom versions of manifests by overlaying declarative specifications on top of existing ones. Declarative refers to the standard way to describe resources in Kubernetes: declaring what you want a resource to be and how it should look and behave, in contrast to imperative, which defines the steps to create it.

Overlaying describes the process where separate files are layered over (or stacked on top of) each other to create altered versions. Kustomize leverages specific kinds of overlays of the original manifest. The changes to be made in the rendered versions are declared in a separate, dedicated file named kustomization.yaml, while leaving the original files intact.

Kustomize requires this special file, kustomization.yaml, to drive its behavior. One section of the kustomization.yaml file is dedicated to listing the resources. This section lists the names (and sometimes the paths) of the original manifests to base changes on. When you run a kustomize build command, after loading the resources, Kustomize applies the overlays and renders the result.

You can think of this as applying the specified customizations on top of a temporary copy of the original manifest. The build action produces a customized copy of the manifest that can be fed directly into Kubernetes via a kubectl apply command. (Or if you are running a version of kubectl with support for Kustomize, you can use the kubectl apply -k syntax to simplify the overall process.)

The types of functions built into Kustomize transform your Kubernetes manifests, given a simple set of declarative rules. So, these sets of rules are called transformers.

The simplest kind of transformer applies a common identifier to the same set of resources, as Figure 1-1 demonstrates.

This example has a simple directory with a set of YAML files for a web app (named roar) with a MySQL backend. The files excerpted in the diagram are:

roar-web-deploy.yaml

A Kubernetes Deployment manifest for the web app

roar-web-svc.yaml

A Kubernetes Service manifest for the web app

kustomization.yaml

The Kustomize input file that declares the type of transformations you want to make to the manifests

Figure 1-1. Example structure and content for basic Kustomize use

In the kustomization.yaml file in Figure 1-1, the commonLabels section (the bottom left) is an example of a transformer. As the name implies, this transformer’s intent is to make the designated label common in the files after the transformation.

The kustomization.yaml file also includes a resources section, which lists the files to be included and possibly customized or transformed (highlighted in Figure 1-2).

Figure 1-2. Kustomize’s resources section denotes which original manifests to include

The kustomization.yaml file is a simple set of declarations about the manifests you want to change and how you want to change them; it is a specification of resources plus customizations. The modifications happen when you run the kustomize build command. The build operation reads the kustomization.yaml file, pulls in the resources, and applies the transformers appropriate to each file. This example pulls in the two roar-web YAML files, produces copies of them, and adds the requested label in the metadata section for each one.

By default, the files are not saved anywhere, and the original files are not overwritten. The “transformed” content created by Kustomize can be piped directly to a kubectl apply command, or redirected and saved to another file. However, it’s generally not a good idea to save generated files because it becomes too easy for them to get out of sync with the source. This is especially true if you are storing content in source control. You can view the output from the kustomize build step as generated content.

Instead of saving the generated files, you can save the associated original files and the kustomization.yaml file. Since the kustomization.yaml file pulls in the original files and transforms them for rendering, they can stay the same and be reused in their original form. But if the original files need to be updated, then the Kustomize build process can just use the updated version to generate new customized content.

How Kustomize Differs from Helm

Both Kustomize and Helm aid Kubernetes users with reusing and customizing manifests. As discussed, Helm takes the approach of allowing users to insert placeholders in place of hard-coded values in manifests. This transforms the manifests into templates that have the advantage of being configurable but the disadvantage of not being usable in Kubernetes in the template form. So the input templates require Helm to determine the right values to plug in templates and render versions of the templates as manifests that Kubernetes can then use.

Here is an example of a Helm template for a simple Kubernetes service:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "roar-web.name" . }}
  labels:
    app: {{ template "roar-web.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  namespace: {{ .Values.namespace }}
spec:
  ports:
    - name: {{ .Values.service.frontendPortName }}
      port: {{ .Values.service.internalPort }}
      targetPort: {{ .Values.service.externalPort }}
  type: {{ .Values.service.portType }}
  selector:
      app: {{ template "roar-web.name" . }}

Now let’s look at an example of a corresponding “values file” that supplies default data values:

# Default values for roar-web chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
  repository: bclaster/roar-web-image
  tag: v1
  pullPolicy: Always
deployment:
  ports:
    name: web
    containerPort: 8080
service:
  portType: NodePort
  frontendPortName: frontend
  externalPort: 8080
  internalPort: 8089

The file is rendered (output with values instantiated) via a helm install or similar operation, resulting in the following fully usable manifest:

# Source: roar-web/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: roar-web
  labels:
    app: roar-web
    chart: roar-web-0.1.0
    release: RELEASE-NAME
    heritage: Helm
  namespace:
spec:
  ports:
    - name: frontend
      port: 8089
      targetPort: 8080
  type: NodePort
  selector:
      app: roar-web

As we’ve previously noted, the original template file was not usable as a manifest due to the placeholders in it. It was only when Helm ran against it and pulled in the intended values that something suitable was created for Kubernetes. In contrast, the Kustomize scenario always ensures that the original, usable manifest is kept intact.

Kustomize drives these modifications based off of directives and rules defined in kustomization.yaml. This means that the data and actions targeted for customization are always managed in a separate file. In the Helm model, the places and actions are managed in the templated instance.

Here is an example of a simple kustomization.yaml file:

# Common label to be applied
commonLabels:
  level: test

# List of resource files and/or directories that kustomize reads,
# modifies, and emits as a YAML string
resources:
- roar-web-svc.yaml
- roar-web-deploy.yaml

The starting version of a file that the preceding code will affect (a service manifest file included via the resources section) is shown in the following:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: roar-web
  name:roar-web
spec:
  ports:
  - name: frontend
    port:8089
    targetPort: 8080
  selector:
    app: roar-web
  type: NodePort

Running a kustomize build command against a directory where these files are stored would produce a separate output file like the one shown here:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: roar-web
    level: test
  name:roar-web
spec:
  ports:
  - name: frontend
    port:8089
    targetPort: 8080
  selector:
    app: roar-web
    level: test
  type: NodePort

Note the addition of the level: test lines in the preceding code that occurred as the result of the Kustomize build. These result from the commonLabels transformer applied to each file listed in the resources section.

There are various other types of transformers that Kustomize provides. We’ll discuss a few more of them in the next chapter, and Appendix A has a full list. They are all executed the same way: the specifics are declared in a kustomization.yaml file with a set of resources to tell it which files to act on, then a kustomize build command produces the updated copy from the original.

There is one other significant difference between Helm and Kustomize. Helm includes functionality to treat all of the files necessary for an application as a unit called a chart. A chart consists of the values file for the default values to be put into the templates, the templates themselves in a separate directory, and a set of supporting files. Figure 1-3 shows an example of the basic structure of a Helm chart.

Figure 1-3. Example structure of a Helm chart

This grouping makes it simple to treat the larger collection of files Helm needs to run the app as a single item in Kubernetes. This simplifies packaging and retrieving the files the application needs as well as deploying and managing them in Kubernetes. In fact, when a Helm chart is deployed into Kubernetes, it creates what is referred to as a Helm release in the namespace. This release can then be upgraded, deleted, rolled back, and more. Figure 1-4 shows a representation of a Helm chart being deployed as a release into Kubernetes.

Figure 1-4. Installing a chart via Helm into a release in Kubernetes

Kustomize does not include such functionality. But in general, the kustomization.yaml file is effective for pulling in the necessary resources that can be transformed via a single kustomize build command. There can also be multiple kustomization.yaml files in a directory structure to handle different parts of an app, apps that need to be deployed together, or varying environments for an app.

The Advantages of Using Kustomize

The templated approach of Helm has some significant pitfalls. In any given template, you will have:

The static text (parts that don’t change)

The templated text to be replaced

The control logic (in the form of placeholders with value references and text/logic transformations)

When the manifest starts getting too cluttered with templates and logic (and the formatting for the templates and logic is complex to get right), it can become challenging to read and maintain. In addition, updating templates in a common way, such as adding the same label to multiple objects, can become problematic and tedious.

You can see this in the comparison between a standard manifest and a Helm manifest in Figure 1-5.

Figure 1-5. Comparison of standard Kubernetes manifest and Helm version

One other pitfall of Helm charts and templates is getting the initial templates and supporting files created. While Helm does provide a way to scaffold charts, it can still require significant effort to map the values to the templates and avoid incorrect syntax when creating them.

Kustomize provides a better structure, where the control logic is kept in the separate kustomization.yaml file. Through mechanisms such as the common label annotation, Kustomize simplifies and standardizes updates to multiple resources at the same time, which can be challenging to do with other rendering tools.

The previously mentioned integration into Kubernetes is an additional advantage of Kustomize. There are two integration points:

To view the resources in a directory with a kustomization file, run:

$ kubectl kustomize < directory >

To apply those resources, use the -k option on kubectl apply:

$ kubectl apply -k < directory >

If you are using an older version of Kubernetes, it might not have an updated version of Kustomize. In most cases, this isn’t a problem unless you need a particular feature or bug fix available in a current version of Kustomize.

Determining the Kustomize Version Included in kubectl

kubectl does not currently have a way for you to determine the version of Kustomize that it has embedded via the command line. However, by convention, the version is noted on the Kustomize home page. See “kubectl integration” in the GitHub repository.

Looking Beyond the Basics

Now that we’ve provided a basic introduction to Kustomize, in the next chapter we’ll dive a little deeper into the unique value-adds that it can provide for simplifying reuse. Understanding these at a high level will help you and/or your organization make decisions about whether Kustomize is a good fit for the particular Kubernetes use cases you need to deal with.

Chapter 2. Key Strategic Functionality Provided by Kustomize

In Chapter 1, we introduced Kustomize and discussed how it allows us to reuse Kubernetes manifests without incurring changes to the original files. In this chapter we’ll cover more of the whys and hows of leveraging Kustomize as a strategic advantage for reusing Kubernetes manifests. This will help you evaluate Kustomize as a potential fit for your use cases.

Specifically, we’ll look at how and why Kustomize provides the following functionality:

Adding common identifiers to a set of resources

Easily creating and managing variations of configurations

Strategically patching (updating) key areas of manifests

Leveraging hierarchical structures with Kustomize

Making customizations more configurable through components

We’ll start with how to easily add common transformations against a set of resources, a topic we discussed only briefly in the first chapter.

Common Transformations

A frequent use case when working with reuse or wider use of Kubernetes objects is applying a common identifier across multiple objects. Kubernetes relies on identifiers such as labels and annotations for this. It can use labels and annotations to easily identify and select resources. They can also be used to embed additional information about or into objects.

Kustomize provides a set of simple transformations that it can apply to a set of resources to update them with one or more of these types of identifiers. These transformers include:

commonLabel

Adds a common label (name: value) to each Kubernetes resource

commonAnnotations

Adds an annotation to all Kubernetes resources

namePrefix/Suffix

Adds a common prefix/suffix to all resource names

namespace

Adds a common namespace to all resource names

Figure 2-1 shows an example of defining some common transformers in a kustomization.yaml file and running the build command to produce the updated output. Notice that in both cases, the rendered files have the common identifier added in each of the resources.

Beyond the basic transformers for common identifiers, there are more specific ones to update the content that will be included, such as the image transformer. This again is intended to simplify something that users commonly need.

Figure 2-1. An example of using common transformers

Kustomize Default Transformers

Appendix A lists all of the default transformers available in Kustomize along with a brief description and references to examples for each.

As the name implies, image transformers produce a version of a manifest with a different newName or newTag for an image spec. Any image can be modified, such as those for a container or an init container. The name value must match the image value in the original resource so that Kustomize can be sure of what to change.

Example 2-1 shows a kustomization.yaml file with changes for an image.

Example 2-1. Using an image transformer

  resources:
  - ../../base

  namespace: roar-staging

  # Change the image name and version
  images:
  - name: quay.io/techupskills/roar-db:v2
    newName: quay.io/bclaster/roar-db-test
    newTag: v4

While it’s useful to do these kinds of transformations, a more strategic feature is creating separate versions for different environments from a set of resources. In Kustomize, these are called variants.

Variants

In Kubernetes, it’s common to need multiple variations (variants) of a set of resources and the manifests that declare them. A simple example is building on top of a set of Kubernetes resources to create different variants for different stages of product development, such as dev, test, or prod.

To facilitate these sorts of changes, Kustomize uses the concepts of overlays and bases. A base declares things variants have in common, and an overlay declares their differences. Both bases and overlays are represented within a kustomization.yaml file.

Example 2-2 shows this structure. It has the original resource manifests and a base kustomization.yaml file in the root of the tree. The kustomization.yaml defines the variants as a set of overlays in the respective subdirectories for the stage and production stages.

Example 2-2. A structure for Kustomize with bases and overlays to implement variants

roar-kz
├── base
│   ├── db
│   │   ├── kustomization.yaml
│   │   ├── roar-db-deploy.yaml
│   │   └── roar-db-svc.yaml
│   ├── Kube-descriptor.yaml
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   └── web
│       ├── kustomization.yaml
│       ├── roar-web-deploy.yaml
│       └── roar-web-svc.yaml
└── overlays
    ├── prod
    │   ├── db
    │   │   ├── kustomization.yaml
    │   │   └── patch_pv.yaml
    │   ├── kustomization.yaml
    │   ├── namespace.yaml
    │   └── storage.yaml
    └── stage
        ├── db
        │   └── kustomization.yaml
        ├── kustomization.yaml
        ├── namespace.yaml
        └── web
            ├── kustomization.yaml
            └── patch_db.yaml

You can think of the bases in this case as the default (or original) configuration and the overlays as how we create alternate versions of the original. With this kind of structure, it’s simple to create the alternative versions by running a kustomize build command. You would run this on the directory containing the kustomization.yaml file(s), which includes the specifications of how to make the changes.

Kustomization Hierarchies

Example 2-2 illustrates another useful concept when working with Kustomize: multiple kustomization.yaml files in a project hierarchy. This example project has two subprojects: one for a database and another for a web app.

The database piece has a customization to update the service name. The web piece simply has a file to include the resources.

At the base level, there is a kustomization.yaml file that pulls in resources from both parts of the project and a simple file to create a namespace. It also applies a common label to the different elements.

Variants can also apply patches, which represent a more surgical approach to targeting one or more specific sections in a resource.

Patches

Patches in Kustomize are a partial specification or a delta (a set of targeted items to change) for a Kubernetes object. They describe what a section should look like after it changes and how it should be modified when Kustomize renders an updated version.

Despite the simple idea, patching can be a complex process to understand. For the sake of simplicity, we’ll use a series of pictures to explain the process Kustomize executes when a patch occurs.

To start, you have an app that is made up of two pieces—a web app and a database piece. Each has their set of core resource files (a deployment and a service) and the associated kustomization.yaml files for them. Figures 2-2 and 2-3 show the structure and one of the service manifests for the database part. The patch in this example renames the database service.

Figure 2-2. Patching content in the database portion of the project

Figure 2-3. The service definition for the database resource

The patch definition in the kustomization.yaml file needs to be able to easily find the specific information to change. Figures 2-4 through 2-7 highlight the patch portion intended for the service:

Line 12 defines the type of patch, a replace in this example.

Lines 13 and 14 identify a location in the YAML hierarchy to find the value to patch and the replacement value to use.

Lines 15–17 identify the specific type of item in the Kubernetes resources you wish to change.

Figure 2-4. Example patch block in kustomization.yaml

Figure 2-5. More detail on the type of patch

Figure 2-6. More detail on the location in the hierarchy in the base files and replacement value

Figure 2-7. More detail on the exact item to search for and replace (per the “op” setting)

When you execute the kustomize build command against this set of files, Kustomize first locates the Kubernetes resource you’re interested in⁠—the Service—and then finds the path identified in the patch block (metadata.name). Then it renders a version of the spec with the value roar-db replaced with mysql. Figures 2-8 through 2-13 illustrate this process.

Figure 2-8. Navigating the YAML structure in the original file

Figure 2-9. Finding the target name

Figure 2-10. Finding the target section

Figure 2-11. Finding the key

Figure 2-12. Identifying the path

Figure 2-13. After the substitution

This is an example of the strategic merge patch option within Kustomize. Kustomize also can also support patches declared via JSON as an alternative.

In addition to patching, Kustomize can make strategic updates by convention in other ways (instead of being specifically told to) to make it easier to keep things in sync, even after Kustomize is out of the picture. We’ll look at one way Kustomize can do this in the next section.

Generators

Kustomize includes an additional type of updating functionality known as generators. Generators specified in Kustomize associate running Kubernetes resources with each other so they are automatically updated when a related running resource is updated. A generator establishes a connection between two resources by generating a random identifier and using it as a common suffix on the names of the objects.

This can be especially beneficial for Kubernetes objects like ConfigMaps and Secrets. Deployments and other objects frequently depend on data stored in instances of those types. If the data is changed in the ConfigMap or Secret, the corresponding Deployment needs to be regenerated and updated to pull in the latest data. This is the functionality that using generators provides.

Figure 2-14 shows an example specification for a Kustomize generator. In this case we are creating a generator to associate a unique ID with a ConfigMap, thus the Kustomize type of configMapGenerator.

Figure 2-14. Example of a Kustomize generator

When they are run through a Kustomize build operation, the new objects produced will have the generated name applied and included in the specs, as shown in Example 2-3.

Example 2-3. Objects and specs resulting from using a Kustomize generator

apiVersion: v1
data:
  mysql.database: registry_test
kind: ConfigMap
metadata:
  labels:
    variant: staging
  name: mysql-db-configmap-289g6c5f4k

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: roar-web
    variant: staging
    ...
    spec:
      containers:
      - env:
        - name: MYSQL_ENV_MYSQL_DATABASE
          valueFrom:
            configMapKeyRef:
              key: mysql.database
              name: mysql-db-configmap-289g6c5f4k

Assume we deploy the objects into Kubernetes and they are running in the cluster. If we then change the data in the ConfigMap associated with the generator (from _test to _prod, as Figure 2-15 shows), Kustomize will generate new values that are incorporated into the specs and objects, as shown in Example 2-4. Then, if we take the build output and apply it, the deployment will also be updated because the ConfigMap was changed, as demonstrated in Example 2-5. This works because the unique name of the ConfigMap was inserted into the deployment spec.

Figure 2-15. Making a change to the ConfigMapGenerator

Example 2-4. Changes after the ConfigMapGenerator is updated and a Kustomize build is run

apiVersion: v1
data:
  mysql.database: registry_test
kind: ConfigMap
metadata:
  labels:
    variant: staging
  name: mysql-db-configmap-d8tt4km77d

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: roar-web
    variant: staging
    ...
    spec:
      containers:
      - env:
        - name: MYSQL_ENV_MYSQL_DATABASE
          valueFrom:
            configMapKeyRef:
              key: mysql.database
            name: mysql-db-configmap-d8tt4km77d

Example 2-5. Changes to the deployment based on changes to the ConfigMap

$ kz build. | k apply -f -

namespace/roar-staging unchanged
configmap/mysql-db-configmap-d8tt4km77d
unchanged
service/mysql unchanged
service/roar-web unchanged
deployment.extensions/roar-db unchanged
deployment.extensions/roar-web configured

Removing Unused Generated Objects

When creating generated ConfigMaps or Secrets, you can use a common label with them to simplify later cleanup. If you’ve done that, run a command like the following:

  kustomize build . | kubectl apply -f - --prune -l app=LABEL

There is currently an enhancement proposal to make this more straightforward.

In summary, a kubectl apply operation on the updated ConfigMap updates any dependent items that reference the hash value of the ConfigMap in the cluster.

Extending Kustomize

If a transformer or generator does not meet a specific need, you can create your own generator or transformer and extend Kustomize to use it via creating a Kustomize plug-in. Be careful of general searches or other references about writing Kustomize plug-ins because several formerly supported approaches have now been deprecated.

You can also see an example of adding a transformer for a custom resource on GitHub.

For our final item in this discussion of Kustomize functionality, we’ll briefly describe Kustomize components, a relatively new feature.

Components

Earlier in this chapter, we talked about variants and how Kustomize can be used to support and create variations of the original files for different configurations such as dev, test, or prod. From the start, the Kustomize model has been very useful for enabling this across a single dimension, meaning that each environment has only one option for the set of changes contained in that configuration (overlays): only one version of a dev instance, one version of a test instance, and one version of a prod instance.

But what happens if we need or want to introduce additional dimensions within a particular variant? For example, what if our dev variant needs to be used on two very different types of hardware, such as H1 and H2?

One option is to create new variants and have dev-H1, dev-H2, test, and prod. But suppose we have additional options, such as needing to run with Feature 1 and Feature 2 on and off in the two new variants. With the single-dimension model, and using F1 or F2 to mean Feature 1 or 2 is on, we could easily end up needing dev-H1, dev-H1-F1, dev-H1-F2, dev-H2, dev-H2-F1, dev-H2-F2, test, and prod. We would also need to create items for each combination to modify ConfigMaps, Secrets, etc. This could be done as an actual implementation, but you can see how the sheer number of variants we’d have to create and maintain quickly becomes unwieldy. If you only occasionally need certain combinations, there is a better way.

This is where Kustomize components come in. Components in Kustomize can be used to define a kind of reusable customization. You can include them from overlays at a higher level to enable subsets of functionality in variants.

Components offer a more DRY (don’t repeat yourself) approach and are a type of the attribute kind. In Kustomize, users can create an object of kind component and define only the pieces that need to be modified without specifying any resources to operate on. Each component definition can define the particular modifications to make to implement one or more options. For example, in a directory structure for the components, you might have:

├── components
│   ├── feature1
│   │   ├── configmap.yaml
│   │   ├── deployment.yaml
│   │   └── kustomization.yaml
│   ├── feature2
│   │   ├── configmap.yaml
│   │   ├── deployment.yaml
│   │   └── kustomization.yaml
│   ├── hardware1
│   │   ├── configmap.yaml
│   │   ├── deployment.yaml
│   │   └── kustomization.yaml
│   └── hardware2
│       ├── deployment.yaml
│       ├── kustomization.yaml
│       ├── secret_key.txt
│       └── site_key.txt

You could then create variants with needed options enabled (or not) simply by including the desired components in the kustomization.yaml file for the variant. This is a better alternative than having to create separate, hardcoded combinations of each possible set.

The kustomization.yaml file for a variant for developing with F1 on H1 might look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base

components:
  - ../../components/feature1
  - ../../components/hardware1

Overall, Kustomize components target providing additional flexibility to turn options or features on and off from within the kustomization.yaml file. This simplifies controlling which options are in effect for a variant and aides overall readability and understanding. It also lessens the amount of Kustomize code and instructions that you have to write once the components are defined.

A final benefit of using components is that in a large organization, they enable a more distributed approach to using and maintaining Kustomize. Separate, distributed teams can author and support their own components that can then be used via simple inclusion in other kustomization.yaml files.

Distinguishing Components from Resources

You may be wondering why we couldn’t just list the components as another element in the resources section. The enhancement proposal for components says it best:

A kustomization that is marked as a Component has basically the same capabilities as a normal kustomization. The main distinction is that they are evaluated after the resources of the parent kustomization (overlay or component) have been accumulated, and on top of them. This means that:

A component with transformers can transform the resources that an overlay has previously specified in the resources field. Components with patches do not have to include the target resource in their resources field.

Multiple components can extend and transform the same set of resources sequentially. This is in contrast to overlays, which cannot alter the same base resources, because they clone and extend them in parallel.

Figure 2-16 shows another example of how components might be defined and used by various overlays.

Figure 2-16. An example of component use in different overlays

Conclusion

Kubernetes is as important and common to containerized workloads as operating systems have been to traditional workloads. Being able to reuse the specifications that describe and define those workloads is essential to getting the most value out of applications in any clustered environment.

While there are multiple tools that aid in reuse, Kustomize stands alone in its approach. It ensures that the original files are still usable and the logic that produces changes or variants is centralized in a standard way and in a standard file. All of this contributes to a more digestible, manageable, and ultimately scalable and usable approach that other tools like Helm cannot currently match.

Appendix A. Kustomize Default Transformers

Kustomize’s transformers are responsible for modifying and collecting information about resources during the kustomize build or kubectl apply -k processes.

Kustomize provides the transformers shown in Table A-1 as defaults.

Table A-1. Default transformers available in Kustomize

Transformer
Purpose
Example

annotations
Adds annotations to the metadata/annotations field for all resources. Annotations are also added to spec/template/metadata/annotations for Deployment, ReplicaSet, DaemonSet, StatefulSet, Job, and CronJob resources, and spec/jobTemplate/spec/template/metadata/annotations for CronJob resources.
See Figure 2-1

images
Updates the image key value in paths that include con⁠tainers and init​containers subpaths. Constructs a new image entry via the newName, newTag, and digest fields. The name field should match the image key value in a resource.
See Figure 2-2

labels
Adds labels to the metadata/labels field for all resources. Per the API requirements, also adds labels to the spec/selector field in all Service resources and to the spec/selector/matchLabels field in all Deployment resources.

commonLabels:
  environment: staging

name​Ref⁠eren⁠ces
Contains a list of name references, which represent all of the possible fields where a type could be used as a reference in other types of resources. Used to note the other places that a particular type of object might be used in a manifest. Needed so that when the name of an object is changed—say, using the namePrefix transformer, then kustomize knows all the other locations in the manifest hierarchies that it needs to check for that type of object and make the same name change if needed. Note that this is not one that the user specifies. Instead, the mappings are built into Kustomize. You can see the full set in the GitHub repository.
If we change the name of a PersistentVolume resource, check to see if any PersistentVolumeClaims or ClusterRoles reference that same name in their respective locations and change there if needed.

- kind: PersistentVolume
  version: v1
  fieldSpecs:
  - path: spec/volumeName
    kind: PersistentVolumeClaim
  - path: rules/resourceNames
    kind: ClusterRole</p>

namespace
Adds a namespace value to all resources

namespace: staging

prefix/suffix
Adds a prefix or suffix to the metadata/name field for all resources
See Figure 2-1

variable reference
Allows you to look up values from the YAML hierarchy in Kubernetes manifests and bind them to environment variables that can then be used as part of a container image’s environment, args, or command
Get the value from the metadata or name field of the my_webapp service and set the $WEBAPP_SERVICE env variable to that value so it can be used in the container invocation

vars:
  - name: WEBAPP_SERVICE
    objref:
      kind: Service
      name: my_webapp
      apiVersion: v1
    fieldref:
      fieldpath: metadata.name

About the Author

Brent Laster is a global trainer, author, and speaker on open source technologies, as well as an R&D DevOps director at a top technology company. He is the founder and president of Tech Skills Transformations, LLC—a company dedicated to making technology understandable and usable. Throughout his career in software development and management, Brent has always made time to learn and develop both technical and leadership skills and share them with others. He believes that, regardless of the topic or technology, there’s no substitute for the excitement and sense of potential that come from providing others with the knowledge they need to accomplish their goals.