A Bit of History

The concept of autoscaling took a significant leap forward in 2009 when Amazon Web Services (AWS) introduced Auto Scaling (known now as Amazon EC2 Auto Scaling). This feature allows you to automatically adjust the compute capacity to meet your application demands at the lowest possible cost. This shift towards automation not only improved efficiency but also allowed organizations to be more responsive to fluctuating demands, rather than getting bogged down in system administration and operational burdens if done manually.After 2009, numerous organizations began implementing and refining autoscaling strategies. Netflix, for instance, documented their approach of aggressive scaling up and down to meet variable consumer demands. Other companies like Dropbox, Spotify, and Airbnb also shared their experiences and best practices for autoscaling in cloud environments. The way I like to see this is that this collective knowledge and experience from various industry leaders contributed to the evolution of autoscaling technologies and practices. As cloud-native architectures became more prevalent, the need for more sophisticated, application-aware autoscaling mechanisms grew.Kubernetes, with its origins in Google's experience running large-scale production workloads, embodies many of these autoscaling best practices. From its beginnings, Kubernetes was designed with the principles of efficient resource allocation and automatic scaling in mind.By building autoscaling into its core functionality, Kubernetes took the lessons learned from years of cloud scaling experiences and made them accessible to a wider range of applications and organizations. This native support for autoscaling has been a key factor in Kubernetes' widespread adoption, as it allows teams to focus on building and improving their applications rather than managing the underlying infrastructure scaling.

Horizontal and Vertical Scaling

Now that you understand the importance of scaling and its automated nature, let's explore the two primary approaches to scaling: vertical and horizontal. These methods offer different ways to increase a system's capacity to handle load, each with its own advantages and challenges.

Vertical Scaling

Vertical scaling, often referred to as "scaling up," involves boosting the capabilities of an existing machine or instance. This method typically focuses on enhancing a single node's processing power, memory capacity, or storage. Picture a scenario where your application server is struggling with its current workload. A vertical scaling solution might involve upgrading a node from 2 vCPUs with 4 GiB of RAM to node of 8 vCPUs with 16 GiB of RAM, as shown in Figure 1.2:

Figure 1.2 show that This approach is simple, particularly for applications not designed with distributed architecture in mind. It's often the go-to solution for improving the performance of monolithic applications or certain types of databases that prefer running on a single, powerful machine.However, vertical scaling isn't without its drawbacks:Costs: The cost of high-end hardware can escalate rapidly.Physical limitations: There is an upper bound to how much you can enhance a node.Potential interruptions: Hardware upgrades cause system downtime.Reliability: Having one (or few) node(s) creates a potential single point of failure.While vertical scaling can offer quick performance boosts, it may not provide the adaptability and robustness required for your workload, especially if it's very dynamic.

Horizontal Scaling

Horizontal scaling, often called "scaling out," takes a different approach to handling increased load. Instead of boosting a single node, horizontal scaling involves adding more nodes to your system. The following diagram illustrates an example of this:

Imagine that Figure 1.3 shows what happens in a restaurant: vertical scaling would be like hiring a super-chef, while horizontal scaling would be like adding more cooks to the kitchen. In practice, horizontal scaling might look like this: if your web application is struggling with high traffic, instead of upgrading a single server, you'd add more similar servers to distribute the load. This approach aligns well with modern, distributed architectures and cloud-native applications.Horizontal scaling offers several advantages that make it particularly useful for dynamic and distributed systems. For starters, the theoretically unlimited scalability it provides; as demand grows, you can continually add more machines to your system to meet these increasing needs. This approach also enhances fault tolerance significantly; if one node experiences problems or fails entirely, the other nodes in the system can compensate, ensuring continued operation and minimizing downtime. Furthermore, with horizontal scaling you can closely match your system's capacity to current demands, effectively avoiding the costly pitfall of over-provisioning resources. Horizontal scaling is particularly well-suited for stateless applications, microservices architectures, and distributed systems.However, horizontal scaling could also pose some challenges:Complexity: Applications need to be designed to work across multiple nodes.Consistency: Ensuring data remains consistent across all nodes can be tricky.Network overhead: Communication between nodes can introduce latency.License costs: Some software licenses may charge per node.In the context of Kubernetes, which we'll explore in depth later, horizontal scaling is a fundamental concept. Kubernetes ability to automatically scale the number of pods running an application is a prime example of horizontal scaling in action. However, you'll see when vertical scaling might make more sense or can even help you right-size your workloads.

What do I mean by efficiency?

To answer this question, let's first talk about what efficiency means in a broader sense. Efficiency, in general terms, is the ability to accomplish a task with minimal waste of time and effort. It's about achieving the desired outcome while using the least number of resources possible. In other words, no wasting of resources. In a formula, as shown in Figure 1.5, it will be like this:

Now, let's apply this concept to Kubernetes, specifically to the data plane nodes. In this context, efficiency means optimizing the utilization of compute resources to achieve the best performance at the lowest cost. It's not just about using all available resources; it's about using them wisely and effectively.As the focus of this book is on compute, this means that efficiency is about maximizing the use of available CPU and memory while minimizing waste. In traditional on-premises environments, efficiency might have been less of a concern as hardware costs were often treated as a sunk cost. However, in the cloud, where you typically pay for what you use (also known as pay-as-you-go), efficiency has a direct impact on your monthly bill. This makes resource optimization not just a best practice, but a financial imperative.So, as this book will focus in the cloud, an efficient Kubernetes data plane in the cloud context involves several key aspects:Right-sizing your resources: Ensuring your pods have enough resources to run effectively, but not so much that they're wasting capacity.Maximizing node utilization: Filling your nodes with an optimal number of pods to avoid running unnecessary nodes.Scaling appropriately: Increasing or decreasing resources in response to actual demand, rather than over-provisioning for peak loads.Avoiding idle resources: Identifying and eliminating unused or underutilized resources that you're still paying for.Choosing appropriate instance types: Selecting the most cost-effective node types for your workloads.By focusing on these aspects of efficiency, you can significantly reduce your cloud costs. For instance, an inefficient cluster might have nodes running at 30% capacity, meaning you're essentially paying for 70% of unused resources. An efficient cluster, on the other hand, might consistently run at 70-80% capacity, dramatically reducing waste and, consequently, costs. Of course, these numbers can vary depending on the type of workloads you have and the technical debt they might carry.Achieving this level of efficiency isn't straightforward. In the following chapters, I'll dive into the details of how you can address the key aspects I mentioned before. The goal is to help you build and maintain an efficient data plane, striking the right balance between performance, reliability, and cost-effectiveness. However, before doing that, it's important to understand the challenges and considerations to achieve efficiency in Kubernetes (actually, for compute in general).

Challenges and considerations

Several factors across the entire application stack influence performance and efficiency. The design of your application components, including layers, dependencies, and how they interact, have an impact on efficiency. The underlying infrastructure, from the physical hardware to the operating system, plays an important role in overall performance. How Kubernetes schedules pods and scales resources directly affects resource utilization. Moreover, network performance factors like latency, throughput, and the configuration of load balancers can impact the efficiency of your cluster.Achieving efficiency is challenging due to several factors. Workloads often have diverse resource requirements; some may be CPU-bound, others memory-bound, leading to uneven resource utilization across nodes. Kubernetes typically scales at the pod or node level, which can be too coarse for optimal efficiency, and make it complicated. And let's not forget that there's a delay between when additional resources are needed and when they become available. Workloads with dependencies often require resource buffers while waiting on these dependencies, leading to periods of underutilization, which might not be much but still are things you need to consider.Non-technical factors can also lead to inefficiency. Overprovisioning out of an abundance of caution is common, especially for critical workloads. This is a very typical scenario for peak seasons when it's better to keep business continuity than an ideal efficiency number.While striving for an efficient Kubernetes data plane is important for cost optimization, it's a complex challenge that requires a holistic approach. It involves not just infrastructure management, but also application design, careful capacity planning, and sometimes, a willingness to accept calculated risks. As you proceed through this book, you'll explore strategies and tools that can help you navigate these challenges and achieve a more efficient Kubernetes cluster.Now that you understand the complexities and challenges of achieving efficiency in Kubernetes environments, let's explore the autoscaling mechanisms that Kubernetes provides to address these challenges.

Application Workloads

The first step in your Kubernetes efficiency journey focuses on application workloads, specifically the pods that run your applications. This is a critical starting point because it directly impacts how efficiently your cluster utilizes resources and responds to changing demands. Central to this optimization is setting appropriate resource requests for your pods. This is crucial because the kube-scheduler uses these requests to make decisions about pod placement. If requests are set too high, you may waste resources and limit the number of pods that can be scheduled. Conversely, if set too low, your applications may underperform or face resource contention.The recommended approach for scaling application workloads is horizontal scaling - adding or removing pod replicas as demand fluctuates. This is typically achieved using the Horizontal Pod Autoscaler (HPA). While I'll dive deeper into HPA in the next chapter, it's worth noting that this tool automatically adjusts the number of pod replicas based on observed metrics like CPU or memory utilization, or custom metrics that you define. There are other tools in the same vein like Kubernetes Event-driven Autoscaling (KEDA) that in essence does a similar job to HPA, but the options to scale are broader. I'll dive into much more details about this in the next chapters.Let's consider an example: Imagine you have a web application that experiences varying levels of traffic throughout the day. During peak hours, the CPU utilization of your pods might spike to 80% or higher, similar to what you see in Figure 1.6:

The preceding figure illustrates that when pods from a deployment are using 80% of their CPU capacity, it's time to add more replicas to prevent the workload from underperforming. In this scenario, the HPA could automatically increase the number of pod replicas to distribute the load. As traffic subsides and CPU utilization drops, the HPA would then reduce the number of replicas, conserving resources.Let's consider an example: Imagine you have a web application that experiences varying levels of traffic throughout the day. During peak hours, the CPU utilization of your pods might spike to 80% or higher. As you can see in Figure 1.7, HPA automatically increased the number of pod replicas to distribute the load. As traffic subsides and CPU utilization drops, the HPA would then reduce the number of replicas, conserving resources.

In general, I advocate for horizontal autoscaling over vertical autoscaling because of:Better fault tolerance: If one pod fails, others can continue serving requests.Improved resource utilization: It's often easier to fit many small pods across your nodes than a few large ones.Easier rolling updates: With multiple replicas, you can update your application without downtime.

Data Plane Nodes

Scaling the data plane nodes becomes necessary when the cluster's capacity to run pods reaches its limit, especially when nodes are added only as needed. What I didn't say before is that when you scale out your workloads by adding more pods (following the recommendation for horizontal scaling), you may eventually reach a point where the cluster lacks the capacity to run these additional pods. This results in unscheduled pods, which serve as a trigger for the second category of Kubernetes autoscaling: scaling the data plane nodes. Tools like Cluster Autoscaler (CAS) and Karpenter are designed to address this challenge. These projects react to the presence of unscheduled pods in the cluster by adding more capacity. Both have built-in logic to interact with the underlying infrastructure and provision additional nodes as needed. I'll dive deep into these tools in the following chapters.Let's continue with our web application example. As the load increases and more pods are added to handle the traffic, you might reach a point where all existing nodes are at capacity. Let's say you have only 1 node capable of running 4 pods, and your HPA or KEDA scaling policy has determined that you need now 10 pods to handle the current load, see the following image:

As shown in Figure 1.8, only 2 additional pods could fit into the existing node, this leaves 6 pods unscheduled.At this point, Karpenter or CAS would spring into action. They would detect the unscheduled pods and initiate the process of adding a new node to the cluster. In a cloud environment, these tools already know which APIs to call to provision a new node and automatically register it to the cluster, eliminating the need for manual intervention. See the following image for an example of this process:

Figure 1.9, you can sese that once the new node is ready, the kube-scheduler will place the unscheduled pods on this newly available capacity. This process happens automatically, ensuring that your application can scale to meet demand without manual intervention or overprovisioning.Conversely, when demand decreases and fewer pods are needed (thanks to HPA or KEDA), these same tools ensure that any extra nodes added during the peak are removed, maintaining the cluster's efficiency, as shown below:

For instance, as you can see in Figure 1.10, if traffic drops and you only need 90 pods, CAS or Karpenter might remove two nodes from the cluster, returning to the original 10-node configurationIt's important to note that we're advocating for horizontal scaling at this level as well - adding or removing entire nodes rather than trying to resize existing ones. Unlike with pods, there are no established projects for vertical autoscaling of nodes, as this approach introduces unnecessary complexity and potential instability to the cluster.By leveraging these node autoscaling capabilities in conjunction with pod-level autoscaling, you can create a highly responsive and efficient Kubernetes cluster. This setup allows your cluster to dynamically adapt to changing workloads, ensuring optimal resource utilization and cost-effectiveness. Hold tight, I'll tell you how to implement all of this in the following chapters.Now that I've covered the theoretical foundations of Kubernetes autoscaling, it's time to setup your practice environment for this book.

Local Kubernetes cluster with Kind

Kind, short for Kubernetes in Docker, is a tool that enables you to create and manage local Kubernetes clusters by leveraging Docker containers as simulated nodes. It was primarily created for testing Kubernetes itself, but has become popular among developers for local development and continuous integration (CI) pipelines. Kind allows you to quickly spin up a multi-node Kubernetes cluster on your local machine, making it ideal for testing, learning, and developing Kubernetes-native applications without the need for a cloud provider or physical servers.

$ brew install kind

$ kind version

$ kind create cluster --name kubernetes-autoscaling

$ kubectl cluster-info --context kind-kubernetes-autoscaling

$ kubectl get nodes

NAME                                   STATUS   ROLES           AGE   VERSION
kubernetes-autoscaling-control-plane   Ready    control-plane   15m   v1.31.0

Cloud Kubernetes cluster in AWS

Let's create an Amazon EKS cluster using Terraform, an infrastructure as code tool that allows you to define your desired cluster state as code. You'll be using a custom Terraform template that leverages the Amazon EKS Blueprints for Terraform project. This template will set up a complete Amazon EKS cluster, including a VPC, a Kubernetes control plane, and the necessary IAM roles and service accounts. It will also configure a managed node group for essential system components.If you're not familiar with Terraform or HCL syntax, reviewing Terraform's documentation can be helpful (https://developer.hashicorp.com/terraform/docs).

Note

Terraform uses AWS credentials that must be configured in your environment. The recommended and more secure approach is to authenticate using: 1/ Named AWS CLI profiles (via aws configure --profile your-profile), 2/ IAM roles via instance metadata (for EC2 or CloudShell environments), or 3/ AWS SSO or IAM Identity Center. Avoid hardcoding AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY unless absolutely necessary, as this approach is less secure and harder to manage.

$ git clone https://github.com/PacktPublishing/Kubernetes-Autoscaling.git

$ cd Kubernetes-Autoscaling/chapter01/terraform

$ export AWS_REGION=eu-west-1

$ sh bootstrap.sh

$ aws eks --region $AWS_REGION update-kubeconfig \
--name kubernetes-autoscaling

$ kubectl get nodes

NAME                STATUS   ROLES    AGE     VERSION
ip-10-0-107-82...   Ready    <none>   4m18s   v1.30.2
ip-10-0-57-88...    Ready    <none>   4m18s   v1.30.2

$ sh cleanup.sh

Configuring requests

When you define resource requests for containers within a pod, you're basically telling the kube-scheduler how much resources the node needs to guarantee. This information helps the scheduler make informed decisions about where to place your pod in the cluster, ensuring it lands on a node with sufficient resources to meet its needs. The following image illustrates how the kube-scheduler places a pod on a node where it fits based on its resource requests:

In Figure 2.3, you can see that the kube-scheduler decided to schedule the pod on the node that has the best ranking score. The second node is the one that has the resources available for the new pod coming to the cluster, while the other two were already at their capacity.While this scheduling mechanism ensures that pods have access to their requested resources, it doesn't always lead to optimal resource utilization. In practice, pods may not use all the resources they've requested. This can result in low node efficiency, as there's often a discrepancy between requested and actually utilized resources. Consequently, this mismatch can lead to underutilization of cluster capacity, highlighting the importance of rightsizing pod resource requests.For an effective autoscaling configuration, you should always configure requests carefully. Moreover, this practice ensures predictable and deterministic results when the kube-scheduler assigns pods to nodes and is crucial for setting up effective scaling policies. Without proper resource specifications, you may encounter unexpected behavior, inefficient resource utilization, and potential performance issues as your application scales.

Configuring limits

Resource management in Kubernetes extends beyond scheduling. While resource requests are used for pod placement, you can also configure resource limits. These limits are enforced during runtime by the kubelet on each node. The kubelet ensures that containers don't consume resources beyond their specified limits, even if the node has additional capacity available. If a container attempts to exceed its limit, the kubelet will restrict its resource usage or terminate the process, depending on the resource type (compressible or non-compressible) and the severity of the violation. Look at the below image:

In Figure 2.4, you can see that the pod is requesting 1 GiB of memory as the minimum guaranteed allocation. However, it's setting a limit of 1.5 GiB to cap its maximum resource consumption. The bar at the side of the pod represents how much of the allocated resources the pod is currently using.

What if you don't specify resource requests or limits?

Pods are scheduled as a best-effort, but will be first ones to be evicted/throttled when the node experiences memory or CPU pressure. If no limits are configured, the pod can consume as much resources as it needs. As a safety mechanism, you could configure a LimitRange for a given namespace. A LimitRange is a Kubernetes policy object that defines resource constraints within a namespace. It will apply default values if you don't set resources in your pod, ensuring efficient resource allocation. Alternatively, you can configure a webhook to set default values before pods are scheduled; tools like Kyverno can simplify this task.

Pod configuration example

In Kubernetes, you configure the resource units like this:

• CPU: It is measured in units, and it means that 1 CPU unit is 1 physical or virtual core. These units can be fractional, like 0.5 (half o a CPU) or 0.2 (200 millicpu or 200m). You'd typically see the "m" format.
• Memory: It is measured in bytes. Can be configured as a number only, or accompanied with a suffix like M or Mi for megabytes. There are other suffixes, but let's keep it short and simple for now. You'd typically see the "Mi" suffix.

Here's an example configuration of a pod with resource and limits configured:

apiVersion: v1
kind: Pod
metadata:
  name: montecarlo-pi
spec:
  containers:
  - name: montecarlo-pi
    image: christianhxc/montecarlo-pi
    resources:
      requests:
        memory: "512Mi"
        cpu: "900m"
      limits:
        memory: "512Mi"
        cpu: "1200m"

Notice how limits are higher than resources. The kube-scheduler will guarantee that the node selected has at least 1200m units of CPU and 512Mi of memory available.

What if the pod exceeds the resource limits?

CPU is a compressible resource; when the CPU utilization limit is met, the container is throttled (no more CPU time for it) but it can continue running. With memory, it is different. If the memory utilization of a container is met, the process that is trying to use more memory is stopped. Typically, this causes Kubernetes to restart the container.

Recommendations for configuring resources and limits

Use your judgment after testing how your application behaves. You could run a set of tests, or monitor live traffic. But generally speaking, you could consider the following recommendations:For CPU, consider setting requests slightly higher: This approach allows for more flexible CPU utilization. When you specify CPU requests with slighly higher limits, you let your workloads to utilize available CPU capacity above what has been requested, adapting to varying demand (especially if it's spiky) without being constrained too much. It's particularly useful for applications with fluctuating CPU needs, as it allows them to use more CPU when necessary with flexible restrictions.For memory, consider setting requests equals to limits: This gives a predictable behavior for your containers, and the kube-scheduler provides the highest priority and stability for your workloads. If you still want to have different values, be cautious about setting limits significantly higher than requests. This can lead to node overcommitment, as the kube-scheduler may place more pods on a node than it can actually handle under peak conditions, potentially causing workload interruptions or out-of-memory events.By now, you have a better understanding how the kube-scheduler schedules pods, and why it's very important to configure resource request to each of the containers in a pod. Next, let's talk about ways of configuring these request units properly, as it's going to become the most important process to have an efficient Kubernetes data plane.

Monitoring

First, you need to set up the tooling to monitor how much resources your pods are actually using. This information is crucial for making proper decisions based on data, rather than speculation or previous configurations on virtual machines or previous environment setup. It's important to have monitoring in place as rightsizing is a continuous process, not something you do just once. Applications are constantly changing, either adding or removing functionality, which affects their resource needs.Note that the ecosystem of tools and platforms to help you monitor and rightsize your applications is growing. Some of these options are paid, but as your Kubernetes cluster keeps expanding, paid options might give you time to focus on solving customer problems rather than trying to reinvent the wheel. In this book, we'll rely on open-source tools like Prometheus and Grafana to help you gain hands-on experience without having to analyze which tool is best for your specific use case. However, keep in mind that in the long run, you might derive more value from using a third-party tool tailored to your needs.

Hands-On: Setting up Prometheus and Grafana

Head over to your terminal, and make sure you have your cluster up and running. If you need to bring it up again, go back to the Chapter 1 to do it, and come back later to this.To install Prometheus and Grafana using Helm, you first need to add the Helm Chart repository from Prometheus, update the Helm repos, and install the Prometheus stack. To do so, run these commands:

$ helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace

When the installation is done, you should be able to open Grafana. To do so, you first need to get the administrator user and password to login.To get the Grafana admin user, run this command:

$ kubectl get secret prometheus-grafana -n monitoring \
-o jsonpath="{.data.admin-user}" | base64 --decode ; echo

To get the Grafana admin password, run this command:

$ kubectl get secret prometheus-grafana -n monitoring \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Finally, to access Grafana, run this command:

$ kubectl port-forward service/prometheus-grafana 3000:80 -n monitoring

Open a web browser, and enter http://localhost:3000/. You should see the Grafana login page like below, enter the user and password and click on Log In:

For now, that's it, you'll comeback to Grafana in a moment.

Hands-On: Determining the right size of an application

Now that you have access to Grafana and Prometheus, let's deploy an application to the cluster, make some calls, and analyze the resource utilization. To get started, run the following command to deploy an application to run Pi simulations using the Monte Carlos method (it involves running multiple simulations with random sampling to obtain numerical results):

$ kubectl create deployment montecarlo-pi \
--image=christianhxc/montecarlo-pi:latest

This command creates a Kubernetes deployment using default values. You can explore the pod definition, but you won't see any resources section. Nonetheless, notice that the pod has been scheduled despite not having configured resource requests.Now, expose the deployment through a service by running this command:

$ kubectl expose deployment montecarlo-pi --port=80 --target-port=8080

Similar as before, this command creates a Kubernetes service using the default values.Let's generate some traffic to the application by making a call every 10 milliseconds:

$ kubectl run -i --tty load-generator --rm --image=busybox:1.28 \
--restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q \
-O- http://montecarlo-pi/monte-carlo-pi?iterations=10000000; done"

This command creates a pod that will terminate once you stop the command. This pod is using a BusyBox image to call the service endpoint using the wget command.As a result, you'll see the output you get after every call to the application. As long as you don't see any type of error, leave that running for about three to five minutes. Then, go back to Grafana, and open the Dashboards page located at the following URL: http://localhost:3000/dashboards. The Grafana stack you've deployed, comes with a default set of dashboards. Let's open the one named Kubernetes | Compute Resources | Workload, change the time range to see the data from the last five minutes.

Note

If the Kubernetes | Compute Resources | Workload dashboard is not available in Grafana by default, we have the JSON model at the GitHub repository under the file: /chapter02/grafana/kubernetes_resource_workload.json. To import it, you need to create a new dashboard, and import it using a JSON model.

You should see something similar to the following image:

For now, let's focus only on CPU usage. In Figure 2.6, you can see that the application is using 976m of CPU in average. Stop the command (Ctrl + C) you had run to generate traffic to the application. You'll see how the CPU usage starts to go down. Now that you have data to support how much CPU your application is using, let's configure proper CPU requests by adding an extra 20% by requesting 1200m of CPU. As this might be the maximum usage the pod could have, you're giving some threshold in case there's a traffic spike. Run the following command to adjust the resources requests:

$ kubectl patch deployment montecarlo-pi \
--patch '{"spec": {"template": {"spec": {"containers": [ \
{"name": "montecarlo-pi", \
"resources": {"requests": {"cpu": "1200m"}}}]}}}}'

Let's send some traffic again for another three to five minutes:

$ kubectl run -i --tty load-generator --rm --image=busybox:1.28 \
--restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q \
-O- http://montecarlo-pi/monte-carlo-pi?iterations=10000000; done"

After five minutes, stop the command (Ctrl + C), and go back to Grafana. You should now see a CPU Requests % column that tells you how much of the requested resources the pod is using, like the below image:

In this case, we could say that the pod has an 80.8% of efficiency, which is not a bad number. You could tune in the requests to achieve higher or lower percentage, but you need to make sure that by changing these values you don't affect the application's performance. We'll use this data to later configure effective autoscaling policies.

Establishing defaults

As you've seen by now, proper monitoring is crucial for setting up appropriate resource requests and limits configurations to pods. However, for some organizations, this process of rightsizing might take some time to implement properly. And now that you understand the importance of rightsizing, you may want to at least enforce defaults when pods are deployed to Kubernetes without specified request values.In Kubernetes, you can establish default value configurations for memory and CPU at the namespace level without any third-party tools. Alternatively, if you need more granular control or different mechanisms, you can configure mutating admission webhooks to modify a pod before it reaches the Kubernetes API server and enforce default request values. Tools like Kyverno, Open Policy Agent (OPA), or Gatekeeper are candidates to consider for this purpose.

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-defaults
spec:
  limits:
  - default:
      cpu: 1200m
    defaultRequest:
      cpu: 900m
    type: Container

$ kubectl delete deployment montecarlo-pi

$ kubectl create namespace montecarlo-pi

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-memory-requests
  namespace: montecarlo-pi
spec:
  limits:
    - defaultRequest:
        cpu: 900m
        memory: 512Mi
      type: Container
EOF

$ kubectl create deployment montecarlo-pi \
--image=christianhxc/montecarlo-pi:latest -n montecarlo-pi

$ kubectl get pod -n montecarlo-pi -o yaml

  ...
  spec:
    containers:
    - image: christianhxc/montecarlo-pi:latest
      imagePullPolicy: Always
      name: montecarlo-pi
      resources:
        requests:
          cpu: 900m
          memory: 512Mi
  ...

Horizontal Pod Autoscaler (HPA)

HPA is often the starting point, and sometimes the only option people use for autoscaling their workloads. It's simple and a native resource of Kubernetes. HPA has been an integral part of the ecosystem since its introduction in Kubernetes 1.1. Developed by the Kubernetes community, HPA addresses the challenge of managing fluctuating workloads running in Kubernetes. It automatically adjusts the number of pod replicas (horizontal scaling) for compatible Kubernetes objects such as Deployments, ReplicaSets, and StatefulSets.HPA's primary purpose is to solve the problem of efficiently scaling applications in response to varying demand. It does this by monitoring specified metrics and adjusting the number of replicas accordingly. While CPU utilization is the default metric, HPA can also work with memory usage, custom metrics, and even external metrics. These metrics are sourced from various components within the Kubernetes ecosystem. The Metrics Server provides CPU and memory data, while systems like Prometheus can supply custom metrics. For more complex scenarios, external monitoring systems can feed metrics into HPA.To illustrate HPA's functionality, consider a web application like the Monte Carlo simulation we've been using, experiencing varying traffic throughout the day. HPA can be configured to maintain an average CPU utilization of 70% across all pods. As shown in Figure 2.5, as traffic increases and CPU usage rises above this threshold, HPA will automatically increase the number of pod replicas. Conversely, during periods of low traffic, it will scale down the number of replicas, optimizing resource usage and potentially reducing costs.

HPA efficiently manages the number of pod replicas but doesn't adjust resources for individual pods nor containers. When it comes to adjusting resources allocated to individual pods HPA isn't the solution. For this type of scaling, we have VPA.

Vertical Pod Autoscaler (VPA)

VPA is designed to automatically adjust the CPU and memory resources allocated to containers in pods. Unlike its horizontal counterpart HPA, VPA focuses on optimizing the resource requests and limits of individual containers, rather than scaling the number of pod replicas. VPA was introduced by Google in 2018 as part of their efforts to enhance Kubernetes' autoscaling capabilities. While it's not a native Kubernetes resource like HPA, VPA has become part of many Kubernetes deployments due to its ability to fine-tune resource allocation.VPA can be applied to various Kubernetes objects, including Deployments, ReplicaSets, StatefulSets, DaemonSets, and even individual pods. VPA addresses the common problem of over- or under-provisioning resources, which can lead to either wasted cluster capacity or performance issues. To make its scaling decisions, VPA relies on historical and current resource usage metrics. It primarily focuses on CPU and memory utilization, gathering this data from the Metrics Server.Consider a scenario with a large, legacy monolithic application or a complex database system that's challenging to scale horizontally due to intricate internal dependencies or stateful components. In Figure 2.6, as the data grows, its resource needs increase. VPA would continuously monitor this application's resource usage to recommend or adjust its requests accordingly. Conversively, if it determines that the system is consistently using less resources, it can recommend or automatically apply lower resource requests.

When VPA needs to adjust the resources for a pod, it typically does so by deleting the existing pod and creating a new one with the updated resource specifications. This process can lead to a brief period of downtime for that specific pod. Additionally, even though VPA can work alongside HPA, they should be used carefully when combined, as their actions can potentially conflict. We'll address these concerns in the next chapter.

Kubernetes Event-Driven Autoscaling (KEDA)

HPA provides a solid foundation for autoscaling in Kubernetes, but relying solely on CPU and memory metrics may not capture the full picture of how an application performs under varying loads. Certainly, HPA can use custom metrics, but this might increase complexity when configuring scaling policies. This is where KEDA comes into play, offering a flexible but simple approach to scaling.KEDA is an open-source project that was initially developed by Microsoft and Red Hat in 2019. Although it's not a native Kubernetes resource, KEDA has gained significant traction in the community due to its versatility and power. It extends Kubernetes' autoscaling capabilities beyond traditional resource metrics, allowing for scaling based on event sources and custom metrics.Similarly to the other autoscalers, KEDA can be applied to various Kubernetes objects, including Deployments, StatefulSets, and custom resources. Its primary purpose is to scale applications based on events and application-specific metrics, rather than just system-level resource utilization. This makes KEDA particularly useful for event-driven architectures, microservices, and applications with unpredictable or bursty workloads.One of KEDA's key differantiators is its ability to use a wide range of metrics for scaling decisions. Unlike HPA, which primarily focuses on CPU and memory, KEDA can scale based on metrics from various sources such as message queues, databases, streaming platforms, and other external systems. This allows for more nuanced and application-aware scaling decisions.For example, as per Figure 2.7, consider a microservice that processes orders from a message queue:

As evident in the figure, with KEDA, you could scale this service based on the number of messages in the queue. As the queue length grows, KEDA would automatically scale up the number of pods to process orders more quickly. When the queue empties, it would scale back down, optimizing resource usage.KEDA is not without its limitations. You might find that a specific connector you need isn't available, or conversely, you might feel overwhelmed by the sheer number of connectors to choose from. It's worth noting, though, that the KEDA maintainers team is addressing quality concerns by requiring any new connector to implement automated test suites, ensuring reliability and consistency across the growing ecosystem of scalersThat's all for the basics of Kubernetes workload autoscalers. Each has its strengths, and knowing when to use HPA, VPA, or KEDA can make a big difference in how your workloads perform. Let's dive deeper into these tools in the following chapters, and see how you can use them to automatically scale your workloads to start working on having an efficient Kubernetes cluster.

Metric Server: The What and the Why

The Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data. While not a built-in component of Kubernetes, it's a crucial add-on that collects and aggregates essential metrics about the resource consumption of nodes and pods in a cluster. The Metrics Server is designed to support Kubernetes' built-in autoscaling mechanisms. It focuses solely on gathering CPU and memory usage for nodes and pods, making it a lean and purpose-built tool.As shown in Figure 3.1, the Metrics Server operates by collecting resource metrics from the kubelet on each node. It aggregates this data and makes it available through the Kubernetes Metrics API. The process begins with the kubelet collecting resource usage statistics via its cAdvisor integration. The Metrics Server then queries each node's kubelet for this data every 15 seconds by default. After aggregating the data, it exposes it through the Metrics API, allowing Kubernetes components like HPA to query for up-to-date metrics.

As you've seen, the Metrics Server only provides the current state of resource usage, not historical data. Its focus is solely on CPU and memory metrics, and the data is stored in memory rather than persisted to disk. Metrics Server is solely used for autoscaling your workloads. For monitoring needs, solutions like Prometheus, or any other paid solution, will server you better for monitoring. You can even use Prometheus as a source for scaling using custom metrics, I'll cover this later in this chapter.Now that you have a better understand the Metrics Server, and why it's important, let's set it up in a Kubernetes cluster.

Hands-On: Setting up Metrics Server

Note

If you're using the Kubernetes cluster in AWS created using the Terraform templates in Chapter 1, you can skip this section as the metrics server comes pre-installed in the Terraform blueprint I've provided for the book.

To install the Metrics Server using Helm, follow these steps.Add the official Metrics Server Helm repository:

$ helm repo add metrics-server \
https://kubernetes-sigs.github.io/metrics-server/

Update your Helm repository cache:

$ helm repo update

Install the Metrics Server:

$ helm install metrics-server metrics-server/metrics-server

Optionally, if you're using a local or self-hosted Kubernetes cluster, you might need to disable TLS certificate verification. In this case, use the following command instead:

$ helm install metrics-server metrics-server/metrics-server \
--set args="{--kubelet-insecure-tls}"

Verify the installation by checking if the Metrics Server pod is running:

$ kubectl get pods -n kube-system | grep metrics-server

Once these steps are completed, your Metrics Server should be up and running. You can verify that this pod is properly configured with CPU (100m) and memory (200Mi) requests. While you can adjust this configuration if needed, these default settings should provide good performance for most clusters with up to 100 nodes.

Hands-On: Using Metrics Server

Once you have this addon installed, HPA and VPA will use it behind the scenes. It will also expose some of the data it collects through the kubectl top command. This command allows you to see how much resources either nodes or pods are consuming.For instance, let's view the resource usage of all the nodes in your cluster with:

$ kubectl top nodes

You should see an output similar to this (depending on how many nodes):

NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-0-...  1676m        42%    1691Mi          11%
ip-10-0-...  2402m        61%    3209Mi          21%

As you can see, it's a much simpler version of the top command in Linux, typically used for monitoring purposes. It only shows you the CPU millicores and Memory bytes each node has, and what percentage of those resources are being used at the moment you ran the command.You can also see the resource usage for all the pods in a given namespace:

$ kubectl top pods -n default

You should see an output similar to this (depending on which pods you have running):

NAME                             CPU(cores)   MEMORY(bytes)
montecarlo-pi-5d9d967b77-hh87b   14m          36Mi

Notice that for pods, it only shows you the usage numbers, not the percentage. But that's sufficient for HPA and VPA. Remember, this command and the Metrics Server shouldn't be used as a monitoring tool. However, for a quick status overview, it's adequate.If you want to learn more about other options like sorting or filtering resources, you can add the --help flag to each command.Now that you have Metrics Server working, let's get into the details of how to configure autoscaling policies with HPA and VPA.

How HPA scale resources?

HPA operates on a feedback loop, using a simple calculation to determine the number of replicas needed. Let's say you have a Deployment targeting 70% CPU utilization across all its pods. The process begins with HPA querying the Metrics Server (typically every 15 seconds) to get the current CPU utilization for all pods in the target Deployment. It then calculates the average CPU utilization across these pods.HPA uses the following formula to determine the desired number of replicas:

desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)]

Following with the 70% CPU target example, if the current average CPU utilization (currentMetricValue) is 80% and we have 5 replicas, HPA would scale out to 6 replicas:

desiredReplicas = ceil[5 * (80 / 70)] = 6

If the current utilization then drops to 60%, HPA would go back to 5 replicas:

desiredReplicas = ceil[5 * (60 / 70)] = 5

When you're using multiple metrics, the desiredReplicas calculation is done for each metric, and the largest number is chosen. Keep this in mind as this will become an important aspect in Chapter 5. It's important to note that HPA has a tolerance (default 0.1 or 10%) to prevent unnecessary scaling for minor fluctuations. If the ratio of currentMetricValue to desiredMetricValue is between 0.9 and 1.1, HPA won't initiate any scaling action.After determining the desiredReplicas value, HPA updates the replica count on the Deployment object. Kubernetes then creates or removes pods to match this new desired state. This feedback loop continues indefinitely, allowing your application to automatically adjust to changing load conditions. However, to prevent rapid fluctuations, HPA typically observes a cooldown period (default is 5 minutes) after each scaling action before performing another.

Defining HPA Scaling Policies

To define an autoscaling policy with HPA, you need to create a Kubernetes object of kind HorizontalPodAutoscaler. The current API version for this resource is autoscaling/v2. Below is an example of the simplified YAML manifest for an HPA object, which we'll break down and explain in detail:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-usage-70
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: montecarlo-pi
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      ...
    scaleUp:
      ...

Let's break down the key fields:

• scaleTargetRef: Specifies the target resource to scale, like a Deployment.
• minReplicas: Define the lower bound for the number of replicas.
• maxReplicas: Define the upper bound for the number of replicas.
• metrics: Specifies the metric(s) to use for scaling, like CPU utilization.
• behavior: Defines policies for scaling up and down to prevent rapid fluctuations.

While using YAML manifests is recommended for Infrastructure as Code (IaC) practices, you can also create an HPA rule using kubectl. Here's an example command:

$ kubectl autoscale deployment montecarlo-pi \
--cpu-percent=70 --min=1 --max=10

This command creates an HPA rule for the montecarlo-pi deployment, targeting 70% CPU utilization, with a minimum of 1 replica and a maximum of 10 replicas. It's exactly the same as the YAML manifest you reviewed before.

Hands-On: Scaling using basic metrics with HPA

Let's put in practice what we've learned so far. Grab your computer, and open the CLI.We're going to use ApacheBench (ab), a benchmarking tool from Apache, to send load to the application. You don't need to install this tool locally as we're going to run it within the Kubernetes cluster.To have an almost real-time visibility of what's happening while you see HPA in action, we're going to use the watch command. This comes natively on Linux, but if you're using Mac or Windows, you need to install watch. In Mac, you simply need to run brew install watch. In Windows, use the Windows Subsystem for Linux (WSL) or install it using choco install watch.Go to the book's GitHub repository folder that you've already cloned, and change to the chapter03 directory. But in case you haven't done it, run these commands:

$ git clone https://github.com/PacktPublishing/Kubernetes-Autoscaling.git
$ cd Kubernetes-Autoscaling/chapter03

$ kubectl apply -f hpa-basic/montecarlopi.yaml

$ kubectl port-forward svc/montecarlo-pi 8080:80

http://localhost:8080/monte-carlo-pi?iterations=100000

$ kubectl apply -f hpa-basic/hpa.yaml

$ kubectl get hpa

NAME    REFERENCE  TARGETS      MINPODS  MAXPODS  REPLICAS  AGE
mont... Deploy...  cpu: 0%/70%  1        10       1         41s

ab -n 10000 -c 10 -t 150 \
http://montecarlo-pi/monte-carlo-pi?iterations=100000

$ kubectl apply -f ab-k8s/loadtest.yaml

$ watch kubectl top pods

$ watch kubectl get pods

$ kubectl logs job/montecarlo-pi-load-test -f

$ kubectl get hpa

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: montecarlo-pi-hpa
spec:
  # ... other fields ...
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: montecarlo-pi-hpa
spec:
  # ... other fields ...
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60  # 1m instead of 5m
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

$ kubectl delete -f ab-k8s

$ kubectl apply -f hpa-basic/hpa.fast.yaml

$ kubectl apply -f ab-k8s/loadtest.yaml

$ kubectl delete -f ab-k8s
$ kubectl delete -f hpa-basic

How does HPA work with custom metrics?

For HPA to use custom metrics, you need two key components:

• A way to collect and store these metrics
• An adapter that can expose these metrics through the Custom Metrics API

You might be able to achieve this with your existing monitoring system. As shown in Figure 3.2, a common approach involves integrating Prometheus with the Prometheus Adapter. In this setup, Prometheus serves as the metrics collector and storage solution. The Prometheus Adapter then acts as a bridge, implementing the Custom Metrics API and exposing the data collected by Prometheus through this standardized interface. This combination allows for HPA make scaling decisions based on a custom set of metrics different than CPU and memory, like latency or number of requests.

Let's get into the details of what you saw in Figure 3.2 flow. The Prometheus Adapter registers itself with the Kubernetes API server as an API Service. This registration informs Kubernetes that the adapter can handle requests for Custom Metrics. The adapter is then configured with rules that define how to translate Prometheus metrics into Kubernetes Custom Metrics, specifying which Prometheus metrics to expose and how to name them in the Kubernetes API.When HPA requests a metric, the API server forwards this request to the Prometheus Adapter. The adapter translates this into a Prometheus query, executes it against the Prometheus server, and then translates the results back into the format expected by Kubernetes.The adapter exposes these metrics at an API endpoint that follows the Kubernetes custom metrics API format. For example, a metric might be accessible at a path like /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/monte_carlo_latency_seconds. HPA can then be configured to use these custom metrics in its scaling decisions. Let's see that in action.

Hands-On: Scaling using custom metrics with HPA

Before you get started, make sure to have the Prometheus stack running. You already did this in the previous chapter, but just in case you need to install the stack again, here's the command to do it:

$ helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Confirm that all the pods are running using this command:

$ kubectl get pods -n monitoring

$ helm install prometheus-adapter \
prometheus-community/prometheus-adapter \
  --namespace monitoring \
  -f prometheus-adapter/values.yaml

$ kubectl apply -f hpa-custom/montecarlopi.yaml

$ kubectl port-forward svc/montecarlo-pi 8080:80

http://localhost:8080/metrics

# TYPE monte_carlo_latency_seconds histogram
monte_carlo_latency_seconds_bucket{le="0.001"} 0
monte_carlo_latency_seconds_bucket{le="0.005"} 1
monte_carlo_latency_seconds_bucket{le="0.01"} 1
...
monte_carlo_latency_seconds_count 11

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: montecarlo-pi
  labels:
    app: montecarlo-pi
    release: prometheus
spec:
  selector:
    matchLabels:
      app: montecarlo-pi
  endpoints:
  - port: http

$ kubectl apply -f hpa-custom/monitor.yaml

$ kubectl get --raw \
"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/"\
"monte_carlo_latency_seconds"

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "montecarlo-pi-574cc7ff9f-h77wp",
        "apiVersion": "/v1"
      },
      "metricName": "monte_carlo_latency_seconds",
      "timestamp": "2024-10-08T20:57:08Z",
      "value": "0",
      "selector": null
    }
  ]
}

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: montecarlo-pi-latency-hpa
spec:
  scaleTargetRef:
    apiVersio dn: apps/v1
    kind: Deployment
    name: montecarlo-pi
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: monte_carlo_latency_seconds
      target:
        type: AverageValue
        averageValue: 500m

$ kubectl apply -f ab-k8s/loadtest.yaml

$ watch kubectl top pods

$ watch kubectl get pods

$ kubectl get hpa

resources:
  requests:
    cpu: 900m
    memory: 512Mi
  limits:
    cpu: 1200m
    memory: 512Mi

$ kubectl delete -f ab-k8s
$ kubectl delete -f hpa-custom

How VPA scale resources?

VPA works with three different controllers you deploy in the cluster:

• Recommender: this controller continuously monitors the resource usage of pods and containers, analyzing historical data to generate optimal resource recommendations. Then, it calculates the ideal CPU and memory requests for each container, taking into account factors like usage patterns, application behavior, and defined constraints.
• Updater: this controller is responsible for applying the recommendations generated by the Recommender. It decides when and how to update the resource requests of running pods. Depending on the VPA mode (Auto, Recreate, or Initial), the Updater may evict pods to apply new resource settings or wait for pods to be naturally recreated during deployments or restarts.
• Admission Controller: this controller intercepts pod creation requests. When a new pod is about to be created, it checks if there's a VPA policy applicable to that pod. If so, it modifies the pod's resource requests according to the latest recommendations before the pod is actually created. This ensures that even newly created pods start with optimized resource settings.

As shown in Figure 3.3, VPA starts by gathering CPU and memory usage data from pods in the cluster from the Metrics Server. It's an ongoing process, so it allows VPA to understand application's resource needs over time. Then, VPA generates recommendations for optimal CPU and memory allocations for each container within a pod. These recommendations aim to strike a balance between ensuring the application has enough resources to perform well, and avoiding over-provisioning that could lead to wasted resources.

Then, VPA can automatically apply these usage recommendations by updating the pod specifications by terminating the pod first. Then, through an Admission Controller, it modifies the pod spec before is persisted. Alternatively, it simply provides the recommendations without taking action. VPA respects configured resource policies, such as minimum and maximum limits. It also considers factors like Out of Memory (OOM) events and CPU throttling when making recommendations.Lastly, keep in mind that using VPA alongside HPA for CPU and memory scaling can lead to conflicts, as both tools attempt to solve the same problem in different ways. However, VPA can work well with HPA when the latter is scaling based on custom or external metrics.

Defining VPA Scaling Policies

Now let's look at how to define a VPA scaling policy. As VPA is not a native Kubernetes feature, you define autoscaling policies through a custom resource defintion (CRD) called VerticalPodAutoscaler.Here's a simple VPA rule:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: montecarlo-pi-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: montecarlo-pi
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: 100m
        memory: 512Mi
      maxAllowed:
        cpu: 1200m
        memory: 512Mi
      controlledResources: ["cpu", "memory"]

In this example, VPA will automatically adjust CPU and memory for all containers in the montecarlo-pi Deployment.Let's break down each section:

• targetRef indicates which Kubernetes object this VPA should manage. In this case, it's a Deployment named montecarlo-pi.
• updatePolicy section defines how VPA should apply changes, and there are four modes you can canfigure:
  • Auto: assigns resource request when pods are created, or evict pods to update resources to existing pods. When in-place update feature (AEP-4016) is generally available (GA) in Kubernetes (it's currently in alpha), VPA will apply recommendations without disruption, and it will become the prefered mode.
  • Recreate: apply recommendations at creation time, or recreates existing pods. For now, it's the same as Auto as in-place updates is not GA yet. However, it will be recommended to use only if you really need to recreate the pods when applying recommendations.
  • Initial: apply recommendations only when pods are created.
  • Off: recommendations can only be seen by inspecting the VPA object.
• resourcePolicy section is used to define rules like limits VPA needs to respect
  • containerPolicies allows you to set policies for specific or all containers
  • minAllowed and maxAllowed set the resource boundaries to respect
  • controlledResources is to specify which resources VPA should manage

This basic policy provides a good starting point, but remember that you can refine these settings based on your application's specific needs and your cluster's resources. You might set different limits for different containers, or create Pod Disruption Budgets (PDBs) to be less aggressive if you're dealing with a critical application.If you don't want to apply recommendations automatically, you can use the Initial mode to still get the benefits of VPA, but apply them the next time a deployment or upgrade happens, as this will create new pods. Alternatively, you can simply use Off mode to get recommendations from VPA, but you'll own the decision of applying them or not. You can also exclude containers that don't need to scale by using Off under the containerPolicies section.It's worth mentioning that VPA will update resources only if there is more than one replica. You can change this behavior either for all VPA rules within the VPA updater component using the min-replicas parameter, or by adding a minReplicas value to the corresponding VPA rule, like this:

...
updatePolicy:
    updateMode: 'Auto'
    minReplicas: 1
...

The recommendation is to configure this value at the VPA rule level, as there might be workloads for which you'd prefer VPA not to perform any action to avoid causing downtime.Now that you understand how to configure VPA policies and what each field does, it's time to see it in action. In the next section, you'll use VPA in Auto mode to dynamically adjust resource allocations based on real usage.

Hands-On: Automatic Vertical Scaling with VPA

To watch VPA in action, we're going to start using VPA in Auto mode to adjust the resources of pods (vertical scaling) without any human intervention.

$ git clone https://github.com/kubernetes/autoscaler.git
$ cd autoscaler/vertical-pod-autoscaler
$ ./pkg/admission-controller/gencerts.sh
$ ./hack/vpa-up.sh

$ kubectl get pods -n kube-system | grep vpa

$ kubectl apply -f vpa/vpa.auto.yaml

$ kubectl apply -f vpa/montecarlopi.yaml

$ watch kubectl get vpa

NAME            MODE   CPU    MEM       PROVIDED   AGE
montecarlo...   Auto   100m   262144k   True       60s

$ kubectl describe montecarlo-pi-vpa-auto

...
  Recommendation:
    Container Recommendations:
      Container Name:  montecarlo-pi
      Lower Bound:
        Cpu:     100m
        Memory:  262144k
      Target:
        Cpu:     100m
        Memory:  262144k
      Uncapped Target:
        Cpu:     25m
        Memory:  262144k
      Upper Bound:
        Cpu:     100m
        Memory:  262144k
...

$ watch kubectl get pods

...
    resources:
      requests:
        cpu: 100m
        memory: 262144k
...

$ kubectl apply -f ab-k8s/loadtest.yaml

NAME            MODE   CPU     MEM       PROVIDED   AGE
montecarlo...   Auto   1554m   262144k   True       6m

$ kubectl rollout restart deployment montecarlo-pi

...
    resources:
      requests:
        cpu: 1554m
        memory: 262144k
...

$ kubectl delete -f ab-k8s
$ kubectl delete -f vpa

KEDA Operator

The KEDA Operator is responsible for monitoring your cluster for KEDA-specific CRDs such as ScaledObjects and ScaledJobs. When it detects these custom resources, it starts to work by creating and managing the necessary HPA rules based on the scaling rules defined in these objects. Moreover, the KEDA Operator interacts directly with external event sources. Whether it's a message queue, a database, or a custom metric endpoint, the Operator can query these sources directly to determine the current scaling needs. As I said before, this direct interaction eliminates the need for intermediate adapters, reducing latency and improving the responsiveness of your autoscaling setup.

KEDA Metrics Server

The KEDA Metrics Server works alongside the Operator. This component acts as a bridge between KEDA and Kubernetes' autoscaling mechanisms. It exposes the metrics collected by the KEDA Operator to the Kubernetes Metrics API as external metrics, making them available for consumption by HPAs.The Metrics Server is what allows KEDA to act as an abstraction layer for HPA. It translates the rich, event-driven metrics that KEDA understands into a format that Kubernetes' native autoscaling can work with. In other words, it's the adapter. This integration means you can leverage KEDA's scaling capabilities without having to modify your existing Kubernetes setup or your applications.

Admission Webhooks

These webhooks serve as a crucial safeguard in the KEDA ecosystem, automatically validating resource changes to prevent misconfiguration and enforce best practices. They function as an admission controller, intercepting requests to the Kubernetes API server before the persistence of the object.One of the primary functions of KEDA's Admission Webhooks is to prevent multiple ScaledObjects from targeting the same scale target. This is crucial for maintaining the integrity and predictability of your scaling setup. The Admission Webhooks are enabled by default when you install KEDA, providing an additional layer of safety and consistency in your Kubernetes cluster(s).

What Kubernetes objects can KEDA scale?

KEDA can scale the following resource types:

• Deployments: The most common use case, perfect for stateless applications.
• StatefulSets: Ideal for applications that require stable, unique network identifiers or persistent storage like databases.
• Custom Resources: KEDA can scale custom resources defined by Operators, opening up possibilities for scaling complex, application-specific workloads, like ArgoCD objects.

Let's dive deeper into the specific CRDs that KEDA introduces and how you can leverage them to define sophisticated scaling behaviors for your applications.

ScaledObjects

ScaledObjects define how a Deployment, StatefulSet, or a custom resource should scale using triggers. Based on the event type configured for the trigger, KEDA will use that scaler to monitor its events or metrics, and expose it so that HPA can use it to scale out or scale in.One of the key features of ScaledObjects is the ability to pause autoscaling, which can be particularly useful during maintenance windows or when you need to temporarily override KEDA's scaling decisions. We'll dive deep into more advanced features in Chapter 5.Here's an example of a ScaledObject that scales a deployment based on the length of a RabbitMQ queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer-app-scaler
spec:
  scaleTargetRef:
    name: consumer-app
  pollingInterval: 30
  cooldownPeriod:  300
  minReplicaCount: 0
  maxReplicaCount: 30
  triggers:
  - type: rabbitmq
    metadata:
      queueName: orders
      mode: QueueLength
      value: "20"
    authenticationRef:
      name: consumer-app-trigger

This ScaledObject will scale the consumer-app deployment based on the length of the orders queue in RabbitMQ. It checks the queue every 30 seconds (pollingInterval), the application then consumes multiple events from the queue, and KEDA can scale the deployment up to 30 replicas (maxReplicaCount). When there are no messages, KEDA will scale down the deployment to zero replicas (minReplicaCount).Now, let's break down the parameters from the previous ScaledObject example:

• scaleTargetRef is the resource KEDA will scale. If the target is a Deployment, you simply specify the name. But if the target is a StatefulSet or a custom resource, you need to specify the apiVersion, and its kind.
• pollingInternval is the internal that KEDA use to check the trigger source. By default, the interval is 30 seconds.
• cooldownInterval is the interval that KEDA waits after the last trigger was active before scaling to zero. By default, the internal is 300 seconds.
• minReplicaCount is the minimum number of replicas KEDA will set the resource when scaling down. By default, the minimum number is zero.
• maxReplicaCount is the maximum number of replicas KEDA will set the resource when scaling up. By default, the maximum number is 100.
• triggers are where you define the properties of the scalers to use.

The ScaledObject CRD has many more parameters and properties that we'll continue exploring throughout this chapter and the next one. For now, you have at least an idea of what a basic configuration looks like. As we progress, you'll learn how to leverage these additional features to create more sophisticated and tailored scaling solutions for different use cases.

ScaledJobs

ScaledJobs are designed for batch jobs or tasks that need to run to completion. They create Kubernetes Jobs based on scaling rules. Instead of processing multiple events within a Deployment, KEDA will create one Job per event, and process it. When the Job finishes processing the event, or fails because there was an error, the Job will be terminated. It's up to the process, or application in this case, how many events will pull down, and based on this you need to configure the scaling rule.Here's an example of a ScaledJob to process messages from a RabbitMQ queue:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: rabbitmq-consumer
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: rabbitmq-client
          image: rabbitmq-client:v1.0.1
          command: ["receive",  "amqp://user:PASSWORD@rabbitmq.default.svc          .cluster.local:5672"]
          envFrom:
            - secretRef:
                name: rabbitmq-consumer-secrets
        restartPolicy: Never
  pollingInterval: 10
  minReplicaCount: 0
  maxReplicaCount: 30
  triggers:
  - type: rabbitmq
    metadata:
      queueName: orders
      mode: QueueLength
      value: "20"
      authenticationRef:
        name: consumer-app-trigger

This ScaledJob will create Jobs to process messages from the orders queue, when there are at least 20 messages in the queue (value). It will create up to 30 Job replicas (maxReplicaCount), and it will keep a history of 5 successful (successfulJobsHistoryLimit) and 2 failed jobs (failedJobsHistoryLimit).Now, let's break down the parameters from the previous ScaledJob example:

• jobTargetRef is the spec you configure for the Job KEDA will create, and the template field is required for that same reason.
• pollingInternval is the internal that KEDA use to check the trigger source. By default, the interval is 30 seconds.
• minReplicaCount is the minimum of pods KEDA creates. By default, the minimum is zero. If it's different than zero, KEDA will keep a minimum of Jobs running, which it's useful in case you don't want to wait for new Jobs to be ready.
• maxReplicaCount is the maximum number of pods KEDA will create every poll interval. If there are Jobs running, KEDA will deduct those pods. By default, the maximum number is 100.
• triggers are where you define the properties of the scalers to use.

The ScaledJob CRD has many more parameters and properties that we'll continue exploring throughout this chapter and the next one. For now, you have at least an idea of what a basic configuration looks like.

TriggerAuthentication and ClusterTriggerAuthentication

There might be some KEDA scalers that will require authentication. To cover this need, the TriggerAuthentication and ClusterTriggerAuthentication CRDs provide a way to securely manage authentication parameters. The difference between these two is that TriggerAuthentication is namespace-scoped, while ClusterTriggerAuthentication is cluster-wide.Continuing with the RabbitMQ example, here's an example of a TriggerAuthentication that uses a Kubernetes secret to authenticate with the RabbitMQ server:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: consumer-app-trigger
spec:
  secretTargetRef:
    - parameter: host
      name: rabbitmq-consumer-secret
      key: RabbitMqHost

With TriggerAuthentication, you decouple the authentication part from the scaling part, and you can re-use authentication within multiple ScaledObject instances. I showed you a very simple example of the CRD manifest, you can check the official documentation to see what else you can configure, and in the next chapter we'll explore a few other parameters. For example, you can configure a Pod Identity when using a cloud provider, use vault secrets from HashiCorp, or work with secrets management solutions from cloud providers like AWS Secrets Manager, Azure Key Vault, or GCP Secrets Manager.

Hands-On: Installing KEDA

Let's start with your first hands-on lab for this chapter. Go to your terminal, spin up your Kubernetes cluster again (go back to Chapter 1 to follow the lab to setup your environment), and run the following commands to install KEDA:

$ helm repo add kedacore https://kedacore.github.io/charts
$ helm repo update
$ helm install keda kedacore/keda --namespace keda --create-namespace

When the installation finishes, you should be able to see the KEDA controllers:

$ kubectl get pods -n keda

You should see an output like this:

NAME                        READY   STATUS    RESTARTS AGE
keda-admission-webhooks-... 1/1     Running   1        3m
keda-operator-...           1/1     Running   1        3m
keda-operator-metrics-...   1/1     Running   1        3m

Alright, you're now ready to use KEDA to scale your workloads. Let's do that next.

Hands-On: Scaling using Latency

You've already practiced scaling your workloads using CPU utilization with HPA in the previous chapter. So instead of that, let's see how you'll scale your application using a custom metric. You already did this with HPA, but this time I want you to see the main difference in KEDA where you don't need the Prometheus Adapter, because KEDA will go directly to the source (Prometheus) to calculate the replicas the workload needs. This has other additional advantages like being able to use multiple Prometheus instances, and no need to configure an adapter through a ConfigMap (which has a limitation of 1 Mb size anyway).So, let's open a terminal, and change directory to the Book's GitHub repository at the chapter04 folder. Then, run the following command to deploy the Monte Carlo application:

$ kubectl apply -f api/montecarlopi.yaml

Notice that similarly as before, the application is intentional on how much CPU and memory is requesting, and it's limiting them as well to influence the response time when the load increase so that we can scale out horizontally:

        resources:
          requests:
            cpu: 900m
            memory: 512Mi
          limits:
            cpu: 1200m
            memory: 512Mi

Before you continue, let's confirm you have all the Prometheus pods running, as well as the Prometheus service by running this command:

$ kubectl get pods -n monitoring

Note

If you don't have the Prometheus stack installed, go back to Chapter 2 to install it.

The application is exposing custom latency metrics at the /metrics endpoint, and in order to push those metrics to Prometheus, you need to add a ServiceMonitor. There's a YAML definition for this in the repository as well, if you explore the file chapter04/monitor.yaml, you'll see that it simply targets the application:

spec:
  selector:
    matchLabels:
      app: montecarlo-pi
  endpoints:
  - port: http

Deploy the service monitor by running this command:

$ kubectl apply -f api/monitor.yaml

Wait around five minutes for Prometheus to consider this new monitor to start scrapping metrics from the application's /metrics endpoint. After this time has passed, deploy the KEDA's ScaledObject to define an autoscaling rule for the application using the monte_carlo_latency_seconds_bucket metric. You should be using the fillowing trigger:

triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc:9090
      metricName: monte_carlo_latency_seconds
      threshold: "0.5" # You can't specify 500m, it needs to be translated
                       # to seconds
      query: sum(histogram_quantile(0.95,
             rate(monte_carlo_latency_seconds_bucket{namespace="default",
             pod=~"montecarlo-pi-.*"}[2m])))

Notice that it's pointing directly to the Prometheus service at the serverAddress parameter, and it's also defining a latency target of 0.5 seconds at the threshold parameter. Additionally, look at the query parameter. It has a very similar query from the one you used at Chapter 3 when using the Prometheus adapter.Deploy the above scaled object by running this command:

$ kubectl apply -f api/latency.yaml

Confirm that the HPA rule has been created by running this command:

$ kubectl get hpa keda-hpa-montecarlo-pi-latency

Let's see autoscaling in action by sending some load test, run this command:

$ kubectl apply -f ab-k8s/loadtest.yaml

Now monitor what happens by running this command:

$ watch kubectl get scaledobject,hpa,pods

You should see that the latency starts to increase by looking at the TARGET column of the HPA rule, and the number of replicas starts to increase very rapidly to try to keep the latency number at its target of 0.5 seconds (or 500ms). Even if with 10 replicas the latency doesn't meet its target, for this lab it's fine as we only wanted to see how KEDA's autoscaling rule reacts to a metric exposed by the application. You can play around by adjusting the maxReplicaCount parameter from the ScaledObject to see if with a maximum of 15 replicas the latency meets its target.Wait around 10 extra minutes to see how the number of replicas go back to 1 after the TARGET column from HPA goes back to 0.To clean up this lab, run the following commands:

$ kubectl delete -f api

Controlling autoscaling speed

In the previous chapter, when we dived deep into HPA speed of scaling, you learned you can control HPA's speed by configuring advance settings such as scaleUp and scaleDown behaviors. For instance, you can make KEDA to react faster when scaling up, and react more conservative (or faster than the default behavior) when scaling down. As KEDA is using HPA under the hood, you can customize the scaling speed through the advanced parameter in a ScaledObject, like this:

...
advanced:
    behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
...

Notice that we continue using the same configuration from the previous chapter. Feel free to give it a try to confirm that you were able to replicate the same autoscaling rule you had in HPA. But effectively, the previous configuration won't wait to scale up as the stabilizationWindowSeconds is 0, and will wait 60 seconds to starts scaling down. Both behaviors are using the Percent type, and the same policy configuration, which means that it can increase or decrease 100% of the current pods every 15 seconds (periodSeconds).

Scaling to Zero

One of the most attractive features of KEDA is the ability to scale down to zero replicas. You can reduce wast/costs when your application isn't needed. KEDA can bring your deployment to zero replicas when needed, and then scale it back up when activity resumes.Scaling to zero works well in certain scenarios. For example, if you're scaling based on the number of messages in a queue, KEDA can scale to zero when the queue is empty. However, scaling to zero isn't suitable for all types of applications or scalers. Take REST APIs as an example. If you're scaling based on latency or number of requests, scaling to zero can cause problems. When there are no replicas running, the application won't be able to handle any requests at all, leading to errors or timeouts. This can result in a poor user experience. Because of this limitation, scaling to zero it's best suited for event-driven workloads where you can afford to have periods of inactivity without impacting service availability. Although, it's worth noticing that scaling to zero even with HTTP-based applications is doable if using the Cron Job scaler or using Knative, we'll dive deep about scaling to zero HTTP-based applications in the next chapter.It's important to consider your application's nature and requirements before deciding to implement scaling to zero. For always-on services or APIs, maintaining at least one replica might be a better approach to ensure continuous availability.Let's see scaling from and to zero in action by scaling a queue-based workload.

Hands-On: Scaling from/to Zero

A queue-based workload is a scenario where scaling to zero makes perfect sense. If there are messages in the queue, KEDA activates the Deployment, add the necessary replicas to process messages in the queue. As messages are consumed, less replicas are needed. When there are no messages in the queue, KEDA pauses the Deployment in order to have zero replicas running.A simple way to demonstrate this scenario is to use a RabbitMQ queue. Think a video or image encoder, you might have a producer that sends messages to the queue, and a consumer that processes messages from the queue. We've provided a simple app that does something similar. I'll get into the details of how it was built, and why we're configuring KEDA in such a way.

Deploy a RabbitMQ queue

Let's start by deploying a RabbitMQ queue to your cluster by running these commands:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install rabbitmq --set auth.username=user \
--set auth.password=autoscaling bitnami/rabbitmq --wait

Wait around two minutes and the RabbitMQ pod should be running.

NOTE

If you're using Kind, you need to add the --set volumePermissions.enabled=true parameter to the helm install command.

It would be nice if you can keep an eye open to the messages that arrive to the queue. To do so, let's make use of RabbitMQ's UI. Run this command to expose the service locally:

$ kubectl port-forward svc/rabbitmq 15672:15672

Open the following URL in the browser: http://localhost:15672/#/queues. You should see a screen similar to the one below:

You'll come back to this screen later.

$ kubectl apply -f queue-deployment/consumer.yaml

...
          env:
          - name: RABBITMQ_URL
            value: >
              "amqp://user:autoscaling@rabbitmq.default.svc.cluster.local:              5672"
          - name: QUEUE_NAME
            value: "autoscaling"
          - name: BATCH_SIZE
            value: "1"
          resources:
            requests:
              cpu: 300m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 128Mi
...

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queue-consumer
spec:
  scaleTargetRef:
    name: queue-consumer
  pollingInterval: 5
  cooldownPeriod:  90
  maxReplicaCount: 15
  triggers:
  - type: rabbitmq
    metadata:
      queueName: autoscaling
      mode: QueueLength
      value: "5"
    authenticationRef:
      name: rabbitmq-auth

$ kubectl apply -f queue-deployment/producer.yaml

Watch KEDA in action

Go back to RabbitMQ's UI, to watch the autoscaling queue, you can go directly to this URL: http://localhost:15672/#/queues/%2F/autoscalingYou'll see that the messages have arrived to the queue, and some of them have been processed already. Look at the following screenshot for reference:

Then, in a new tab, run this command to watch how pods are being created:

$ watch kubectl get scaledobject,hpa,pods

Notice how KEDA started creating 4 to 5 pods to start processing messages. Then, it continues to create new pods as there are still messages in the queue. Finally, when all messages are processed, around three minutes after you launched the producer, you'll see how quickly pods were removed, keeping the number of replicas to zero. You might want to increase the cooldownPeriod to see how pods are kept after the process have finished. Even though this is great for cluster's efficiency, it might not be good for your application's performance, especially if you have a long-running process. In the next section we'll explore an alternative for this. Feel free to run this test again by deploying the producer application again.

$ kubectl delete -f queue-deployment

Hands-On: Scaling Jobs

During this hands-on lab you're going to practice deploying the same application as the previous lab, but as a set of Kubernetes jobs. Even though the sample application is not a representative of a long-running job, it's still serving the purpose of using KEDA to scale a job-based workload, let the job finished when it needs, and avoid trying to guess a proper configuration to avoid disruption when KEDA scales down. This process will make even more sense when we need to protect jobs when underutilized nodes are being deleted by projects like Karpenter.

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: queue-job-consumer
  namespace: default

spec:
  jobTargetRef:
    template:
      spec:
        containers:
        ... # Same spec definition as the Deployment one
  pollingInterval: 5
  maxReplicaCount: 15
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 2

$ kubectl apply -f queue-job/consumer.yaml

$ kubectl apply -f queue-job/producer.yaml

$ kubectl port-forward svc/rabbitmq 15672:15672

$ kubectl get scaledjob,jobs

$ kubectl logs job.batch/queue-job-consumer-4cgp2

... No new messages for 30s. Processed 21 messages. Exiting.

$ kubectl delete -f queue-job
$ helm uninstall rabbitmq

Hands-on lab: Scaling to zero during non-working hours

A very common scenario to optimize costs is to turn down environments during non-working hours; this, of course, applies to any pre-production environment. So, imagine that your workload is being used only from 8:00 AM to 6:00 PM from Monday to Friday. Additionally, within that schedule, you might want to continue using a CPU utilization trigger to spin up replicas for when to keep a CPU threshold. To do so, you need to do the following:You have to use both the cron and the cpu trigger within the ScaledJob rule, it should look like this:

...
spec:
  scaleTargetRef:
    name: montecarlo-pi
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
  - type: cron
    metadata:
      timezone: US/Pacific
      start: 0 8 * * 1-5
      end: 0 18 * * 1-5
      desiredReplicas: "1"
  - type: cpu
    metricType: Utilization
    metadata:
      value: "70"

Notice that the previous ScaledObject is defining a minReplicaCount of zero. This means that outside of the schedule configured in the cron trigger, KEDA will set the target to zero replicas, which is what we wanted to achieve in the first place.To see this in action, let's deploy the Monte Carlo application and the ScaledObject rule. Go to the chapter05 folder from the GitHub repository, and run the following commands:

$ kubectl apply -f montecarlopi.yaml
$ kubectl apply -f cron

If you're running this lab during the schedule configuration, you should see one pod for the application running.

Note

You might need to adjust the schedule configuration to see it working, depending on when you're running this lab.

To confirm that the other trigger is working, let's send for a very short period of time some load to the application to make it use more CPU by running the following command:

$ kubectl apply -f ab-k8s

Then, to watch how pods are being created and how the triggers are making KEDA scale the workload to use more replicas, run the following command:

$ watch kubectl get scaledobject,hpa,pods

Notice that the cpu trigger started to work together with the cron trigger, and the replicas might grow to something around three replicas. When the load test finishes and the stabilization window from HPA has passed, KEDA will bring the replicas to 1 as per the schedule from the cron trigger.If you'd like to see how KEDA sets the replicas to zero, change the timezone configuration from the ScaledObject to a different one, like Europe/London. The idea is that you force KEDA to apply the configuration you might have outside of the schedule configured. The trigger should look like this:

- type: cron
    metadata:
      timezone: Europe/London

Apply the changes by running this command:

$ kubectl apply -f cron

Wait for the cooldownPeriod to finish, which by default is 300 seconds, and you'll see that KEDA removes the HPA rule.To clean up all the resources for this lab, run this command:

$ kubectl delete -f cron, montecarlopi.yaml, ab-k8s

In this lab, you learned how to combine a cron-based trigger with a CPU utilization trigger to create a cost-efficient autoscaling pattern as it optimizes costs by scaling to zero during non-working hours, while still being responsive to real-time CPU load during active periods.Next, let's explore a different approach to autoscaling based on incoming HTTP traffic using KEDA.

Caching metrics

The KEDA Metrics server queries the scaler with each request coming from HPA, which by default is every 15 seconds, or every time during the pollingInterval period, which by default is 30 seconds. When you have multiple ScaledObjects making queries every 15 seconds to the same scaler, the scaler might get throttled or be unresponsive due to the high demand. For this reason, KEDA has a property called useCachedMetrics, within a trigger section, to enable or disable (default) a cache for metric values from the scaler during the polling interval. This means that you could reduce to 50% (in the default scenario) the number of calls you make to the scaler, or even more if you extend the polling interval configuration. When you enable the metrics cache, every time a request to the KEDA metrics server reads the value from the cache instead of making a direct query to the external service.Let's continue using one example from previous labs and enable cache for the RabbitMQ trigger. The configuration will now look like this:

triggers:
  - type: rabbitmq
    useCachedMetrics: true
    metadata:
      queueName: autoscaling
      queueLength: "5"
    authenticationRef:
      name: rabbitmq-auth

Consider enabling cache for commonly used scalers for applications that can wait for some time to get a fresh value from the scaler. However, if your application needs to scale as soon as possible, caching the metric value might not be suitable. In this case, it's better not to use caching to ensure that KEDA retrieves the most up-to-date metrics directly from the source.

Note

You can't cache metrics for cpu, memory, cron scaler, or ScaledJobs.

Pausing autoscaling

Imagine that a KEDA scaler is not available, is not working properly, or you simply need to run a maintenance operation where any scaling action might actually cause a problem. KEDA has the ability to pause autoscaling, so instead of removing any ScaledObject resource, modifying its scaling behavior, or even making a change within the application, you simply need to add (or change) a special set of annotations in the ScaledObject resource.To pause a ScaledObject, you can add the following annotation(s):

metadata:
  annotations:
    autoscaling.keda.sh/paused-replicas: "0"
    autoscaling.keda.sh/paused: "true"

You don't need to specify both annotations; with either one or the other, you can pause autoscaling. However, in some scenarios, using both annotations together can be beneficial. This combination is useful during maintenance windows or operational tasks where you need to guarantee a stable replica count and prevent any scaling changes.Let me explain what each of these annotations is doing so that you can choose properly:

• autoscaling.keda.sh/paused-replicas will set the desired number of replicas to the configured value in this annotation and will pause autoscaling. Use this annotation if you need to override the existing number of replicas for your workload. The existing number of replicas might be too low or too high, and you'd like to set the number of replicas that can offer service continuity for your application.
• autoscaling.keda.sh/paused will pause autoscaling, but it will keep the existing number of replicas for your workload. This approach is simpler, and it might be the option that works most of the time, especially if the pausing period is short or scaling up or down is too costly (and not worth it).

To resume autoscaling, you can either remove the above annotations or simply change the autoscaling.keda.sh/paused annotation to false.

Fallback scaling actions

Imagine that you might have no idea that a scaler is not working, or something goes wrong with the scaler during non-working hours. You might have your monitoring systems configured to let you know about it, but as a safety mechanism, you might want to configure a ScaledObject to fallback to a certain number of replicas if KEDA fails to collect metrics from a target certain number of times.To configure fallback, you need to use the following section:

  fallback:
    failureThreshold: 3
    replicas: 6

In this code, the failureThreshold is the number of consecutive times KEDA fails to get metrics from the source. And replicas is the desired number of replicas you want to have if the failure threshold is met.

Note

You can't configure fallback for ScaledJobs objects, for cpu and memory scalers, or any other scaler whose metric type is a value, like the rabbitmq or cron scalers.

So far, you've learned how to handle scenarios when KEDA can't scale using tools like pausing and fallback configuration. These mechanisms ensure that your applications remain stable and available even if they can't scale.Now, let's dive deeper and look at how KEDA can support complex autoscaling logic with advanced features such as scaling modifiers and external scalers, which let you setup auto scaling rules to use very specific scaling requirements.

Complex triggers with scaling modifiers

There might be times when, even with the ability to configure multiple triggers for complex autoscaling rules for your workloads, you might need to configure custom conditions to scale. Why? Consider that when you have multiple triggers, HPA calculates how many replicas are needed for each metric and uses the maximum one. If you have three cron triggers, the maximum desired number of replicas will be used. But maybe you'd like to sum all of them, particularly in scenarios where each trigger represents an independent source of expected load. Summing them can ensure that your application scales appropriately to handle combined demand across all those dimensions.Let me give you another example. Let's say you have an application that processes messages from a RabbitMQ queue and then persists some aggregated data into a MySQL database. Scaling only based on the number of messages in the queue might, at some point, cause problems for the database. You might have no problems processing tons of messages, but you might need to be careful not to bring the database down. Therefore, you need to consider metrics like write latency or replica lag time to decide if you want to scale based on the RabbitMQ trigger or stop adding replicas until the database is stable.To solve this problem, KEDA offers a feature called scalingModifiers that can help you take autoscaling to the next level. When you opt in for this option, KEDA will use this configuration to decide how to scale. You define a formula where you can reference metrics obtained from scalers and use mathematical and/or conditional statements. This formula returns a value, a composed metric, that joins all the metrics from the scalers that support scalingModifiers (i.e., CPU and memory are ignored) into only one, and the calculated value will be the one used to scale. In other words, the new composed metric takes precedence over what each scaler would have scaled independently.Let's continue using the previous example to help you understand how it works and how to use it. If the database write latency is below 100 ms, it means the database is working fine. Therefore, you can continue using the number of messages in the queue to decide how to scale. If not, you need to pause autoscaling by matching the formula value to the target value. It's like saying to KEDA, "Hey, we're good, no need to scale up or down."Here's how the configuration will look like:

...
advanced:
  scalingModifiers:
    target: "5"
    activationTarget: "1"
    metricType: "AverageValue"
    formula: "db_write_latency < 100 ? queue : 5"
...

Let's break down each of these parameters:

• target is the new value you define to scale; it's the target the autoscaling rule needs to maintain. It will add or remove replicas to keep the target.
• formula is where you manipulate the new metric that needs to be a single value (not boolean). You can use metrics from other scalers (in this example, it's using db_write_latency and queue scalers), use mathematical operations, and use conditional statements. KEDA uses the expr expression language, you can learn more about what expressions you can use at github.com/expr-lang/expr. If fallback is configured, the formula won't modify any metric.
• activationTarget is optional, and it's the target value you define to activate scaling, in other words, to go from 0 to 1 replica. By default, it is 0.
• metricType is optional, and it's the metric type to use for the composed metric from formula. By default, it is AverageValue, but it can be Value too.

Note

You can't configure scalingModifiers for ScaledJobs objects or for cpu and memory scalers.

Let's see scaling modifiers in action with the following hands-on lab.

Hands-on lab: Pausing autoscaling when resources are constrained

In Chapter 4, you already practiced scaling a Kubernetes deployment using a RabbitMQ trigger. I'm going to continue using that lab with the additional complexity I've been discussing to help you explore scaling modifiers. Imagine the application is persisting some processed data to a MySQL database. You want to avoid bringing down the database if too many messages end up arriving in the queue, and you need to scale up accordingly. To prevent that, we'll use an external metrics API to get the database write latency in milliseconds. If the write latency is 100 ms or higher, you need to pause autoscaling and resume it when the database is stable.Start by creating a RabbitMQ queue by running this command:

$ helm install rabbitmq --set auth.username=user \
--set auth.password=autoscaling bitnami/rabbitmq –wait

Next, deploy the consumer application that processes one message from the queue at a time. We won't explore the YAML manifest as it's the same as in the previous chapter. Run this command to deploy the consumer:

$ kubectl apply -f formula/consumer.yaml

For now, the consumer will have only one replica consuming messages.To make this lab easier to configure, I've created a dummy API that returns a static number representing the database write latency. It returns a JSON like this:

{
  "database": {
    "metrics": {
      "write_latency": 150
    }
  }
}

Deploy the dummy API by running this command:

$ kubectl apply -f formula/dummyapi.yaml

Confirm that the API is working by running this command:

$ kubectl port-forward svc/dummyapi 8080:80

Then, open http://localhost:8080/ in a new browser window. Notice that the write latency is 150, which means the database is not stable and the consumer shouldn't scale up with new messages in the queue.To configure that autoscaling rule, let's deploy a ScaledObject with two triggers. One for the RabbitMQ queue, and one to query the dummy API, like this:

...
  triggers:
  - type: rabbitmq
    name: queue
    metadata:
      queueName: autoscaling
      mode: QueueLength
      value: "5"
    authenticationRef:
      name: rabbitmq-auth
  - type: metrics-api
    name: db_write_latency
    metadata:
      targetValue: "100"
      url: "http://dummyapi.default.svc.cluster.local/"
      valueLocation: 'database.metrics.write_latency'
...

Notice that each trigger now has a name setup, which is required when using scalingModifiers as you need a way to reference them. Also, we're using the metrics-api scaler, which is used to scale based on a metric provided by an API you might own, like in this lab.Then, notice how these two triggers are used in the formula field to configure the autoscaling rule we saw previously. It should look like this:

...
  advanced:
    scalingModifiers:
      target: "5"
      formula: "db_write_latency < 100 ? queue : 5"
...

Remember, the formula is saying that when the database write latency is 100 ms or higher, pause autoscaling to avoid creating new replicas. You don't need to write this autoscaling rule from scratch; to deploy it, simply run the following command:

$ kubectl apply -f formula/scalingmodifiers.yaml

Before you start testing this scenario, let's run a command to see that no new replicas are being created when you send messages to the queue, as the database isn't stable for now. Run this command in a new terminal:

$ watch kubectl get pods,scaledobject,hpa

Ignore any CrashLoopBackOff error you might have; the pod will be restarted and it will work again as we discussed in Chapter 4. Notice that HPA only has one target and not two because even if you have two triggers, KEDA created the HPA object using the one you provided in the formula field, the output should look similar to this:

NAME  REFERENCE      TARGETS   MINPODS MAXPODS REPLICAS AGE
...   queue-consumer 5/5 (avg) 1       15      1        2m

Let's send a few messages to the queue by running the following command:

$ kubectl apply -f formula/producer.yaml

You might want to see how many messages you have in the queue, so run the following command to expose the RabbitMQ UI:

$ kubectl port-forward svc/rabbitmq 15672:15672

Then open http://localhost:15672/#/queues/%2F/autoscaling in a new browser window. When prompted for login, type user as the user, and type autoscaling for the password. You should see that messages are being consumed slowly, as you only have one replica running. If the consumer pod is in CrashLoopBackOff, simply delete it and let Kubernetes recreate it.Let's simulate that the database is stable now by changing the static value configuration from the dummy API deployment. You need to change the following environment variable:

        env:
        - name: DUMMY_VALUE
          value: "150"

To make this change, run the following command:

$ kubectl set env deployment/dummyapi DUMMY_VALUE=85

And this is when scaling modifiers shine. Notice that as soon as HPA is able to get the new value from the dummy API, new pods will be created, as the database is stable now. Messages from the queue will be consumed very rapidly now.To clean up the resources created by this lab, run these commands:

$ kubectl delete -f formula
$ helm uninstall rabbitmq

In this hands-on lab, you learned how to use scaling modifiers to build conditional autoscaling logic in KEDA. By combining multiple triggers, one for message queue length and another for database write latency, you were able to pause scaling when a backend system was under pressure and resume it once the system stabilized. This is useful when building context-aware autoscaling strategies that respond not only to application demand, but also to the health of dependent systems.Now let's look at how to go beyond those limits. In the next section, we'll explore external scalers, which allow you to extend KEDA's functionality when built-in scalers aren't enough.

Extending KEDA with external scalers

Another way to extend KEDA's capabilities is by using external scalers. KEDA offers several built-in scalers, which I'd recommend you explore before creating an external scaler. However, if the scaler you need doesn't exist yet, or your organization needs to own the code because you need to interact with private APIs, then you might consider creating an external scaler. Another reason might be that the scaler's complexity requires a separate controller to create additional resources, similar to what the HTTP Add-on does when creating a resource to hold incoming requests when scaling from 0 to 1. This book won't cover how to create an external scaler as it will need go deeper into coding one and will require its own chapter; instead, we'll simply cover at a high level what you'd need to do to create one, and how to use it.KEDA's built-in scalers run within the KEDA operator, while external scalers run outside of KEDA, and you'd need to communicate with them through a remote procedure call (or gRPC, an open-source framework for communication between services originally developed by Google). To build an external scaler, the endpoint needs to implement KEDA's built-in scaler interface so KEDA knows when the scaler is active or not, and can query metrics used to define how many replicas are needed.To use an external scaler, you have either the external or external-push scaler types. The main difference is that the external-push scaler is able to connect to an endpoint that can stream KEDA requests to receive updates from the scaler. The following is how you'd use an external scaler:

  triggers:
  - type: external
    metadata:
      scalerAddress: your-internal-scaler-endpoint:8080
      organization: eCommerce
      business_unit: Payment

The scalerAddress is the endpoint of the scaler API that implements the scaler interface, and the other two fields are custom fields used to pass parameters to the scaler. KEDA sends the entire metadata object to the external scaler endpoint. There are other fields like caCert, tlsCertFile, tlsClientCert, tlsClientKey, and unsafeSsl that are used to configure TLS authentication to the scaler endpoint.External scalers allow you to extend KEDA's scaling features to your needs, but you need to consider the added maintenance for operating and using an external scaler. So, make sure you have a valid reason to do it, and you've explored the existing list of scalers first.

KEDA on Amazon EKS

To use KEDA on Amazon EKS, the recommendation is that you first give KEDA's controller the capability to assume other IAM roles, as you don't want to give too many permissions to KEDA's controller for the following reasons:

• For security purposes
• You might not want to restart KEDA's controller every time you need to modify its permissions.

So, when you install KEDA, you need to configure which IAM role, which is tied to a Kubernetes service account, the controller will use. Regardless of the identity model you're using on EKS, IAM Roles for Service Accounts (IRSA), or EKS Pod Identity, a Kubernetes service account is tied to an IAM role. We'll skip these configuration steps and how it works for now, as we'll cover them in Chapter 11.Let's say that the Kubernetes service account you'll use for KEDA's controller will be named keda-operator. Therefore, if you're installing KEDA through Helm, you need to make sure the service account is properly configured. By default, it is like this:

serviceAccount:
  operator:
    create: true
    name: keda-operator
    automountServiceAccountToken: true
    annotations: {}

This service account will be tied to an IAM role called keda-operator as well, and how you do it depends whether you use IRSA or EKS Pod Identity. If you're using EKS Pod Identity, the IAM policy should look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ],
            "Condition": {
                "StringEquals": {
                          "aws:SourceAccount": "ACCOUNT_NUMBER"
                    },
                       "ArnEquals": {
                         "aws:SourceArn": "EKS_CLUSTER_ARN"
                   }
                 }
           }
    ]
}

This IAM policy would be the one that allows KEDA to assume other IAM roles, which in this case will be the ones assigned for workloads (pods) that you want to scale using KEDA. We're recommending this approach as the workload you want to scale is already interacting with an AWS service. Therefore, it already has the permissions to interact with the AWS API. For instance, if the workload is consuming messages from an AWS SQS queue, it should have permissions to at least read messages, delete messages, and describe the queue to get its queue length, something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ConsumeSQSQueue",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl"
            ],
            "Resource": "arn:aws:sqs:${AWS_REGION}:${ACCOUNT_ID}:${QUEUE_NAME}"
        }
    ]
}

Once KEDA's controller is able to assume other IAM roles, and the workloads are able to interact with the AWS APIs, the next step you need to consider is the authentication configuration for the ScaledObject.We recommend that you use the authenticationRef property to delegate this configuration to the TriggerAuthentication object, which should have the following manifest:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: aws-credentials
spec:
  podIdentity:
    provider: aws
    identityOwner: workload

Notice that you're saying it should use AWS as the provider and that it should take the identity (identityOwner) from the workload. KEDA's controller will assume the IAM role that the workload is using to collect the metrics for the AWS scaler configured in the triggers section from the ScaledObject, like this:

...
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: aws-credentials
    metadata:
      queueURL: autoscaling
      queueLength: "5"
      awsRegion: "eu-west-1"

The advantage of using a TriggerAuthentication to configure the AWS credentials is that you can reuse it within other ScaledObjects. Moreover, because we're recommending you to configure authentication using the workload's service account, it will simplify the permissions management for you as KEDA will assume the workload's IAM role, allowing you to configure only the permissions a scaler will need instead of giving KEDA's controller superpowers.

Note

At the time of writing this book, KEDA has five built-in scalers for AWS: AWS CloudWatch, AWS DynamoDB, AWS DynamoDB Streams, AWS Kinesis Stream, and AWS SQS Queue. And regarding authentication providers, KEDA has support for: AWS IRSA, and AWS Secret Manager to use secrets within KEDA's CRDs. AWS has an alternative, and recommended approach called EKS Pod Identity, which we'll explore deeper in Chapter 11.

KEDA on Azure Kubernetes Service

To use KEDA on Azure Kubernetes Service (AKS), you can simply enable the KEDA add-on that installs all the KEDA components integrated with AKS. You can enable it through an Azure Resource Manager (ARM) template by adding the following section:

"workloadAutoScalerProfile": {
        "keda": {
            "enabled": true
        }
    }

Or, you can also use the Azure CLI and run the following command:

$ az aks update --resource-group RESOURCE_GROUP_NAME \
--name AKS_CLUSTER_NAME --enable-keda

Notice that you basically add the --enable-keda parameter.

Note

As there are KEDA external scalers for AKS, like the one for Cosmos DB, these will need to be installed separately, as the KEDA add-on only installs KEDA's built-in scalers.

Once KEDA is enabled, you first need to enable the AKS cluster to interact with the Azure APIs that your workloads will need to scale. For this book, we go in depth with the commands you need to run. We'll only cover at a glance what the process looks like.To continue with KEDA's configuration, the next step is to create an Azure Identity for the workloads and the KEDA operator. Then, to avoid having to specify secrets, you need to create a federated credential between the Azure Identity and the Kubernetes service account that the workload you'll scale is going to use. You also need to create a second federated credential to give permissions to KEDA's controller.Then, depending on the Azure scaler you'll use, you need to provide the corresponding permissions to the Azure Identity you created. For instance, if you're using the Azure Service Bus scaler, you need to give permissions to this API to the Azure Identity.Now you need to create a TriggerAuthentication object to allow scalers to use the Azure APIs. The manifest should look like this:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-credentials
spec:
  podIdentity:
    provider:  azure-workload
    identityId: $MI_CLIENT_ID

Notice that the provider is an Azure workload, and that the identityId is the Azure Identity ID you created before. With this configuration, KEDA will be able to use the workload identity to collect metrics from the scaler configured to scale the workload.

Note

At the time of writing this book, KEDA has nine built-in scalers for Azure: Azure Monitor, Azure Pipelines, Azure Event Hubs, Azure Storage Queue, Azure Application Insights, Azure Data Explorer, Azure Log Analytics, Azure Service Bus, and Azure Blob Storage. Additionally, there are two external scalers: KEDA External Scale for Azure Cosmos DB and Durable Task KEDA External Scaler. And regarding authentication providers, KEDA has support for Azure AD Workload Identity and Azure Key Vault Secret.

KEDA on Google Kubernetes Engine

Using KEDA on Google Kubernetes Engine (GKE) is pretty similar to what you've seen so far. We won't go in depth on how you'd configure the security aspects of using KEDA on GKE, but we'll explore at a glance what the process looks like.First, you need to deploy KEDA's components to GKE, and the Kubernetes service account that KEDA's controller will use is going to be tied to a Google Service Account (GSA) that will have the proper permissions to use the API of Google Cloud Platform (GCP). Then, depending on the scalers you'll use to scale your workloads, you need to provide the corresponding permissions to the GSA account.Similarly, as with the other cloud providers, the recommended approach, if your workloads are running in GKE, is to use the TriggerAuthentication object, which for GCP, the manifest will look like this:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: gcp-credentials
spec:
  podIdentity:
    provider: gcp

Notice that here you simply configure GCP as the provider. When your ScaledObject references this authentication object, KEDA's controller will impersonate the IAM service account the workloads you're scaling have.

Note

At the time of writing this book, KEDA has four built-in scalers for GCP: Google Cloud Platform Pub/Sub, Google Cloud Platform Storage, Google Cloud Platform Cloud Tasks, and Google Cloud Operations. And regarding authentication providers, KEDA has support for GCP Secret Manager and GCP Workload Identity.

As you can see, the approach is the same in all three cloud providers, but the difference remains in the service names each provider uses and how they implement such security configurations. The reason we dedicated a section to explore how KEDA works on cloud providers is to show you how KEDA adapts very well to what each of these cloud providers needs security-wise. Avoid giving extra permissions to KEDA's controller, and instead use the permissions each of the workloads that you're scaling have.

Troubleshooting HPA

One of the most common problems in HPA relies on the process of retrieving metrics. You might have some misconfigurations in place, or the metrics provider might not be working. The most basic problem could be:

• The Metrics Server component is not installed in the cluster or isn't working properly. Therefore, HPA won't be able to get the basic metrics like CPU and memory from the pod(s).
• Custom or External metrics aren't available. For instance, you might have configured a rule to scale using metrics from Prometheus, but there's no adapter translating those metrics, or the query you've configured has an error.
• Networking issues between the cluster and the metric sources.
• Role-based access control (RBAC) permissions to interact with the Metrics API.

The solution for all of the problems listed here might be obvious (i.e., deploy the metrics server, restart the metrics server, fix the scaling rule configuration, give proper RBAC permissions, etc.). But these might not be the only problems you could face. So instead of giving you an extensive list of problems with their possible solution(s), let's explore a few commands that can help you identify and confirm what could be the problem with HPA scaling rules.

$ kubectl describe hpa montecarlo-pi

Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable  to get the target's current scale: deployments/scale.apps 
  "montecarlo-pi" not found

Events:
  Type     Reason          Age  From                       Message
  ----     ------          ---  ----                       -------
  Warning  FailedGetScale  5s    horizontal-pod-autoscaler  deployments/scale.apps "montecarlo-pi" not found

$ kubectl logs -n kube-system -l app.kubernetes.io/name=metrics-server \
--all-containers=true --tail=20

Troubleshooting VPA

As you already know, VPA doesn't come as a built-in feature in Kubernetes. To use VPA, you need to deploy its three components:

• VPA updater
• VPA recommender
• VPA admission controller

Most of the troubleshooting will be done within the controller and the recommender.So, what do you need to do if VPA is not doing what you're expecting?

1. Check that the VPA components are up and running
2. Confirm that the VPA's CRDs were deployed
3. Validate that the Metrics Server component is up and running
4. Review the events from the VPA objects
5. Read the logs from VPA's components to find out what's wrong
6. Verify why VPA can't apply recommendations
7. The first three bullet points are pretty straightforward, and you already know how to do this after doing the labs from Chapter 3. Therefore, let's focus on the rest of the bullet points, as some need you to understand what to look for and which commands to run.

$ kubectl describe vpa montecarlo-pi

...
Status:
  Conditions:
    Last Transition Time:  2024-11-27T22:03:02Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  montecarlo-pi
...

$ kubectl logs -n kube-system -l app=vpa-recommender \
--all-containers=true --tail=20

$ kubectl logs -n kube-system -l app=vpa-updater \
--all-containers=true --tail=20

$ kubectl logs -n kube-system -l app=vpa- admission-controller \
--all-containers=true --tail=20

Troubleshooting KEDA

Similar to VPA, KEDA doesn't come as a built-in feature in Kubernetes. Therefore, when you deploy KEDA to your cluster, you deploy a few pods (components) you'll use to troubleshoot when scaling isn't working. Scaling might not be working mainly because KEDA can't collect metrics from the source, but there might be other reasons we'll explore in a moment.As a quick reminder, as you learned in Chapter 4, there are three KEDA components:

• KEDA's operator
• KEDA's metric server
• KEDA's admission webhook.

You'll likely spend more time reading the KEDA's operator logs, as that's like the KEDA's brain.So, what do you need to do if KEDA is not doing what you're expecting?

1. Check that the KEDA components are up and running
2. Confirm that KEDA's CRDs were deployed
3. Validate that the Metrics Server is working if using CPU and memory scalers
4. Validate that you have properly authenticated to the scaler API
5. Confirm that there aren't any network policies denying traffic to KEDA
6. Review the events from ScaledObject or ScaledJob objects
7. Review the events from the HPA objects (if scaling is active)
8. Read the logs from KEDA's components to find out what's wrong
9. Let's see how you can check events and review logs from KEDA's components.

$ kubectl apply -f troubleshooting/scaledobject.yaml

$ kubectl describe scaledobject queue-consumer

...
  Warning  ScaledObjectCheckFailed  1s (x12 over 11s)  keda-operator  
  Target resource doesn't exist
  Warning  ScaledObjectCheckFailed  1s (x12 over 11s)  keda-operator  
  ScaledObject doesn't have correct scaleTargetRef specification

$ kubectl logs -n keda -l app=keda-operator --all-containers=true \
--tail=20

$ kubectl logs -n keda -l app=keda-operator-metrics-apiserver \
--all-containers=true --tail=20

$ kubectl logs -n keda -l app=keda-admission-webhooks \
--all-containers=true --tail=20

...ERROR Reconciler error{"controller": "scaledobject", \
"controllerGroup": "keda.sh", "controllerKind": "ScaledObject", \
"ScaledObject": {"name":"queue-consumer","namespace":"default"}, \
"namespace": "default", "name": "queue-consumer", \
"reconcileID": "ee2eb037-62eb-4386-8322-ae5378400ca0", \
"error": "deployments.apps \"queue-consumer\" not found"}

Hands-on lab: Deploying KEDA's Grafana dashboard

Before you deploy the Grafana dashboard, you need to enable KEDA's component to expose the metrics we explored before at the /metrics endpoint. By default, the Prometheus integration is disabled, and as we've deployed KEDA using Helm, let's update its configuration using Helm as well.Open a new terminal, and change the directory to the chapter06 folder from the GitHub repository. You'll find a monitoring/keda-values.yaml file with the additional settings you need to enable metrics exposure. The new Helm settings should look similar to this:

prometheus:
  operator:
    enabled: true
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus
... // Same for metricServer and webhooks

To update KEDA's Helm release, run the following command:

$ helm upgrade keda kedacore/keda --namespace keda -f \
monitoring/keda-values.yaml

KEDA's pods should be recreated with the new settings, and you should also see that a new ServiceMonitor has been created to let Prometheus know that it should scrape the /metrics endpoint for each KEDA's component.You can confirm this by running the following command:

$ kubectl get servicemonitor -n keda

You should see three Prometheus service monitors, one for each KEDA component.Now let's deploy one of the labs you did in Chapter 4 to generate some metrics, so that when you deploy KEDA's dashboard, you can see something. In your terminal, don't change directories, and simply run the following command to deploy the application and all its dependencies to configure autoscaling:

$ kubectl apply -f ../chapter04/queue-deployment/consumer.yaml

Then, run the following command to generate a small load test:

$ kubectl apply -f ../chapter04/queue-deployment/producer.yaml

While you wait for the load test to finish, let's deploy the Grafana dashboard. To do so, open the Grafana UI by exposing the service locally by running the following command:

$ kubectl port-forward service/prometheus-grafana 3000:80 -n monitoring

In a browser, open the http://localhost:3000/ URL.For the user, type user, and for password, type prom-operator. If these credentials don't work, go back to Chapter 2 to retrieve the username and password from Prometheus stack secrets.Next, you import the Grafana dashboard, and to do so follow these steps:

1. Click Dashboards from the left menu, or open the /dashboards endpoint
2. Click on the New button, and select Import in the drop-down menu
3. Click on the Upload a dashboard JSON file section, and pick the file located at chapter04/monitoring/keda-dashboard.json
4. Click on the Load button.

You should now see the KEDA's dashboard, like the one in Figure 6.1 as follows:

The first part is the list of filters for the dashboard. You can filter by namespace, scaledObject, scaler, or metric. If you don't see any data, try switching the datasource field.Notice that the first three visualizations show the error rates from the scaler(s) and scaled objects. This is a good source to know when things aren't going well in KEDA. Next, you'll see the current scaler metric values to understand which values KEDA is using to scale. Moreover, you'll see a graph showing the maximum number of replicas and how KEDA has been adding and removing replicas while the load test was running. In this case, I ran the load test twice, that is why you see two changes in Figure 6.2 :

If you continue scrolling down, you will see other visualizations to understand changes in the replica numbers for the workloads you're monitoring. Additionally, you'll see a graph that represents how close you're getting to the maximum number of replicas.The version of the dashboard we showed in this book might differ from what you get when doing this lab. The reason is simple: KEDA occasionally updates the dashboard, and we may have a different version of the JSON manifest at the time of writing this book. If you'd like to get the latest version of this dashboard, make sure you check KEDA's official repository.In this lab, you set up KEDA's Grafana dashboard to visualize key metrics and error indicators across scalers and scaling objects. This provide insights into how your workloads are behaving and helps you quickly detect issues when autoscaling isn't working as expected.Now, let's look at how to keep your installation up to date. In the next section, we'll walk through different strategies for upgrading KEDA.

Cluster Autoscaler

Cluster Autoscaler (CAS) was one of the first projects to address the challenge of automatically scaling the data plane of a Kubernetes cluster, and for a long time, it was the only one. The primary function of CAS is to ensure that there are always sufficient nodes available in the cluster to run all scheduled pods. When there are pods that cannot be scheduled due to insufficient cluster capacity, CAS will automatically provision new nodes to accommodate these unscheduled pods. Conversely, when nodes are empty, CAS removes these unnecessary nodes for cluster efficiency.In cloud environments, CAS integrates with cloud provider APIs to manage node groups or auto-scaling groups. For example, in AWS, it can work with EC2 Auto Scaling Groups to add or remove instances as needed. In GCP, it can manage node pools, and in Azure, it works with Virtual Machine Scale Sets. This integration allows CAS to leverage cloud-native scaling capabilities, making your Kubernetes cluster efficient and cost-effective in cloud deployments.For on-premises environments, CAS can also be configured to work with various infrastructure provisioning tools. However, the process is often more complex and may require additional setup and integration work. On-premises setups might involve integrating CAS with tools like OpenStack, VMware vSphere, or bare-metal provisioning systems to manage the lifecycle of physical or virtual machines.CAS works with various cloud providers and on-premises setups, making it a valuable tool for managing Kubernetes cluster capacity across different environments, though the ease of implementation can vary significantly between cloud and on-premises deployments.

Karpenter

For a very long time, since April 2017, CAS was the only project available for autoscaling the data plane in Kubernetes. However, the landscape changed with the introduction of Karpenter, a Kubernetes cluster autoscaler developed by AWS as an open-source project. Karpenter was announced as ready for production in November 2021, and its v1 version was released in August 2024.Karpenter takes a dynamic and fine-grained approach to node provisioning and manages nodes directly without any abstraction layer like Auto Scaling groups in AWS. Like CAS, Karpenter provisions new nodes in response to unschedulable pods, and it attempts to optimize node selection to maximize resource utilization based on pods' resource needs, considering scheduling requirements like affinities or tolerations. When a node is not needed, Karpenter can remove it or replace it with a smaller one, this process is also known as consolidation. We'll explore all the details about how this feature works in the next chapter.Additionally, it's worth mentioning that Karpenter works in tandem with the kube-scheduler. This means that Karpenter won't schedule pods in the cluster; it will only provision nodes. It's very important that you keep this in mind to make the most of Karpenter. When we look at the implementation details in the next chapter, this will be clearer, and you'll understand why I'm emphasizing this now.Karpenter uses Kubernetes custom resources for configuration, making it easier to manage and integrate with existing Kubernetes workflows. While initially designed for AWS, Karpenter's design principles aim to make it adaptable to other Kubernetes environments in the future. At the time of writing this book, besides the EKS provider, Microsoft has released the Karpenter provider for Azure Kubernetes Service (AKS). In the next chapter, we'll dive deeper into how the Karpenter project is structured.In the meantime, let's explore how Cluster Autoscaler works in AWS.

Hands-on lab: Cluster Autoscaler on AWS

Note

This hands-on lab is going to use Terraform to create or update (if you created it already in Chapter 1) an EKS cluster. We recommend that you use this method as it will help you to get started quickly and focus on learning about how CAS works.

The setup we've been using so far for the Kubernetes cluster has been static. This means that if you ended up deploying too many pods in the cluster, you'd have seen that some pods were unscheduled, as there was no capacity in the cluster. If you've been using the EKS cluster from Chapter 1, you've been working with a managed node group of only two nodes. During this lab, you'll deploy around 60 pods to force CAS to react when there are unscheduled pods and see how new nodes are being added automatically.

...
  enable_cluster_autoscaler = true
...

$ sh bootstrap.sh

$ kubectl get pods -n kube-system -l app.kubernetes.io/instance=cluster-autoscaler

$ kubectl delete vpa --all

...
        resources:
          requests:
            cpu: 900m
            memory: 512Mi
...

$ kubectl apply -f montecarlopi.yaml

$ kubectl describe pods | grep -i requests -A 2

$ kubectl scale deployment montecarlo-pi --replicas=12

$ kubectl get pods --field-selector=status.phase=Pending

$ kubectl get nodes

$ kubectl delete -f montecarlopi.yaml

$ kubectl logs -n kube-system -l app.kubernetes.io/instance=cluster-autoscaler --all-containers=true -f --tail=20

...
  enable_cluster_autoscaler = false
...

$ sh bootstrap.sh

$ kubectl get all -n kube-system | grep autoscaler

Cluster Autoscaler Best Practices

Even though this book doesn't go in depth about CAS, you might be in a transition phase to Karpenter, and it would be wise to know about the minimum set of configurations and considerations you need to implement when using CAS for EKS and configure efficient Kubernetes clusters. Moreover, it will help you understand why some implementation details in Karpenter are different and why the following recommendations will impact your efficiency score.

apiVersion: v1
kind: ConfigMap
metadata:
  name: spot-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*-spot-large
    20:
      - .*-spot-xlarge
    30:
      - .*-spot-2xlarge

Descheduler

While the initial scheduling of pods is handled by the Kubernetes scheduler, over time, the distribution of workloads can become uneven. This is where the Descheduler comes into play. The Descheduler is an open-source project that aims to improve cluster resource utilization by moving pods from over-utilized nodes to under-utilized ones. It runs as a Kubernetes add-on and uses various informers to monitor cluster resources and make decisions based on predefined policies.Imagine a scenario where some nodes in your cluster are running at 90% capacity while others are barely breaking 30%. The Descheduler can identify this imbalance and evict pods from the highly utilized nodes, allowing them to be rescheduled on the under-utilized nodes. This rebalancing act not only improves overall cluster performance but can also lead to cost savings by optimizing resource usage. To use the Descheduler, you'll need to deploy its controller in your cluster and configure a Policy ConfigMap with your desired strategies. Common strategies include LowNodeUtilization and RemoveDuplicates. Once set up, the Descheduler will automatically analyze and rebalance your cluster based on these policies.When using the Descheduler, it's important to adopt a cautious approach to minimize potential risks. For instance, it's recommended to use Pod Disruption Budgets (PDBs) to ensure a minimum number of pods remain available during descheduling operations. Moreover, carefully configure the Descheduler's strategies to match your cluster's specific needs, and consider using the simulation mode to test configurations without affecting live workloads. Implement a gradual rollout, starting with a small subset of your cluster and expanding as you gain confidence. Leverage node affinity and anti-affinity rules to guide pod placement and prevent critical workloads from being disrupted. At this point in the book, you should have guessed already that proper resource requests and limits for pods will help the Descheduler make more informed decisions.

Cluster Proportional Autoscaler

CPA is an open-source project that scales workloads based on the size of the cluster itself, and doesn't rely on metrics such as CPU or memory usage. The primary purpose of CPA is to maintain an appropriate number of replicas for cluster-wide services as your cluster grows or shrinks. This makes it particularly useful for managing system components like cluster-dns or monitoring services, which typically need to scale in proportion to the overall cluster size. CPA is currently in beta, but I've seen many customers using it to scale workloads without problems.CPA is a controller that continuously monitors the number of schedulable nodes, nodes in the cluster that have available capacity to run new pods, and CPU cores available in your cluster. Based on this information, it adjusts the number of replicas for the target resource using either linear or ladder scaling methods. To use CPA, you'll need to deploy it as a separate controller in your cluster and configure a ConfigMap with your desired scaling parameters. You'll then set up the target workload (such as a Deployment) to be managed by CPA.It's important to note that CPA may not be the best choice for application-specific workloads that need to scale based on their own unique metrics or usage patterns.

Cluster Proportional Vertical Autoscaler

While CPA focuses on horizontal scaling, CPVA addresses the vertical scaling needs of cluster-wide services. CPVA's primary function is to automatically adjust resource requests for workloads based on the size of your cluster. This makes it a nice fit for system components or workloads that need to scale vertically as your cluster grows or shrinks. It's worth noticing that CPVA is also in beta.Similar to CPA, CPVA continuously monitors the number of schedulable nodes and cores in your cluster, using this information to adjust the CPU and memory requests for target workloads. CPVA supports linear scaling and allows you to set configurable minimum and maximum resource limits. To use CPVA in your cluster, you need to deploy the controller and configure a ConfigMap with your desired scaling parameters. You then target the workloads you want CPVA to manage, which can include Deployments, DaemonSets, or ReplicaSets.Before I finish with the relevant autoscalers section, it's important to remember that our aim wasn't to provide an exhaustive deep dive into each tool. Rather, we sought to broaden your awareness of the autoscaling tools related to the Kubernetes data-plane. The Descheduler, CPA, and CPVA each offer unique approaches to optimizing cluster resources, and knowing when and how to leverage them can significantly enhance your cluster's efficiency. However, the key takeaway is not about the tools themselves, but about the overarching goal they serve. Regardless of which autoscalers you choose to implement, the ultimate objective remains the same: to create and maintain an efficient Kubernetes cluster that optimally utilizes resources, scales smoothly with demand, and provides a robust platform for your applications.