Remediating at Any Phase

Although we posit three phases, at any phase, your goal is always to remediate problems. If a single alert is firing off and you can remediate the issue by using only visibility (phase 1), you should do so. You don’t have to triage or do a root cause analysis every time if these are unnecessary.

To illustrate this point, let’s say a scheduled deployment breaks your production environment. There is no need to triage or do root cause analysis here, since you already know that the deployment caused the breakage. Simply rolling back the deployment when errors become visible remediates the issue.

The Three Phases Illustrated

In real life, if your system is crashing, you don’t focus on the data. You focus on fixing the problem immediately. No one does root cause analysis without fixing the current issue and mitigating customers’ pain.

Take, for example, a burning house (Figure 1-3).

Figure 1-3. Remediating a burning house: you should put out the fire before you start investigating the cause

If your house is on fire, how do you know? Most likely, your smoke alarm goes off, emitting a loud, unmistakable noise that notifies you of the problem. That smoke alarm is the alert, triggered by sensors detecting smoke in the room. Metrics can tell you what the issue is and give you enough information to address it. This is surface-level detection but enough to continue investigating. That’s phase 1. Metrics should give you a low mean time to detect (MTTD), so a sensitive fire alarm that goes off at the first sign of smoke or heat will be better than one that lets the fire spread for several minutes before notifying you—and better still than no alarm at all.

What now? You might jump out of bed and look around the house to see where the fire is, then get everyone out immediately and call emergency services. That’s a temporary remediation (phase 2): you’re all safe, but the house is still on fire. It’s also triaging: you are choosing to prioritize safety over other things, like saving your favorite electronics.

The sooner you can call emergency services, the faster they will arrive to put the fire out. In observability, we call this interval mean time to remediate (MTTR). This, too, should be as low as possible: if the firefighters arrive quickly and start hosing down the house right away, part of the house could be saved. If anyone is injured, you’ll want the paramedics to arrive quickly to help them: that is, you want a low MTTR.

The next morning, with everyone safe and the last embers extinguished, the fire marshal and insurance investigator examine the house to see what started the fire (a root cause analysis, phase 3). Perhaps they learn that a faulty cord on an appliance overheated. The appliance manufacturer might even recall the product to ensure that the faulty cords don’t start any more fires, a still more permanent remediation that keeps even more people safe.

No one does an investigation during an active fire as there is still a threat of injury. Similarly, the worst time to do a deep dive on how exactly a system misbehaves is when there is an ongoing outage.

You do phases 1 and 2 immediately, before you try to figure out where the fire started or why. You focus on the outcome of keeping everyone safe. One way to fulfill that is to use metrics as your starting point. In this case, since smoke in the room is the metric, then once you smell smoke, you automatically evacuate the house.

Metrics as a Starting Point

Sridharan defines a metric as “a numeric representation of data measured over intervals of time,” adding, “Metrics can harness the power of mathematical modeling and prediction to derive knowledge of the behavior of a system over intervals of time in the present and future.”¹ Figure 2-1 shows an example of measuring HTTP requests as a metric.

Figure 2-1. An example metric

The Case for Metrics

If solving your problem requires a deep dive, you might need all three signals. Logs will tell you what happened in a specific period of time. Traces allow you to track a request from beginning to end.

However, when you are starting your investigation, you need a bird’s-eye view. Starting with metrics is logical because it lets you move from the broadest view down to the narrowest. Metrics also can provide that perspective with what we call a low-latency impact analysis, which provides an efficient view of the system’s current state. What’s more, metrics are easy to implement and use, and they let you aggregate data quickly and compare it over time. Let’s look at each of these factors in turn.

Metrics Provide an Efficient Snapshot of the System

First, metrics allow you to understand the current state of the system efficiently. If you have some contextual knowledge, you can ask questions and prove or disprove assumptions so that you can start troubleshooting right away: Is there an HTTP:503 Service Unavailable error? When did it happen? Combined with context, metrics will get you those answers quickly—you don’t need to do a JSON filter or a full-text search for all HTTP response codes.

One of the most popular metrics and monitoring platforms is Prometheus, which collects metrics by scraping metrics HTTP endpoints on monitored targets. (We’ll look more closely at Prometheus in Chapter 3.)

Let’s say you have an HTTP server that’s producing an HTTP:503 Service Unavailable error. We’ll show you how you can detect that quickly by graphing a metric in Prometheus.

In Figure 2-2, you can see that the HTTP:503 Service Unavailable errors started at 17:38 GMT, without even looking at the logs. However, it is not very simple to see if there is a new HTTP:503 Service Unavailable. If you want to alert on it, usually you will need to aggregate counters rate() or irate().

Figure 2-2. Illustration of a Prometheus 503 error

Here, the metric myapp_request_count_total is a Prometheus metrics type counter. There are four types of metrics in Prometheus:

Counter

Counters are cumulative metrics that can only increase. In the preceding example, the myapp_request_count_total is a counter, and it either increases or does not move. As the name implies, counter metrics are best used for counting things like HTTP requests, RPC calls, or even business metrics like number of sales.

Gauges

Gauges are singular metrics that can either increase or decrease. Examples of metrics measured in gauges include temperature, speed, or memory usage. Gauges can also be used for metrics that decrease, such as concurrent HTTP requests.

Histogram and summary

Histograms and summaries are sample observation metrics. Examples of metrics expressed in histograms and summaries include request duration, request sizes, or ranking. Histograms and summaries are similar, but it is advisable to use a histogram for aggregation, since you can use quantiles. For more information about the difference between histograms and summaries, visit the Prometheus histogram documentation.

For more information about the types of Prometheus metrics, refer to the Prometheus documentation.

sum(metric_name)
max(metric_name)
count(metric_name) 

sum(rate(myapp_request_count_total[1m]))
max(rate(myapp_request_count_total[1m]))
rate(myapp_request_count_total[1m])
increase(business_sales[1h])
rate(business_question_set_completed[1h])

  rate(http_request_duration_seconds_sum[5m])
  rate(http_request_duration_seconds_count[5m])

sum by (status) (metric_name), 
sum without (status) (metric_name), 

max(rate(myapp_request_count_total{status="200"}[1m]))

rate(myapp_request_count_total{status="503"}[1m]) > 0 

Metric Data Is Growing in Scale

The massive shift of systems around the world from monoliths to the cloud has resulted in an ongoing explosion of metric data in terms of both volume and cardinality. This is especially prevalent in cloud native systems. To achieve good observability in a cloud native system, you will have to deal with large-scale data.

We describe this explosion of data as a metric scale, or the number of metrics produced during instrumentation. How many things are you measuring, and how much data does that measurement produce?

Say you’re working with MyApp (a fictional app). You start by using a metric called myapp_request_count_total, which counts the number of HTTP requests MyApp receives:

myapp_request_count_total{endpoint="/test"} 226

Now you can add more metric data, like all created requests, all requests, and all bounced requests:

myapp_request_count_total{endpoint="/test"} 226
myapp_request_count_created{endpoint="/test"} 163
myapp_request_bounce_total{endpoint="/test"} 440

You can see how fast the volume of data can increase when you add more metrics. The volume of logs and traces doesn’t change very much, as a rule, but metrics are a different story.

Understanding Cardinality and Dimensionality

Each increase in cardinality multiplicatively increases the volume of metrics, which requires more system resources for Prometheus. High cardinality therefore degrades Prometheus’s performance. As you gather more dimensions for each metric, you add observability context—at the cost of Prometheus performance.

Before we go any further, let’s take a quick detour to discuss these two important concepts that you’ll need to understand: cardinality and dimensionality. Cardinality is the number of possible groupings depending on the dimensions the metrics have. Dimensions are the different properties of your data, as Rob Skillington explains.³ Think of the labels on a shirt on a store shelf. Each label (in this simplified example) contains three dimensions: color, size, and type.

Each dimension increases the amount of information we have about that shirt. You could slice that information into many shapes, based on how many dimensions you use to sort it. For example, you could look by just color and size, size and type, or all three. Dimensionality means being able to slice the metrics into multiple shapes.

Increased dimensionality can greatly increase cardinality. Cardinality, in this example, would be the total number of possible labels you could get by combining those dimensions from the shirts in your inventory. Figure 2-3 visualizes this example by looking at two dimensions: color and size.

Figure 2-3. Shirt inventory with two cardinalities

There are only two cardinalities represented in Figure 2-3, even though there are three possibilities. That’s because the last combination, while theoretically possible, is not in the inventory and therefore is not emitted to the metrics platform.

The term metric cardinality refers to how many unique combinations of metric data “are produced by a combination of metric names and their associated label [dimension] names/values,”⁴ while the total number of combinations with data that exists are cardinalities.

Now let’s look at what happens if we increase the number of dimensions, and therefore of possible cardinalities, by adding the type of shirt. Figure 2-4 shows how increasing the number of dimensions also increases the cardinality of the metrics.

Figure 2-4. Shirt inventory with three cardinalities

Let’s take another example, this time from the software world. Say we want to count the total number of HTTP requests in an API. We use the metric api_http_requests_total with two dimensions, method and handler. Working again in Prometheus, we could run:

api_http_requests_total{method="POST", handler="/messages"} 60

The total number of requests, in this case, is 60.

If we decide to track the HTTP status code as well, this would add a third dimension:

api_http_requests_total{method="POST", handler="/messages", \ 
    status="200"} 30
api_http_requests_total{method="POST", handler="/messages", \ 
    status="503"} 30

As Bastos and Araujo note, “Cardinality is multiplicative—each additional dimension will increase the number of produced time series [metric data] by repeating the existing dimensions for each value of the new one.”⁵

job_data_processed{database="bronze", pod="pod1"} 1000

job_data_processed{database="bronze", pod="pod1"} 10
job_data_processed{database="bronze", pod="pod2"} 10
...
job_data_processed{database="bronze", pod="pod100"} 10

job_data_processed{database="bronze", pod="pod1", \ 
  type="json"} 5
job_data_processed{database="bronze", pod="pod2", \ 
  type="json"} 5
...
job_data_processed{database="bronze", pod="pod100", \ 
  type="json"} 
job_data_processed{database="bronze", pod="pod1", \ 
  type="csv"} 5
job_data_processed{database="bronze", pod="pod2", \ 
  type="csv"} 5
...
job_data_processed{database="bronze", pod="pod100", \ 
  type="csv"} 5

api_http_requests_total{method="POST", handler="/messages" \ 
  pod="pod2"} 600

api_http_requests_total{method="POST", handler="/messages" \ 
  pod="pod2" from="frontendservice"} 300
api_http_requests_total{method="POST", handler="/messages" \
  pod="pod2" from="backendservice"} 300

The Risk of Losing Focus on Outcomes

As you’ve seen, metric data can grow very quickly. Even just instrumenting metrics creates a high cognitive load for any site reliability engineering (SRE) team. Because of this, paradoxically, it’s not uncommon for SRE teams to lose sight of why they’re instrumenting metrics in the first place. SRE teams’ performance is often graded on how “well instrumented” systems are, which then leads to increasing the number of data points they instrument. Further, this constricts their focus on the data being collected rather than on the outcomes. They might instrument too much without knowing what to do with all that data.

This approach tends to involve asking questions like:

• Do you instrument your code?

• Which data do you collect?

• How many ways can you slice and dice your data?

• Are there dashboards to visualize this data?

Worse yet, they may even instrument the wrong data and present it to the dashboard and the alerting system.

You don’t have to instrument literally everything, just the metrics that matter in your organization and that allow you to focus on outcomes.

The loss of focus is even more apparent if we improve the questions:

• Do you get alerted appropriately when there is an issue?

• Does the alert give you a good place to start your investigation?

• Are the alerts too noisy?

• How do you visualize the data you collect?

• Do you even use it at all during incidents?

• Can you use the dimensions of the metrics to help triage and scope the impact of the issue?

• Are the alerts useful and helpful before and after incidents?

Again, we recommend always focusing on outcomes. To achieve this focus, follow the three phases of observability to refine your processes and tools iteratively, and be vigilant in measuring your MTTR and MTTD.

Metrics are an efficient and powerful tool for all three phases of observability. But how do we solve the problem of scale and data growth?

User-Controlled Metric Data Collection

To solve the performance issues, the SRE team at ecommerce site Etsy wrote a new type of agent, which they called StatsD. StatsD, writes Etsy engineer Ian Malpass, is a simple open source agent that “listens for messages on a UDP [User Datagram Protocol] port.”¹ It improves the performance of agent-based monitoring by sending messages over a “fire-and-forget” protocol, which minimizes system resource use and does not wait for a response from the aggregation server, ultimately improving performance.

As StatsD grew in popularity, the financial services company Stripe adopted it. Once they began using it, however, Stripe found two flaws in StatsD: StatsD uses UDP, which is inherently unreliable and does not work with metric types that rely on timer.

Stripe’s engineers decided to start an open source project of their own: a metrics sink called Veneur. Metrics sinks send metric data to vendors or aggregators in various formats, as shown in Figure 3-2.

Figure 3-2. A metrics sink sending data to three different vendors

The problem with metrics sinks, however, is that every vendor needs a unique sink implementation. Vendors want their clients to use their native agent, so they have little incentive to write a sink implementation. Sinks make it all too easy for clients to migrate out of the vendor’s product and toward the competition. Indeed, as of late 2021, Dynatrace and AppDynamics still do not have a sink implementation in Veneur.

In 2012, while most organizations were making the switch to microservices architecture, Soundcloud ran into a set of similar challenges while scaling their existing monitoring system. To solve these challenges, SoundCloud created Prometheus: a way to instrument once and output everywhere.

Prometheus is inspired by Google’s Borgmon monitoring system. Instead of using a sink that pushes data to an aggregator system, Prometheus instrumentation exposes a metric endpoint (usually an HTTP endpoint in /metrics). The Prometheus server scrapes the metric endpoint. While most other systems are push-based, pushing data out toward an aggregator, Prometheus is pull-based. This represents a major innovation: because push-based systems must wait for servers to respond to requests, they can cause delays and performance degradation. Pull-based systems expose data by using a broadcast system, “listening” to and then broadcasting data without affecting or even notifying the system producing the data. This eliminates the need for agents, and for most applications, its impact on performance is almost negligible. Figure 3-3 shows agentless metrics in Prometheus.

Figure 3-3. Prometheus’s exposition format, supported by vendors

The shift to pull-based metrics collection has allowed SRE teams to better control the metrics they collect. The caveat is that for a pull-based system to be effective, it needs a standard data format to eliminate the need for conversion. Similar to Borg, Prometheus created its own exposition format, Prometheus Exposition (followed by OpenMetrics), then wrote clients that use it to expose metrics simply.

Additionally, Prometheus scrapes application endpoints asynchronously with labels included! This makes it much simpler to detect when an application is down. It also makes the metrics reliable and as scalable as the application itself. This also allows for metrics collection with more dimensions, which is a better fit for microservices-based architectures.

Metrics instrumentation becomes part of the application rather than a separate process, as shown in Figure 3-4.

Figure 3-4. Push-based agent instrumentation versus pull-based agentless instrumentation

Another contributor to Prometheus’s reliability is the nature of the pull model. The pull model is inherently reliable because if the collector is down then it simply waits longer to pull the metric, while the push model will fail.

Prometheus thus solved the two big problems Stripe had identified: reliability and collection scalability. It has since been adopted so widely that most open source tools in the cloud native ecosystem support Prometheus metrics exposition.

Its pervasiveness² became especially evident when cloud native ecosystems started to build tools and standards on top of Prometheus. Of these, the most important is Open Metrics, a format standard that is 100% compatible with the Prometheus exposition format. This means that any toolsets or vendors that are compatible with Prometheus are now forced to be interoperable with each other.

Both metrics sinks and Prometheus tools and standards, then, give SRE teams greater control over their metrics instrumentation. What impact has this had on the cloud native ecosystem?

Prometheus: The Good and the Not-So-Good

For good or bad, the industry is adopting Prometheus. After Kubernetes, it was the second project to attain “graduated” status from the Cloud Native Computing Foundation, which requires meeting stringent criteria.

Prometheus itself has many advantages, the foremost of which are:

Dimensional metric data models

Prometheus uses a dimensional metric data model that allows flexibility when labeling metric data. You can use these dimensions to query metrics using the PromQL language.

Service Discovery

Prometheus can use Service Discovery native to the system Prometheus is monitoring. For example, Prometheus can self-discover pods endpoints using Kubernetes’s own Service Discovery APIs.

Deep integration between PromQL and alerting

Prometheus has a built-in Alertmanager subsystem that can push to paging systems like PagerDuty and Slack. Alertmanager uses PromQL to build alerts and thresholds.

However, with all good systems, it has disadvantages as well:

Generic use case

The use case for Prometheus is too generic: it isn’t built for any one type of application, so you have to configure it for your specific system, including creating metadata labels for each metric type. The relabeling configuration becomes complex as you collect more metrics.

Annotation leads to complexity

The more dimensions your metrics have, the more complicated it gets to configure Prometheus scraping. You can solve this problem easily by using tools like PromLens and taking care to annotate metrics only when absolutely necessary.

Hard to operate reliably

Prometheus is hard to operate reliably. Prometheus runs as a single binary, which means it’s easy to stand up but harder to keep running on unexpected errors. Having Prometheus run in production means tweaking and fine-tuning to keep Prometheus reliable. You end up spending time on Prometheus that you could (and should!) be spending on your core business applications instead. The only exception to this rule is if your business runs Prometheus full time, as with the fully managed options we will discuss later in this chapter.

Horizontal scalability

The biggest disadvantage of Prometheus is that its server uses vertical scaling.

We’ll discuss horizontal scaling in more depth later, but first you need to understand how scaling works, so let’s take a quick detour. In general, there are two types of scaling: horizontal and vertical. Horizontal scaling, also sometimes called fan-out scaling, is based on multiple servers, while vertical scaling is based on the resources of one server. Most distributed systems are scaled horizontally because it is faster and more cost-effective.

Prometheus does not support horizontal scaling out of the box. Most Prometheus servers instead scale vertically. For big Prometheus deployments, then, you need a giant server with lots of CPU and memory. This is bad for many reasons; let’s look at four of them:

Single point of failure

If there is an outage in the Prometheus server’s region or data center, that server becomes a single point of failure. Compare that with cloud native systems, which are built on the assumption that networks are unreliable and compute is ephemeral.

Easily overloaded

The Prometheus server tends to get overloaded with tasks. As more applications go online, it scrapes metric data more slowly.

Hard to automate

Vertical scalability means that the Prometheus server must be treated like a pet, not cattle, as the famous analogy goes. Patches and configurations cannot be automated easily—and should not be—in case of any version incompatibility.³

Limited scalability

In the cloud, there is a high ceiling of compute types available for machines, but this is by no means infinite. At some point, vertically scaling Prometheus is no longer an option.

That said, there is a way to scale Prometheus servers horizontally.

Horizontal Scalability

A Prometheus server is itself an intricate web of tools. Take Alertmanager for example, a component of the Prometheus server that generates alerts on the back of PromQL-based queries. Each component has to be scaled separately. The same is true for most other Prometheus components, like HTTP Service Discovery, Prometheus Storage, Prometheus Querying API, and Prometheus WebUI.

Self-Managed Remote Storage Options

There are two main options for horizontally scaling Prometheus: self-managed remote storage and fully managed SaaS solutions. We’ll look at three self-managed options first, then move to four fully managed options.

Thanos

Thanos adheres to two classic Unix philosophies: “do one thing well” and “keep it simple.” This is important because its architecture can get complicated fast.

Thanos does not divide the Prometheus server into multiple functionality components. Instead, it uses its own APIs to extend the Prometheus server. You might say Thanos runs as a sidecar attached to a Prometheus pod.

The biggest advantage of this is that you can pick and choose which functionalities you need while running your Prometheus server similarly to how you would run a single server. Persistence is handled by your cloud provider’s blob storage, and there is no need for sharding individual Prometheus components.

According to Thanos documentation, “The only explicitly scalable components are query nodes, which are stateless and can be scaled out arbitrarily. Scaling of storage capacity is ensured by relying on an external object storage system.”⁴

In short, the only parts of Thanos you need to run are the sidecar and the blob storage. Everything else is optional. The only additional cost is the price of storing and querying data.

However, the disadvantage is that blob storage performance is worse than reading data in file storage, making queries slower. File storage is not used to cache data in chunks; only index data is cached. This means that each time you run queries that would need blob storage retrieval, the performance suffers.

Cortex

The first project that tried to horizontally scale Prometheus using remote storage was Cortex, famously dubbed the “Frankenstein project” because it stitched together a big cobweb of functionalities, illustrating the difficulty of this task. It has since compacted some of these disparate functionalities into a smaller set of logical units to run.

Cortex has a different approach from Thanos’s minimalism: it designed a set of microservices that individually reuse codes and shards from Prometheus libraries. Its architecture (Figure 3-5) essentially reimagines Prometheus, recreating various components to work as if Prometheus had originally been written with horizontal scalability in mind.

Figure 3-5. High-level concept diagram of Cortex’s architecture⁵

Cortex separates reads from writes. The read part of the architecture uses Cortex Querier to create a horizontally scalable way to query Prometheus remote storage. Querier heavily relies on caching query results, queueing large queries, and splitting large queries into different Queriers to produce quick results.

The write part of Cortex is much more complex. Cortex Distributor handles incoming data from Prometheus scraping, ensuring it is valid and not duplicated. It uses a hash ring algorithm with a key value database (like etcd or consul) to ensure high availability.

Storage is done through the Cortex ingesters, which are also highly available and semistateful. They achieve high availability by splitting the storage using a hash ring algorithm, similar to writes. The Cortex ingester retains the data in memory until a batch job is run, then puts the data into long-term object storage.

You may have noticed some parallels with Thanos. ​​Cortex is much more complicated to run than Thanos, but also generally more scalable since it shards individual components and runs them as microservices.

In March 2022, Grafana Labs announced they forked Cortex and relicensed it as Mimir under the AGPLv3 license. As a result, they will no longer contribute to Cortex. Since Grafana Labs employees were the primary maintainers of Cortex, its future remains uncertain as of this report’s publication. For the time being, it continues to be a project licensed under Apache 2 as part of the CNCF, and new maintainers will need to maintain the project independently from Grafana’s fork, Mimir. As a fork, Mimir reuses similar components and architectural principles as Cortex.

M3

M3 provides a turnkey, scalable, and configurable multitenant store by using M3DB, its scalable backend-storage solution, as a Prometheus remote storage backend. M3 Coordinator acts as a Prometheus sidecar to horizontally scale writes and reads.

Not only do M3DB instances scale Prometheus’s storage horizontally, but they can also improve its performance. Because M3 reads and writes to disk rather than blob storage, reads and writes can be much faster, as they are with Thanos or Cortex. The trade-off is that M3’s storage is not infinitely scalable. In practice, this is less of a concern when running in the cloud, where you can request block storage and nodes on demand. Additionally, M3 and Cortex are more complicated to run than Thanos because they use sharded architecture for individual components.

Both Thanos and Cortex have larger communities than M3DB does. At a production-ready scale, however, M3DB has proven to be a good fit for big companies like Walmart and Uber. M3 also allows for more specific granularity in retaining metric data: for example, all metrics tagged with keep_30days can be kept for 30 days with a simple configuration. You can also downsample metric data in long-term storage to minimize its size.

In a review of M3DB for Prometheus-as-a-service, Logz.io notes that its capabilities are “not as robust as one would expect for a production-level database.”⁶ Be sure to run your own tests to assess whether it fits your use case.

Choosing the right self-managed remote storage option

The primary drawback to self-managed remote storage is that it is complicated to build and run with high levels of availability. The lifecycles and configurations of these three systems (M3, Thanos, Cortex) require specialized knowledge of the inner workings of both that system and Prometheus.

How much do ease of configuration and performance matter to you? If you need high-performance reads and writes with easy configuration and proven production scale, running M3 may be preferable. If cost matters more, Thanos might be the right option, since it utilizes cheaper blob-storage services. If, however, community-wide adoption and ease of scaling matter most to you and you have a dedicated team that is not afraid of complexity, then Cortex is likely your best choice.

Retention

How long are you keeping your data? Prometheus’s default retention period is 15 days. However, most organizations we’ve worked with retain all metric data for 13 months, whether it relates to production, staging, or even development environments! Do you actually need or use 13 months’ worth of metric data?

Let’s say you’re collecting metrics for development environments and retaining them for 13 months. Is that useful if the development environment gets recycled every week? What if you retained those development metrics for a few weeks instead?

Don’t just stick with the default: base your retention periods for different kinds of data on the outcomes that you can gain by retaining it. If you reduce the retention period for data that you do not need, the overall volume will grow at a much more reasonable rate.

In Prometheus, you can configure retention globally by using --storage.tsdb.retention.size and --storage.tsdb.reten⁠tion​.time flags at startup. The self-managed remote storage systems Thanos and Cortex use blob storage as an ultimate backend store, suggesting the possibility of infinite retention. However, M3 has a different approach, which we’ll examine in a moment.

Resolution

How often do you collect metric data? Can you improve your outcomes by collecting data more frequently? The frequency of data collection is called resolution—more data points, just like more pixels in a photograph, would mean a higher resolution.

Let’s say your development environments are deploying multiple times a day. Do you need per-minute metrics for all of them? Perhaps for some applications that would be helpful. But other environments or applications might need only metrics every 10 seconds or every minute. Some might not need metrics at all! Collecting at a lower resolution for certain environments can drastically reduce your metric data’s growth.

Using Prometheus, you can resolve individual scrape jobs using scrape_interval. This example below resolves metrics every minute:

- job_name: nginx_ingress
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https

Feel free to make the scrape_interval as granular as possible, since it is tied to individual scrape jobs.

Applying Resolution and Retention in M3

If you are using M3, you can apply both resolution and retention strategies with the mapping rules feature. A helpful doc by M3 examines the following rule:⁴

downsample:
  rules:
    mappingRules:
      - name: "mysql metrics"
        filter: "app:mysql*"
        aggregations: ["Last"]
        storagePolicies:
          - resolution: 1m
            retention: 48h
      - name: "nginx metrics"
        filter: "app:nginx*"
        aggregations: ["Last"]
        storagePolicies:
          - resolution: 30s
            retention: 24h
          - resolution: 1m
            retention: 48h

In this rule, the authors note, “We have two mapping rules configured—one for mysql metrics and one for nginx metrics. The filter determines what metrics each rule applies to. The mysql metrics rule will apply to any metrics where the app tag contains mysql* as the value (* being a wildcard). Similarly, the nginx metrics rule will apply to all metrics where the app tag contains nginx* as the value.”

M3’s metric data storage policies can define dynamic settings that allow for a mix of retention and resolution, unlike those of Thanos and Cortex, which only has a single resolution and retention setting for all the metrics when scraping metrics, supported by default in Prometheus.

Aggregation

Aggregation is perhaps the most effective of these three strategies. Most applications that generate metrics generate high cardinality and volume by default.

Imagine you have a web application that is running behind NGINX Proxy. NGINX produces very granular HTTP metrics: when you start scraping metrics, by default, it’s all or nothing. Do you really need data for every dimension? If you’re trying to measure the latency of your HTTP responses, will it be useful to know the NGINX version or which data center it’s running on?

Combining the right dimensions and dropping any that aren’t useful is important in controlling the growth of your metric data.

Using the aggregation strategy is about paying attention to what you’re capturing, choosing the metrics and dimensions you really need and combining them, and dropping the rest to keep metrics growth under control. Additionally, the aggregation strategy can be used in conjunction with resolution and retention strategies.

In Prometheus, you can aggregate metrics via recording rules, which periodically compute expensive queries in the background, such as aggregating and dropping a high-cardinality dimension.

While recording rules can help improve performance by computing aggregates, they do not make it possible to drop the original data; doing so requires a federated setup and is quite cumbersome to manage.

To successfully reduce metric data through aggregation in Prometheus, you have to federate Prometheus instances and then combine and drop metrics via recording rules, as in Figure 4-2.

Figure 4-2. Showing a combination of federation and recording rules to aggregate metrics

The reason we aggregate then federate is to work around the limitation of having to store data in Prometheus before we can use recording rules to aggregate it. In this setup, we store, then aggregate, then forward only the aggregated data to another Prometheus instance. This allows us to realize a net reduction in metric data at our final storage location.

Here is an example of a recording rule for aggregation:

groups:
 - name: node
   rules:
    - record: job:process_cpu_seconds:rate5m
      expr: >
        sum without(instance)(
          rate(process_cpu_seconds_total{job="node"}[5m])
        )

This runs the rate(process_cpu_sec⁠onds_total{job="node"}​[5m]) query, dropping the dimension instance, then creates a new metric, job:process_cpu_seconds:rate5m. This new metric has a lower cardinality than the original process_cpu_seconds metric.

Applying Aggregation in M3

Applying aggregation in M3 has a similar effect as in Prometheus. The main difference is that M3 has functionalities called mapping rules and rollup rules, which, when combined, drop unnecessary metrics efficiently by performing aggregation before anything is written and choosing what to keep. This makes aggregating cost-effective. These functionalities also allow M3 to do the same aggregation as Prometheus—without needing to create a federated setup. Consider an example from M3:⁵

downsample:
  rules:
    mappingRules:
      - name: "http_request latency by route \ 
          and git_sha drop raw"
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
      - name: "http_request latency by route \ 
          and git_sha without pod"
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
      - name: "http_request latency by route \ 
          and git_sha without pod"
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
            metricName: "http_request_bucket" \ 
              # metric name doesn't change
            groupBy: ["le", "git_sha", "route", \ 
              "status_code", "region"]
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
            metricName: "http_request_bucket" \ 
              # metric name doesn't change
            groupBy: ["le", "git_sha", "route", \ 
              "status_code", "region"]
        filter: "__name__:http_request_bucket k8s_pod:* \ 
          le:* git_sha:* route:*"
            metricName: "http_request_bucket" \ 
              # metric name doesn't change
            groupBy: ["le", "git_sha", "route", \ 
              "status_code", "region"]
            type: "Increase"
        - rollup:
            metricName: "http_request_bucket" \ 
              # metric name doesn't change
            groupBy: ["le", "git_sha", "route", \ 
              "status_code", "region"]
            aggregations: ["Sum"]
        - transform:
            type: "Add"
        storagePolicies:
        - resolution: 30s
          retention: 720h

The rollup rule above eliminates the k8s_pod label for the http_request_bucket metric that we’re matching against. To do this, we add up the http_request_bucket metric grouped by the other dimensions it has that we want to keep. In addition, we pair it with a mapping rule that drops the original data, which allows us to retain the original http_request_bucket metric name rather than creating a new metric name for the aggregate.

Conclusion

Explosive data growth does not equate to better observability—there is so much more to it. Finding the right balance between too much information and not enough is key. The metrics you capture and retain should be useful to your business goals and outcomes and should measure crucial business and application benchmarks. These three key strategies—resolution, retention, and aggregation—can help you control the growth of your metric data, so you’re only getting and keeping what counts. Remember, it’s all about outcomes.

Out-of-the-Box Standard Instrumentation and Dashboarding

Life is increasingly complex in the DevOps world. Software engineers are expected to do much more than write code: we productionize applications, architect and build systems, make everything compatible across the organization, and ensure that everything we make follows standards.

Thanks to open source tools like Envoy and NGINX, though, SRE teams and software engineers can enable standardized metrics and dashboards right out of the box. For example, if you’re collecting Prometheus metrics with Envoy or NGINX, you can give any HTTP-based application a Grafana dashboard. If you need organization-specific metrics, such as for sales on a specific API, you can build a sales dashboard for use by any team that implements the API. You can even create a RED (Request Rate, Request Error, Request Duration) metrics dashboard out of the box, so that your software engineers can observe and monitor these customized, prebuilt applications without needing to learn PromQL or add intrusive instrumentation to their code.

Rely on your software engineers or SRE/observability teams to help you build and create standardized dashboards. They know the organizational context better than any vendor. For example, if your organization relies heavily on Remote Procedure Calls (RPCs), you’ll want to create RPC dashboards and alerts. You can collect these metrics using appropriate middleware (such as the Java or Go Prometheus gRPC middleware libraries) or metrics exposed by an RPC proxy. (Envoy, for instance, supports proxying gRPC traffic and exposes the corresponding metrics). If your main event bus management system is Kafka, you could start collecting metrics with the Kafka metrics exporter for Prometheus, then create dashboards and alerts that monitor all Kafka topics for each application. What do your colleagues need to know to achieve their desired outcomes?

Adding Business Context to Standardized Metrics

Once you have a respectable toolset of standardized metrics and dashboards, it’s time to tie them into your business context.

Perhaps you decide to enrich your metrics by adding a new label, client_name, to the standard out-of-the-box instrumentation. You can use client_name to understand how traffic is being served between different clients. For example, if client_name ACMECorp is creating more requests to your services, then you can scale the servers hosting ACMECorp and possibly send an email to ACMECorp about the increased usage.

Another example of adding business context is customizing how you route alerts. You might add the names of the app and the team that owns it to the alert, so it’s always routed to the right people. You can even go one step further by adding dependent applications. This way, the alerts go not only to the team that owns the app but also to the teams downstream from that app. This allows the organization to detect a cascading failure and pinpoint where it starts.

Tiering applications is another common use case for adding business context to metrics. Some businesses have systems that should never stop running, under any circumstances—if they did, it would cost the business more than any other applications. You can add labels with tiers to differentiate the most crucial applications, then route alerts differently based on tier. This means that if a system labeled tier=1 goes down, you can alert not only to your technical staff but also sales, PR, and the executive team, allowing everyone to act quickly and get ahead of potential problems.

Creating SLOs from Standardized Instrumentation

Service levels, which you can derive from your metrics, are a great way to align site reliability with your business goals. Stavros Foteinopoulos of Mattermost lays out three concepts that are key to understanding service levels: service level indicators (SLIs), service level objectives (SLOs), and service level agreements (SLAs).¹

Foteinopoulos defines the SLI as a “carefully defined quantitative measure of some aspect of the level of service provided”—in short, a metric. An SLO is “a target value or range of values for a service level that is measured by an SLI,” or what you want your metric’s value(s) to be. An SLA is a contract that promises users specific values for their SLOs (such as a certain availability percentage) and lays out the consequences if you don’t meet those targets (SLOs).

Let’s say your company builds an API that provides memes about cats for app developers to use. Your SLI is the percentage of time your API is available to all downstream external customers. If your SLO is 99% availability, that means you’ve promised your customers no more than 3.65 days of downtime per year. This is measured by an error rate formula, since down time is at its core a kind of error. The formula, as Foteinopoulos notes, is:

If downtime errors exceed the limit specified in the SLO, your SLA states that you agree to donate $1 to an animal shelter per additional minute of downtime.

The beauty of SLAs is that they are not restricted to external customers. Some companies use internal SLAs to assign employees an “error budget,” then rotate staff between software engineering and SRE based on whether their error counts stay within the specified values. Steven Thurgood of Google writes in The Site Reliability Workbook that error budgets “are the tool SRE uses to balance service reliability with the pace of innovation.”²

If you use availability as an SLI, you’ve likely already instrumented a set of standardized metrics, like the Prometheus RED metrics. You can use those metrics to build a dashboard, which will make it fairly straightforward to create realistic SLOs based on your performance. Sloth by Slok (Xavier Gallego) is a tool for onboarding service levels. It lets you create Prometheus rules and Grafana dashboards quickly and easily, in a way that stays up-to-date as you add new applications to your system over time.

In addition to using standardized metrics like the RED metrics, it’s also important to standardize across the organization what each SLI means. For example, what does 99% availability mean? Does that mean if any error occurs in a time window, it’s not available? Does it mean the error rate has to be above 99%? What is the time window we’re checking? Every minute, every hour? The answer to this varies, but the rule of thumb is that it needs to be consistent across an organization.

Monitor the Monitor

Who monitors the monitoring system? Do you even know? What happens when it goes down?

For the sake of reliability, your monitoring should not live in the same region where you run the infrastructure. If it does and the region is misconfigured or runs out of resources, then you can no longer monitor the infrastructure within that region.

If possible, use multiple regions or a region separate from where your application lives. This way you can monitor the cluster and applications even when there is a regional data center outage. This matters whether you run your self-managed monitoring system in the cloud or data center or even if you are utilizing a fully managed option.

Ideally, you’ll want to use a different cloud provider from the one you use for production workloads. This ensures you will still have a monitoring system even if an entire cloud service goes down. SaaS fully managed service providers usually use cloud providers as well. Ensure that their system does not utilize the same provider or the same region where your production workloads are running.

Finally, use an external probe on the internet (if it is a publicly facing application) or an internal probe in a different region but within your network. This gives your engineers real-time user monitoring: they can see what your end users see. End users’ experiences, whether they are external or corporate, should always be your top priority when monitoring applications.

Write and Read Limits

Because metric cardinalities are multiplicative, one engineer can write one query, but that query can read metrics with a cardinality in the order of millions or, in the absolute worst case, billions. Imagine, for example, that a software engineer accidentally lists all of the cardinalities within Prometheus by using a wildcard star (*).

No matter how powerful your backend system is, whether you are writing metrics or reading them, you cannot safely guarantee that your system will not be overloaded. Build a detection system that allows you to understand if one of your queries or writes will cause an outage.

For write use cases, show your developers the cost of each metric write they add by allowing them to monitor their publishing rate. Once they can view this, they can see their usage relative to other applications and teams in their organization, and it is easy to show them when they are using more than their fair share of resources. This will help them assess and think twice about whether they really need a metric before adding it to the observability system. Furthermore, once they can view writes, you can start blocking these writes.

For read use cases, remember that every query uses the same set of resources. Users should only query for the data they actually need and make sure the cardinality matches their appropriate use. For instance, if you never query the P99 latency on a per container or pod basis, then there is no need to query the high cardinality metrics from your dashboards or alerts. Instead, query aggregated metrics that don’t have the pod or container granularity (either using recording rules or an intermediate aggregator).

Safe Ways to Experiment and Iterate

Let’s say you have a basic monitoring system. You want to keep it modern and highly upgradeable, but the observability tool’s landscape changes too fast for you to keep up. What’s more, SREs are in high demand and tend to have high turnover. This can create a sense of fear that anything you touch will break the observability system and lead SREs to conclude that it is much more prudent to leave things as they are.

The problem with that is that you lose the ability to experiment and iterate and to learn new technologies and tools without breaking the system observability tech stack that your team is used to running. This causes a downward spiral of being stuck but also responsible for innovating. This can feel like trying to run a marathon with your shoelaces tied together.

How do we get away from this dilemma? A lot of it is about making it safe to create a whole new set of monitoring data (or a subset of it) in another observability system. Simply spin up another observability system like Prometheus that can scrape data from the /metric endpoint every 10 seconds instead of every second (see Figure 5-1). Try to get 10% of all production observability data onto the new system without your developers doing anything additional.

Figure 5-1. Production Prometheus and experimental production Prometheus with 10% of production data

You can use this different set of observability systems to experiment with things like new relabel rules, upgrading your Prometheus version, creating a new aggregation across different types of metrics, ingesting a new set of data from a completely different type of tech stack, and measuring the daily metric size.

You can also use it to iterate safely on existing tasks, like adding new cardinality to existing metrics. You can even destroy and recreate your observability stack to teach new SRE engineers how its system works. The idea is to make it safe for your SRE team to iterate and experiment without your developer needing to change or redeploy code. This removes the element of fear and empowers your teams to learn more about your observability system and your entire application suite.