Evals for AI Engineers

Systematically Measuring and Improving AI Applications

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Shreya Shankar and Hamel Husain

Evals for AI Engineers

by Shreya Shankar and Hamel Husain

Copyright © 2027 Hamel Husain and Shreya Shankar. All rights reserved.

Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

Acquisitions Editor: Nicole Butterfield

Development Editor: Angela Rufino

Production Editor: Katherine Tozer

Interior Designer: David Futato

Interior Illustrator: Kate Dullea

September 2026: First Edition

Revision History for the Early Release

2025-10-30: First Release

2025-11-13: Second Release

See https://oreilly.com/catalog/errata.csp?isbn=9798341660724 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Evals for AI Engineers, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

979-8-341-66067-0

Brief Table of Contents (Not Yet Final)

Chapter 1: Introduction (available)

Chapter 2: LLMs and Evaluation Basics (available)

Chapter 3: Error Analysis (unavailable)

Chapter 4: Implementing Automated Evaluators (unavailable)

Chapter 5: Collaborative Evaluation Practices (unavailable)

Chapter 6: Evaluating Multi-Turn Conversations (unavailable)

Chapter 7: Specific Architectures and Data Modalities (unavailable)

Chapter 8: Continuous Integration and Deployment (unavailable)

Chapter 9: Interfaces for Continuous Human Review (unavailable)

Chapter 10: Cost Optimization (unavailable)

Preface

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at arufino@oreilly.com.

Large language models have moved from research curiosity to production reality with remarkable speed. Organizations now routinely embed LLMs in customer service systems, content generation pipelines, decision support tools, and information extraction workflows. Yet this rapid adoption has outpaced our ability to systematically evaluate whether these systems actually work.

This book addresses that gap.

Unlike traditional software with deterministic outputs, LLM pipelines produce responses that are often subjective, context-dependent, and multifaceted. A response might be factually accurate yet inappropriate for the context. It might sound persuasive while conveying incorrect information. It might also address most but not all parts of the user’s question. These ambiguities make evaluation fundamentally different from conventional software testing or even traditional machine learning validation.

The challenge we tackle is straightforward to state but difficult to solve: How do you assess whether an LLM pipeline is performing adequately? How do you diagnose where it’s failing? And how do you systematically improve it?

Note

This book has been refined through something rare for technical books: extensive real-world iteration with over 3,000 engineers building production AI systems. Through multiple cohorts of our course AI Evals for Engineers and Technical Product Managers and hands-on consulting engagements, we’ve worked directly with practitioners facing the same challenge: they had built LLM applications, but couldn’t reliably assess whether those applications actually worked. The approaches you’ll learn aren’t theoretical—they’ve been tested and improved through real-world deployment across hundreds of companies.

What This Book Covers

This book introduces a practical framework for evaluating LLM pipelines: the Analyze-Measure-Improve lifecycle. This iterative approach helps you:

Analyze your pipeline’s behavior on representative data to identify failure modes

Measure those failures quantitatively using automated evaluators

Improve your system based on concrete evidence rather than guesswork

We cover the complete evaluation journey, from initial error analysis through production monitoring:

Understanding the unique challenges of LLM evaluation (the “Three Gulfs” model)

Systematic error analysis using qualitative coding techniques

Collaborative evaluation practices for building shared understanding across teams

Implementing automated evaluators, including LLM-as-judge approaches

Evaluating specific architectures: multi-turn conversations, RAG systems, tool-calling, and agentic systems

Continuous integration and deployment for LLM pipelines

Building interfaces for ongoing human review and error analysis

Strategies for improvement: accuracy optimization, cost reduction, and model cascades

Each chapter combines conceptual frameworks with practical implementation details. You’ll find code examples, real-world case studies, common pitfalls to avoid, and exercises to test your understanding.

Who Should Read This Book

This book is written for engineers and technical product managers building software products with generative AI components. We assume you have:

Familiarity with large language models and how they work

Experience building (or planning to build) applications that use LLM APIs

A practical need to know whether your LLM pipeline is actually working

You don’t need to be an expert in machine learning or statistics. While we include code examples and some mathematical notation, we always explain the underlying concepts. If you’re not comfortable with code or math, focus on the high-level frameworks and practical patterns, you’ll still gain tremendous value from understanding the evaluation process, forming the right mental models, and recognizing common pitfalls.

Product managers will find particular value in the chapters on error analysis, collaborative evaluation, and the strategic framework for thinking about evaluation across the product lifecycle. Engineers will appreciate the detailed implementation guidance, code examples, and production deployment strategies.

What This Book Doesn’t Cover

This book focuses exclusively on application-centric evaluation, or assessing whether your specific LLM pipeline works for your specific use case. We deliberately don’t cover:

General-purpose foundation model benchmarking (like MMLU, HellaSwag, or other academic benchmarks)

Training or fine-tuning foundation models from scratch

Comparative model selection based solely on benchmark performance

If you’re trying to decide which foundation model to use, start by prototyping with your actual application and data, then use the techniques in this book to evaluate whether each model meets your requirements.

How This Book Is Organized

The book follows a logical progression through the evaluation lifecycle:

Part I: Foundations (Chapters 1-2) establishes why LLM evaluation is uniquely challenging and introduces the core concepts you’ll need.

Part II: The Analyze Phase (Chapters 3-4) teaches systematic error analysis and collaborative evaluation practices for understanding what your pipeline is actually doing.

Part III: The Measure Phase (Chapters 5-8) covers implementing automated evaluators and handling specific architectures like RAG, multi-turn conversations, and agentic systems.

Part IV: Production & Improvement (Chapters 9-11) addresses continuous integration, monitoring, interfaces for ongoing review, and systematic improvement strategies.

You can read the book linearly or jump to chapters relevant to your immediate needs. However, we strongly recommend reading Chapters 1-3 before diving into later sections, as they establish the foundational framework that everything else builds upon.

Conventions Used in This Book

The following typographical conventions are used in this book:

Italic

Indicates new terms, URLs, email addresses, filenames, and file extensions.

Constant width

Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values or by values determined by context.

Tip

This element signifies a tip or suggestion.

Note

This element signifies a general note.

Warning

This element indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/ai-evals-book.

If you have a technical question or a problem using the code examples, please send email to support@oreilly.com.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: "Evaluating LLM Applications by Shreya Shankar and Hamel Husain (O’Reilly). Copyright 2026 Shreya Shankar and Hamel Husain, 978-0-596-xxxx-x.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

O’Reilly Online Learning

Note

For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com.

How to Contact Us

Please address comments and questions concerning this book to the publisher:

O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-889-8969 (in the United States or Canada)

707-827-7019 (international or local)

707-829-0104 (fax)

support@oreilly.com

https://www.oreilly.com/about/contact.html

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://learning.oreilly.com/library/view/evals-for-ai/9798341660717/.

For news and information about our books and courses, visit https://oreilly.com.

Find us on LinkedIn: https://linkedin.com/company/oreilly-media

Follow us on Twitter: https://twitter.com/oreillymedia

Watch us on YouTube: https://youtube.com/oreillymedia

Acknowledgments

This book would not exist without the thousands of engineers and product managers who have participated in our course, AI Evals for Engineers & Technical PMs, and the many companies we’ve partnered with through consulting engagements. Your real-world challenges, questions, feedback, and shared experiences have shaped every chapter.

We’re grateful to our technical reviewers [NAMES TK] for their thorough feedback and for catching our mistakes.

Thanks to our editor [NAME TK] at O’Reilly for believing in this project and for patience as we translated course materials into book form.

A Note on the Pace of Change

The field of LLM evaluation is evolving rapidly. By the time you read this, new models, tools, and techniques will have emerged. However, the core principles in this book; understanding your data, systematically analyzing failures, measuring what matters, and improving based on evidence; remain constant regardless of which model or framework you’re using.

We’ve designed the Analyze-Measure-Improve framework to be tool-agnostic and model-agnostic. If a new LLM comes out tomorrow, the methodology in this book will still help you evaluate whether it works for your use case.

If, after reading this book, you want to deepen your skills and connect with others working on LLM evaluation, check out our course. It offers an interactive, hands-on environment with live office hours, cohort- based discussions, and ongoing access to an expert community.

Let’s get started.

Chapter 1. Introduction

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at arufino@oreilly.com.

The past two years have seen rapid advances in the development and deployment of large language models (LLMs). Organizations are now embedding LLMs in critical pipelines in applications: customer service, content creation, decision support, and information extraction, among others Bommasani 2021, Zaharia 2024. However, adoption is outpacing our ability to systematically evaluate LLM applications Ward 2024.

Unlike traditional software, LLM applications do not produce deterministic responses, nor do they have formal specifications for the logic they should and should not perform. Their outputs aren’t predictable. Sometimes the outputs are obviously incorrect. Other times, it is hard to describe what is wrong. An output can be factually right but seem inappropriate (the “vibes are off”), or sound completely convincing while being totally incorrect. These ambiguities make evaluation fundamentally different from conventional software testing and even traditional machine learning (ML).¹

The core challenge of evaluation is as follows: How do we assess whether an LLM pipeline is performing adequately? And how do we diagnose where it is failing? Our book focuses on this gap. In this book, we will build a practical and theoretical foundation for LLM evaluation, with methods that you can immediately apply to your own projects.

What is Evaluation?

Evaluation is the systematic measurement of quality in an LLM application. A good evaluation produces results that are interpretable and actionable. In some cases evaluation produces a quantitative metric. For example, if you are building a travel planning assistant, you might measure how often the system returns valid flight and hotel options within budget, or whether itineraries satisfy the user’s constraints. In other cases, the “metric” is generated with more judgment involved. A set of experts, or sometimes the product team itself, may review outputs and rate them for qualities such as clarity, tone, or usefulness. Both approaches fall on a spectrum of rigor: from quick, informal checks that guide early iteration, to carefully defined rubrics and systematic sampling that support production systems.

A single application will usually require multiple evaluations, each designed to capture a different dimension of performance. Evaluations can then be integrated into the application in several ways:

Monitoring: continuous checks that run in the background to detect drift or degradation over time.

Guardrails: evaluations placed in the critical path, allowing the system to block, retry, or fall back when outputs fail predefined criteria.

Improvement mechanisms: evaluations used to generate feedback, such as labeled examples for fine-tuning, curated few-shot prompts, or systematic identification of failure cases that motivate architectural changes.

So, overall, evaluations serve multiple purposes. They are necessary to detect errors, maintain user trust, enforce safety constraints, monitor ongoing behavior, and guide systematic improvement. Without them, teams are left uncertain about how their systems perform and where they need to intervene.

Evaluation Throughout the Lifecycle of an LLM

Evaluation is not confined to applications of LLMs. It is a constant presence across the entire lifecycle of an LLM. Interestingly, at each stage (pre-training, post-training, and then any applications we build to top) the role of evaluation is different, reflecting distinct goals and constraints. Understanding these stages helps clarify the spectrum of capabilities being tested, and why there can be no single evaluation framework that applies everywhere.

Pre-training

Pre-training is the first stage in the lifecycle: building a foundation model from scratch. A large neural network is exposed to massive text (and sometimes code) corpora with a single objective: to predict the next token in a sequence. In pretraining, the model acquires broad linguistic knowledge and some capacity for reasoning and world knowledge.

Pre-training is carried out by large research labs and commercial model providers. The process requires substantial computational resources, often costing hundreds of millions of dollars. Evaluation at this stage is largely intrinsic: measuring how accurately the model predicts held-out tokens from its training distribution. Pretraining metrics indicate whether the model has absorbed the statistical patterns of language, but do not yet assess its usefulness for many of the downstream tasks that users want to leverage LLMs for.

Post-training

Post-training takes a pre-trained model and adapts it for practical use. It turns out that models trained only on next-token prediction can already generalize to many unseen tasks, or tasks that LLMs weren’t explicitly pre-trained on. Post-training builds on this ability by steering the model toward behaviors that users care about; e.g., following instructions, staying on topic, and avoiding unsafe responses.

Post-training typically combines supervised fine-tuning on curated prompt–response pairs with preference-based methods such as Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). Evaluation here shifts from intrinsic language modeling metrics to benchmarks and datasets that approximate user-facing tasks: question-answering, reasoning challenges, safety and bias tests, coding, and more. These evaluations are designed and run by the model providers themselves, and their results are often reported as headline benchmarks.

Application

The application stage is where most engineers and product managers engage with LLMs. At this point the model has already been trained and aligned by a model provider, and the task is to integrate it into a concrete workflow; e.g., triaging emails, assisting customer support, or analyzing documents. Evaluation at this stage must be tailored to the goals and data of the specific application. In some cases, existing post-training evaluations overlap directly with the task: for example, a coding assistant can draw on established programming benchmarks. In most cases, however, the application is not represented in public leaderboards. A system for reviewing contracts, summarizing medical notes, or checking compliance will require its own evaluation methods.

Responsibility therefore falls to the application development team to define what success means, design appropriate metrics, and implement monitoring to verify that the system behaves reliably for its intended use.

Evaluation in Applications

At the application stage, you are responsible for more than just plugging in a model. You have to decide what the product should achieve, how to translate that goal into prompts and pipeline logic, and how to verify that the model’s outputs are acceptable. Each of these steps introduces its own difficulties:

Objectives are vague at first. A goal like “summarize emails” sounds simple, but only when you inspect outputs do the ambiguities appear: how detailed, in what format, and with what exclusions?

Instructions might have gaps. Prompts and pipeline logic expressed in natural language rarely capture the full intent, which leads to inconsistent or incomplete behavior.

Behavior changes on new data. A setup that works on a handful of test cases may fail once it encounters the variability of real inputs.

In practice, these issues show up again and again when building LLM applications. To reason about these challenges more systematically, we use the Three Gulfs framework: comprehension, specification, and generalization. Each gulf highlights a different point where development typically goes wrong, and each requires a different strategy to address—​a theme we will return to throughout this book.

The Three Gulfs of LLM Application Development

Figure 1-1. Three Gulfs Model of Challenges in LLM Pipeline Development. Adapted from Steering Semantic Data Processing with DocWrangler Shankar 2025.

A useful way to understand the challenges of building with LLMs is the Three Gulfs model (Figure 1-1), adapted from Shankar 2025 and inspired by Norman 1988. The model highlights three common gaps that developers face when building and evaluating applications: comprehension, specification, and generalization. Later in this book, we will use this framework to diagnose why an application is failing, and select the most appropriate strategy to fix it.

Example 1-1. Example: Email Processing Pipeline

Consider a pipeline that processes incoming emails sent to an organization. The goal is to extract the sender’s name, summarize the key requests, and categorize the email. At first glance, this seems straightforward. But each stage of development exposes different challenges across the Three Gulfs.

The Gulf of Comprehension (Between the Developer and Data)

The first gulf is between the developer and the data. To build effective prompts or pipelines, you need a solid grasp of what your inputs actually look like. Without that, you will design against an imagined version of the task, not the real one.

Consider the email processing example. If you only look at a handful of clean, well-written emails when drafting your prompt, you might write instructions that work fine on those cases. But the production data will also include forwards, replies with long quoted threads, messages with signatures in multiple languages, and poorly formatted text copied from PDFs. A prompt that ignores these realities might lead the LLM to, for example, summarize an entire quoted thread instead of the new message, or mistake a signature block for the main request.

The gulf of comprehension is not only about knowing your inputs. It is also about having a clear sense of how the LLM behaves on those inputs. You cannot read every output it generates, but you still need to recognize the common ways it succeeds and the common ways it fails. Otherwise, it’s easy to be misled by a few good examples and miss the patterns that only show up at scale. In the email pipeline, for example, the LLM might usually extract the sender and write a decent summary. But when you look closer, you may notice recurring mistakes: pulling the wrong name if several are mentioned, dropping key details in longer messages, or copying boilerplate like confidentiality notices. These failures may only happen occasionally, but if they occur, the system will feel unreliable to users.

Bridging the Gulf of Comprehension means developing strategies to characterize both your inputs and your outputs at scale—​so you know what the data really looks like, and how the system actually behaves—​without having to inspect everything by hand.

The Gulf of Specification (Between the Developer and the LLM Pipeline)

The second gulf is between what we want and what we actually tell the LLM to do. Our intent—the task we have in mind—is often only loosely captured by the prompts or pipeline logic we write. Natural language is imprecise, and small gaps in how we phrase things can lead to large differences in behavior.

Take the instruction:

Extract the sender’s name and summarize the key requests in this email.

At first glance it looks specific, but there are important details left unstated. Should the summary be a paragraph or a bulleted list? Should the sender be the display name, the full email address, or both? Should implicit requests be included, or only explicit ones? How concise should the summary be?

The LLM cannot fill in these choices for you. If you don’t make them explicit, the system will guess—​and those guesses will not always line up with what you expect. That’s why two runs of the same pipeline might produce outputs in different formats, with varying levels of detail, or with inconsistent handling of edge cases.

The Gulf of Specification is this gap between our intent and our instructions. Closing it requires making expectations explicit and aligning them closely with the data the pipeline will actually see.

The Gulf of Generalization (Between the Data and the LLM Pipeline)

The third gulf is between the data your system sees and how the LLM generalizes to it. Even if you have a solid grasp of your inputs and have written a clear, well-scoped prompt, the LLM is still prone to errors. LLMs are not perfect, and they will make mistakes.

For example, imagine you’ve clearly specified that the system should extract the sender’s name from the email header. Most of the time the LLM follows this instruction. But occasionally, when the body of the email mentions another person—​say, a public figure—​the LLM will incorrectly return that name instead. Nothing about your understanding of the data or your prompt was wrong; the LLM simply failed to apply the instructions consistently. Maybe the LLM is too biased in favor of names it has seen in its training data.

This is the Gulf of Generalization: the fact that LLMs can behave unpredictably even when comprehension and specification are handled well. These errors usually become visible only when the system is exposed to the full variety of real-world data. And they are unavoidable: no matter how advanced the models become, there will always be cases where they produce the wrong output.

Note

Why the Three Gulfs Framework Matters

The Three Gulfs framework provides a shared vocabulary for discussing LLM application failures. When debugging with your team, you can quickly identify whether a problem is a “specification failure” (the prompt didn’t make requirements clear enough) or a “generalization failure” (the LLM isn’t applying instructions consistently). This common language helps teams diagnose issues faster and choose the right fixes, rather than debating vague notions of the model “not working.”

Why LLM Pipeline Evaluation is Challenging

Evaluating LLM applications is challenging because every application requires bridging all three gulfs—​comprehension, specification, and generalization—​again in its own context. Evaluation is not a one-time check; it is an iterative process where requirements become clear only after you interact with real outputs.

When you start examining outputs (closing the Gulf of Comprehension), you will inevitably find mistakes. The next question is whether these mistakes are due to incomplete specification or to failures of generalization. If the prompt does not make the requirement explicit, you may fix it with better instructions or pipeline design. If the prompt is already well-specified and the LLM still fails, then the issue lies in generalization—​and automated evaluations can help measure how often that failure occurs.

Applications also differ in how much they can rely on users to bridge the gulfs on their own. In a coding assistant, you can assume programmers have high agency: they know how to specify requirements precisely, can read and debug generated code (comprehension), and can run tests that expose when the LLM has failed to generalize (e.g., generating a working React app in some cases but not others).

In contrast, consider an LLM-powered personal stylist. A user asking for an outfit may not know how to specify requirements clearly. The application itself may need to ask clarifying questions, such as about weather or formality, to reduce specification gaps. Users also cannot easily detect generalization failures: if the LLM produces good suggestions for casual winter wear, they may assume it will also perform well for winter weddings—​an assumption that may not hold.

Effective evaluation requires thinking about the stakes in each gulf for your application. Where the stakes are higher, because users cannot easily catch mistakes themselves, or because the cost of errors is significant, you will need to design more rigorous processes to close the gaps.

The LLM Evaluation Lifecycle: Bridging the Gulfs with Evaluation

The Three Gulfs—Comprehension, Specification, and Generalization—capture the main ways LLM applications go wrong. In this book, we approach evaluation through the Analyze–Measure–Improve lifecycle (Figure 1-2). This lifecycle is the framework we will use to show how evaluation can make these gulfs visible and help you decide how to address them.

Figure 1-2. The Analyze-Measure-Improve evaluation lifecycle. This iterative framework bridges the Three Gulfs by using evaluation to systematically understand failures (Analyze), quantify them (Measure), and fix them (Improve).

Analyze

The cycle begins with Analyze, which addresses the Gulf of Comprehension. At this stage you look at real inputs and outputs to see how the system is behaving. The goal is to identify failure modes, not to quantify them.

Example: In an email summarization pipeline, you might review a batch of outputs and notice two recurring problems: the LLM sometimes includes irrelevant text from signatures, and sometimes omits the most important request. These observations tell you where things break down and give you clues about whether the issue is poor specification or a generalization failure.

Measure

After Analyze, you will know the kinds of mistakes the system makes. Some will clearly be specification problems: the prompt did not say enough, so the fix is obvious. Others will look like generalization errors: the prompt was clear, but the LLM still misbehaves.

The Measure step is about quantifying those generalization issues. You want to know how common they are, and in what contexts they appear. To do this, you design evaluators that can run at scale, on unlabeled data.

Example: If the LLM sometimes pulls signature blocks into summaries even after you clarified the prompt, you might build a rule-based check that flags boilerplate phrases across thousands of outputs. Or, if you suspect summaries are dropping key requests, you might assemble a labeled set of emails with ground-truth requests and check how often they are captured. Measurement tells you whether a failure is an occasional annoyance or a systematic problem that needs attention.

Improve

The final step is Improve. Here you decide how to adjust the system based on what Analyze and Measure have revealed. The path you follow depends on the source of the errors:

If they stem from specification gaps, the fix is usually to tighten the prompt or adjust pipeline logic.

If they stem from generalization failures, you may need stronger interventions such as adding more examples, refining retrieval, or fine-tuning on labeled data.

Example: In the email pipeline, suppose your measurements show that missing key requests is common because the prompt never specified the level of detail. You can fix this by revising the instruction to say, “Summarize the sender’s requests in three concise bullet points.” By contrast, if you discover that the LLM frequently mistakes names in the body for the sender—​even after a clear prompt—​that is a generalization issue. Improving performance here may require adding labeled training data, or redesigning the application to extract the sender from metadata rather than relying solely on the LLM.

By cycling through Analyze, Measure, and Improve, you progressively reduce uncertainty about application behavior. Each pass helps you understand the gulfs you are facing and choose the right level of evaluation rigor for your application.

Putting it Together

This chapter introduced three building blocks for evaluating LLM applications:

The Three Gulfs—comprehension, specification, and generalization—describe the main ways LLM applications fail.

Each gulf has different implications: understanding your data and outputs, making intent explicit, and recognizing that LLMs will still misapply instructions in some cases.

The Analyze–Measure–Improve lifecycle is the process we will use throughout this book to identify these failures, measure their impact, and decide how to fix them.

In the next chapter, we turn to the first practical step: building a prototype application and writing initial prompts. You need something concrete in place—even a rough version—​before you can analyze how it behaves.

¹ In traditional ML, outputs are well-defined—numbers, classes, or structured predictions, and we often have ground truth labels. We can use standard metrics like accuracy or F1 score. In contrast, LLM output can be open-ended, unstructured, and often subjective.

Chapter 2. LLMs and Evaluation Basics

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at arufino@oreilly.com.

In Chapter 1, we motivated why evaluation is essential. But before we can evaluate, we need something to evaluate! This chapter equips you with everything you need to get a first application running and to understand its moving parts.

Specifically, we will:

Ground the discussion with a simple example application.

Break down the components of an LLM application—​from the simplest single call to complex agent loops.

Describe some fundamentals of creating prompts or initial versions of these components.

Clarify the two main evaluation modes we will use throughout this book:

Absolute evaluation: judging whether one configuration (e.g., a specific prompt, model, or tool setup) is correct enough to ship.

Comparative evaluation: comparing multiple options (A/B tests, pairwise, ranking, leaderboards) to decide which is better.

By the end of this chapter, you will have a clear mental model of what makes up an LLM application and some of the types of evaluation you might do for your application.

Components of LLM Applications

Most LLM applications can be described in terms of a few recurring components. Each of these components brings different capabilities, and each comes with its own characteristic failure modes to analyze. We will cover these components in increasing order of complexity, showing how they build on one another to create sophisticated applications.

Before diving into the components, we need to understand how we interact with LLMs programmatically. All the code examples in this chapter use LiteLLM (https://github.com/BerriAI/litellm), a popular Python library that provides a unified interface for calling different LLM APIs (e.g., OpenAI, Anthropic, Google Gemini, etc.) with consistent syntax.

LLM APIs expect input as a list of message objects, where each message has two fields: role and content. The role field specifies who is speaking and can be one of three values:

system: Sets the overall behavior and context for the assistant (e.g., “You are a helpful assistant that summarizes emails”). This is optional but helps establish consistent behavior across requests.

user: Represents input from the human or application making the request. This is where you provide the actual task or query.

assistant: Represents previous responses from the LLM. You include these when maintaining conversation history across multiple turns.

tool: Used to pass the results of a tool invocation back to the LLM. Not supported by all models, but used with LLMs that support tool calling (which we will discuss later in this chapter).

For a single request, you typically send a list with one or two messages: an optional system message to set context, and a user message with your query. For multi-turn conversations, you append each exchange to the list as alternating user and assistant messages.

Tip

When debugging conversation issues, check that you’re alternating user and assistant messages correctly. Mismatched roles are a common source of errors.

For each component below, we present an example code snippet that pertains to the following email summarizer example.

Imagine you are building a simple email summarizer for your organization. It should take an incoming email and produce a short summary that includes the sender’s name and their key requests.

Single-step LLM calls

The simplest component is a single call to a model. You provide an input prompt, the model generates a response, and that response is passed straight back to the user or to the next stage in your pipeline.

For the email summarizer, this could be one prompt that takes the body of an email and asks the model to extract the sender’s name and summarize the requests. In code, a single-step call might look like this:

from litellm import completion

response = completion(
    model="gpt-5",
    messages=[
        {
            "role": "system",
            "content": "You are an assistant that summarizes emails."
        },
        {
            "role": "user",
            "content": "Extract the sender's name and summarize the key requests:\n\n"
                      + email_text
        }
    ]
)

print(response["choices"][0]["message"]["content"])
# Example output:
# "Sender: Alice Johnson
#  Key requests:
#  - Reset her account password
#  - Verify email on file"

Single LLM calls like this are a natural starting point when building applications. But modern applications often combine additional components: e.g., retrieval to bring in related messages, tools to query internal systems, or agent loops that coordinate multiple steps. Each component brings new capabilities, but also introduces its own potential failure modes.

Conversations

Many applications don’t stop at a single request–response exchange. They allow the user and the model to interact over multiple turns. This conversational setting is what most people associate with chatbots, customer-support assistants, or copilots.

In our email example, instead of just producing a summary given a single user message, the system could support follow-up questions:

User: Summarize the email.
System: Sender: Alice Johnson. Key requests: reset her account password; verify email on file.
User: Does she mention her preferred contact method?
System: Yes. She asks to be contacted by phone.

Technically, a conversation is just a series of LLM calls where the prompt includes the full dialogue history (or at least the recent turns). Here is a minimal illustration:

from litellm import completion

messages = [
    {
        "role": "system",
        "content": ("You are an assistant that summarizes and "
                    "answers questions about emails.")
    },
    {"role": "user", "content": "Summarize this email:\n\n" + email_text}
]

response1 = completion(model="gpt-5", messages=messages)
print(response1["choices"][0]["message"]["content"])
# "Sender: Alice Johnson. Key requests: reset her account password; verify email on file."

# Add the model's reply to the conversation and continue
messages.append({
    "role": "assistant",
    "content": response1["choices"][0]["message"]["content"]
})
messages.append({
    "role": "user",
    "content": "Does she mention her preferred contact method?"
})

response2 = completion(model="gpt-5", messages=messages)
print(response2["choices"][0]["message"]["content"])
# "Yes. She asks to be contacted by phone."

Conversation brings the system closer to how people naturally interact with assistants. But it also introduces new design choices: how much dialogue history to keep, how to manage context length, and how to ensure consistency across turns. These are all aspects we will return to when we discuss evaluation.

Retrieval

Sometimes, the LLM needs background knowledge that isn’t in the immediate input. Retrieval-augmented generation (RAG) is a common pattern: the system searches a knowledge base or document store, pulls in relevant passages, and then gives both the query and the retrieved context to the model.

In the email summarizer, retrieval might bring in prior messages from the same sender or related support tickets. That way, the model can understand the new message in light of past correspondence—for example, recognizing that “I still can’t log in” refers to an ongoing password reset issue.

There are two common approaches to retrieval:

Keyword-based methods (for example, BM25). These rank documents by the amount of word overlap with the query. They are fast and work well when domain terminology is stable.

Embedding-based methods. These represent queries and documents as vectors and rank them by semantic similarity. They can match meaning even when wording differs (“reset my login” vs. “can’t sign in”).

A typical pipeline has two steps: (1) search for relevant items, then (2) insert those items into the LLM prompt.

# Small archive of past support emails
past_emails = [
    "Ticket #4521: Please reset my password. I cannot log in.",
    "Ticket #4605: Could you verify my email address on file?",
    "Ticket #4710: I need help updating my phone number."
]

# Embedding-based retrieval
from litellm import embedding, completion
import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

corpus_embeddings = [
    embedding(
        model="text-embedding-3-small",
        input=doc
    )["data"][0]["embedding"]
    for doc in past_emails
]

query_embedding = embedding(
    model="text-embedding-3-small",
    input=email_text
)["data"][0]["embedding"]
scores = [cosine_similarity(query_embedding, e) for e in corpus_embeddings]
retrieved = past_emails[int(np.argmax(scores))]

# Combine retrieved context with the new email
messages = [
    {
        "role": "system",
        "content": "You are an assistant that summarizes emails with "
                    "context."
    },
    {
        "role": "user",
        "content": (f"Email:\n{email_text}\n\n"
                    f"Related past email:\n{retrieved}")
    }
]

response = completion(model="gpt-5", messages=messages)
print(response["choices"][0]["message"]["content"])
# Example output:
# "Sender: Alice Johnson.
#  Key requests: reset her account password and confirm contact information.
#  Context: this continues a previous unresolved login issue."

Keyword-based retrieval could be swapped in here using a library like whoosh or elasticsearch, returning the top BM25 match instead of computing embeddings.

Whichever method you choose, retrieval changes the system’s behavior: the quality of the output now depends both on what gets retrieved and on how the model uses it. These are issues we’ll return to when we discuss evaluation of multi-component pipelines.

Tools

Some tasks are better handled by software than by the model itself. Think about doing exact math, pulling information from a system of record, or reformatting something like a date or a phone number. In these cases, it is easier and more reliable to let the model call a tool instead of trying to produce the result on its own.

A tool is basically a Python function. We describe the function to the model, along with the arguments it expects, so the model knows how to invoke it. When the model decides to call the tool, our runtime executes the function and feeds the result back into the conversation.

In the email summarizer, suppose the sender includes a callback number like: “Call me at (415) 555-9021 ext. 7.” If you want to store this in your Customer Relationship Management (CRM) or pass it into an automated dialer, you’ll need the number in a standard format. Rather than relying on the model to rewrite phone numbers consistently, you can give it access to a normalize_phone tool that always produces the same output format.

from litellm import completion
import json
import phonenumbers

# Define the tool as a Python function
def normalize_phone(number: str):
    parsed = phonenumbers.parse(number, "US")
    return phonenumbers.format_number(
        parsed, phonenumbers.PhoneNumberFormat.E164
    )

# Tell the model what the tool is and how to call it
tools = [
    {
        "type": "function",
        "function": {
            "name": "normalize_phone",
            "description": "Convert a phone number to "
                           "standardized E.164 format",
            "parameters": {
                "type": "object",
                "properties": {"number": {"type": "string"}},
                "required": ["number"]
            }
        }
    }
]

messages = [
    {
        "role": "system",
        "content": "You summarize emails and may call tools when helpful."
    },
    {"role": "user", "content": f"Summarize this email:\n\n{email_text}"}
]

resp = completion(model="gpt-5", messages=messages, tools=tools)
msg = resp["choices"][0]["message"]

# If the model calls the tool, run it and send back the result
if "tool_calls" in msg:
    for call in msg["tool_calls"]:
        if call["function"]["name"] == "normalize_phone":
            args = json.loads(call["function"]["arguments"])
            result = normalize_phone(**args)
            messages.append({
                "role": "assistant",
                "tool_call_id": call["id"],
                "name": "normalize_phone",
                "content": json.dumps(result)
            })

# Continue the conversation until the model produces a final summary
resp2 = completion(model="gpt-5", messages=messages, tools=tools)
print(resp2["choices"][0]["message"]["content"])
# Example:
# "Sender: Alice Johnson. Key requests: reset her account password.
#  Preferred contact: +14155559021"

This approach is not limited to small utilities. In practice, applications often expose tools such as calculators, calendar lookups, search engines, or CRM APIs. These let the model handle parts of the task that require precision, data access, or integration with existing systems.

Warning

Tools give LLMs access to real actions. Always validate tool inputs and consider the security implications before allowing an LLM to call functions that modify data or external systems.

Note

Retrieval can also be seen as just another tool. Instead of returning a phone number, the function returns a set of documents given a query. We treat it separately here only because retrieval is such a common building block in modern applications.

A recent effort to make this easier is the Model Context Protocol (MCP) (https://modelcontextprotocol.io/docs/getting-started/intro). MCP defines a standard way for a tool server to describe available functions—​including their names, arguments, and authentication requirements—​and for a client to discover and call them. Instead of writing custom glue code for every service, you can point your application at an MCP server and the model immediately has a catalog of tools it can use.

Agents

With single calls, conversations, retrieval, and tools, we already have the main building blocks of an LLM application. An agent is simply a LLM that uses these components in a loop. The agent receives the current state, decides what action to take, executes it, and then observes the result. It repeats this cycle until it decides the task is complete.

In the email summarizer, an agent might start by reading the new message and deciding that more context is needed. It retrieves past correspondence from the same sender, looks at the result, and then chooses to call a tool to normalize a callback number. Only after those steps does it decide that it now has enough information to produce a final summary. What makes this an agent is not the specific sequence of actions, but the fact that the model is deciding them one by one as it goes.

# Minimal sketch of an agent loop
from litellm import completion

messages = [
    {
        "role": "system",
        "content": ("You are an agent that can retrieve past emails, "
                    "call tools, and summarize.")
    },
    {
        "role": "user",
        "content": f"Summarize this new email:\n\n{email_text}"
    }
]

for step in range(10):  # cap the number of steps
    resp = completion(model="gpt-5", messages=messages, tools=tools)
    msg = resp["choices"][0]["message"]

    # If the model requests a tool, run it and append the result
    if "tool_calls" in msg:
        # (execute retrieval or tool functions here)
        messages.append({
            "role": "assistant",
            "content": "Tool result goes here"
        })
        continue

    # If the model produces a final answer, stop
    if msg.get("content"):
        print("Final output:", msg["content"])
        break

Frameworks like LangChain, LlamaIndex, and AutoGPT wrap this pattern with additional features such as memory, planning strategies, and error handling. When reading blog posts or papers about agents, it is easy to feel overwhelmed with jargon and seemingly complex ideas. But the idea is simple: an agent is just repeated LLM calls that decide, step by step, whether to generate text, retrieve information, call a tool, or stop when the task is complete.

Note

Agent loops can, in theory, run indefinitely if not bounded. Always include a maximum step count (as shown in the example with range(10)) and consider implementing timeouts in production systems.

Fundamentals of Prompting

When you put together the first version of an application, you need a prompt that is specific enough to guide the model and consistent enough to give you predictable outputs. Over time you’ll refine it, but a strong starting point has a few recurring pieces. These reflect both our own experience, as well as the best practices you’ll find in other prompting guides.

Prompts for Single LLM Calls

A good first draft prompt usually includes:

Role. Tell the model who it is meant to be. This keeps it anchored in the right domain.

Objective. State clearly what the task is and what success looks like.

Instructions. Spell out do’s and don’ts. For example, if you need three bullet points, say so. If quoted text should be excluded, make that explicit.

Context. Provide the data the model should act on, like the email, the document, or any other text snippets. Without context, it can only guess.

Reasoning. For harder tasks, you can ask the model to think step by step, or explain how the model should “think” before returning the answer.

Output format. Describe how the answer should be structured so it’s easy to consume downstream.

Examples. Optional but powerful. A short input–output pair shows format, style, and level of detail.

Delimiters. Use section headers, code fences, or tags to keep instructions, context, and examples separate.

In practice, you typically organize these elements into a prompt template—a reusable structure with placeholders for variable content that will be filled in at runtime. For example, the [PASTE EMAIL TEXT HERE] placeholder would be replaced with the actual email text when the application runs. Using templates makes prompts easier to maintain, debug, and version control, since you can modify specific sections without rewriting the entire prompt each time.

The following example shows how the aforementioned elements come together in a prompt template for our email summarizer.

Example 2-1. Example: Prompt template for email summarizer

### Role
You are an assistant for our internal support team.

### Objective
Extract the sender’s name and summarize the key requests in an incoming email.

### Instructions
- Produce exactly three concise bullet points for the requests.
- Do not include quoted text from prior threads or signatures.
- Keep the tone plain and professional.

### Context
[PASTE EMAIL TEXT HERE]

### Reasoning
Think through the steps needed to extract the sender and requests,
but return only the final formatted output.

### Output format
- Sender: <name>
- Requests:
  - <bullet 1>
  - <bullet 2>
  - <bullet 3>

### Example
Input: "My account was locked after travel. Please reset it and call me at (415) 555-9021."
Output:
- Sender: Alice Johnson
- Requests:
  - Reset account access
  - Call back at (415) 555-9021

Another benefit of using a template is that it makes prompts easier to refine. Suppose you discover the model sometimes includes quoted email threads in its summaries, and you don’t want the summaries to include too much information from previous emails in a thread. With a structured template, you know exactly where to fix this; add a line in the instructions section that says, “Do not include quoted text from prior threads.” Or imagine you realize the summaries don’t always include a clear call to action for the support agent. You can add a new field in the format section, for example, “Next action for agent: <short instruction>.” Overall, instead of piling ad-hoc notes into one long block, the template keeps changes organized and easy to maintain.

Tip

Store your prompt templates as separate files or configuration rather than hardcoding them. This makes it easier to version, test, and iterate on prompts without changing application code.

Retrieval Fundamentals

When implementing retrieval for your application, we recommend starting with keyword search. It is fast to set up, easy to interpret, and often a very strong baseline. With keyword methods like BM25, you can see exactly why a document was retrieved, because the query terms overlap with it. This transparency makes debugging straightforward, and in many domains with stable vocabulary (like support tickets), keyword search performs surprisingly well.

Here is a toy example using the rank_bm25 library on a small list of past support emails:

from rank_bm25 import BM25Okapi
import numpy as np

# Small archive of past emails
past_emails = [
    "Ticket #4521: Please reset my password. I cannot log in.",
    "Ticket #4605: Could you verify my email address on file?",
    "Ticket #4710: I need help updating my phone number."
]

# Build BM25 index
tokenized_corpus = [doc.lower().split() for doc in past_emails]
bm25 = BM25Okapi(tokenized_corpus)

query = "I still can’t log in."
tokenized_query = query.lower().split()

# Score and select the best match
scores = bm25.get_scores(tokenized_query)
best_doc = past_emails[int(np.argmax(scores))]

print(best_doc)
# Example output:
# "Ticket #4521: Please reset my password. I cannot log in."

This is only a toy demonstration. In practice, your corpus may be thousands of emails or tickets stored in a database, and you would use the database’s own retrieval features or a dedicated search library to handle indexing and ranking. The important part is that you begin with a simple, interpretable baseline. From there, if you find it misses relevant cases, you can move on to more flexible or complex methods such as embedding search.

Tools and Agents

When you add tools to an application for the first time, the main step is to describe them clearly to the model. Most modern APIs expect a specification in the style of OpenAPI: a function name, a short description, and a JSON schema for its arguments and return type. It is worth being precise here: write descriptions that explain exactly what the function does, define arguments unambiguously, and think through edge cases. For example, if a number must be a string, say so. If the tool only works for U.S. phone numbers, make that explicit. The clearer the specification, the less room there is for the model to make mistakes while using the tool.

Here is an example tool spec for normalizing phone numbers:

{
  "type": "function",
  "function": {
    "name": "normalize_phone",
    "description": "Convert a phone number to standardized E.164 format",
    "parameters": {
      "type": "object",
      "properties": {
        "number": {
          "type": "string",
          "description": "Phone number to normalize, assumed to be US "
                         "if no country code is provided"
        }
      },
      "required": ["number"]
    }
  }
}

A first-pass tool prompt can reuse the same template we developed earlier (role, objective, context, instructions, format). The only addition is a section that lists the available tools and explains when they should be used. For example: “You summarize emails. You may call normalize_phone whenever a phone number is present. Always include the tool’s result in the final summary.” Combined with the tool spec, this is enough to get a simple tool-enabled pipeline running.

Once you have prompts, retrieval, and tools in place, you can combine them into an agent. An agent is simply a system that loops through the following cycle: the model reasons about the current state, decides whether to act (by calling a tool or retrieving context), observes the result, and continues until it produces a final answer. This pattern is often called ReAct (“Reason + Act”) Yao 2023. The following example shows a ReAct-style agent prompt for our email summarizer.

Example 2-2. Example: Agent prompt (ReAct-style) for email summarizer

### Role
You are an agent that helps summarize support emails.

### Objective
Read a new email, use retrieval and tools when helpful, and produce a final summary for the support team.

### Instructions
- You may retrieve past tickets if needed.
- You may call `normalize_phone` when a phone number is present.
- Work step by step. At each step, decide on one action
(retrieve, call a tool, or stop with a final answer).
- After you take an Action, you will receive an Observation,
with the result. Use it to decide your next step.
- End with a final answer in the specified format.
- If the user’s request is ambiguous, [choose one]:
**(a)** ask a clarifying question before proceeding;
**(b)** infer the most likely intent and continue.

### Reasoning (ReAct pattern)
Follow the Thought → Action → Observation cycle:
- **Thought:** (optional, for development) a brief statement of what you intend to do.
- **Action:** the tool to call, with JSON arguments.
- **Observation:** will be provided after each action.
- Repeat until you produce the final answer.

### Output format (for Final)
Final answer:
- Sender: <name>
- Requests:
  - <bullet 1>
  - <bullet 2>
  - <bullet 3>
- Phone: <normalized number if present>

### Step format example
Thought: I should check prior tickets for context.
Action: retrieve_tickets
Args: {"query": "can’t log in; password reset", "sender": "<sender_email>"}

Observation: [{"id":"4521","text":"Please reset my password. I cannot log in."}]

Thought: A phone number is present; I will normalize it.
Action: normalize_phone
Args: {"number": "(415) 555-9021 ext. 7"}

Observation: "+14155559021"

Thought: I have enough information to write the summary.
Final:
- Sender: Alice Johnson
- Requests:
  - Reset account access
  - Verify email on file
  - Call back at +14155559021

One important design choice in building an agent is how much freedom it should have. For example, if you want it to be cautious and avoid guessing on behalf of the user, include some instruction such as: “If anything is ambiguous, always ask the user to clarify before proceeding.” If you want it to be more autonomous, instead say: “If anything is ambiguous, infer the most likely intent and continue.” Making this decision explicit in the prompt will make the agent’s behavior more predictable and easier to evaluate later.

Evaluation Setups

There are a wide variety of evaluation setups described in the literature and used in practice. We can bucket them all into two types of high-level questions: “Is this system good enough to ship?” and “Which of these options works better?” The first question requires absolute evaluation, or judging whether your system meets some standard of correctness. The second requires comparative evaluation, or deciding which of several alternatives performs best.

Absolute Evaluation

This is where most teams start. You’ve built a system—​say, the email summarizer with its first draft prompt—​and you need to know if it’s good enough to use. That means defining what “good enough” actually means, then checking whether your outputs meet that bar.

In the literature, metrics for absolute evaluation are often grouped into one of the following:

Reference-based evaluation. You compare the model’s output against labeled data, often called “gold” summaries or ground truth. For the email summarizer, you’d prepare a dataset of emails with human-written summaries, then score each model output for exact match, overlap with the reference, or semantic similarity. This approach is appealing because it gives clear right-wrong comparisons. But it has two problems: labeled data is expensive to create, and many tasks have multiple valid outputs. One good summary might say “Reset password; confirm email on file” while an equally valid one says “Help me log in; verify my registered address.” Scoring against only one reference penalizes perfectly good outputs. We recommend reference-based methods only when there’s genuinely a single correct answer, like verifying whether a field was extracted correctly.

Reference-free evaluation. Here, you check the output against criteria you define, without needing a gold answer. For the email summarizer, you might verify that the output includes a “Sender:” field, produces exactly three bullet points, and avoids quoted text from previous threads. You might also check that it doesn’t invent details that aren’t explicitly mentioned in the email. Correctness is usually a set of criteria like this—​format, completeness, factuality—​rather than matching a single right answer. You can automate these checks with rules or use another model as a judge. Most teams don’t have labeled datasets for their specific application, so reference-free evaluation is usually the only practical option.

Note

This book focuses on reference-free evaluation methods. Most teams don’t have the time or resources to create labeled datasets for their specific application, so reference-free approaches are the practical starting point. In Chapter 3, we’ll show you how to systematically identify what to evaluate.

Comparative Evaluation

Once you have something working reasonably well, you might want to compare alternatives, such as different prompts, models, or retrieval methods. Now the question isn’t “is this correct enough?” but “which of these is better?” omparative evaluation typically takes one of these forms:

Pairwise comparison. For each input, you generate two outputs—​one from prompt A, and one from prompt B. An evaluator (human or model) picks which one better meets some criterion. For the email summarizer, the question might be: _"Which output more accurately captures the sender’s requests without adding extra details?”

Ranking and leaderboards. Instead of comparing just two outputs, evaluators rank three or more options for the same input. Aggregate these rankings across many inputs and you get system-level scores—a leaderboard. Public benchmarks like LMArena use this format to compare different AI models. Some companies maintain internal leaderboards to track which prompt, model, or configuration performs best.

Note

We won’t cover comparative evaluation in detail in this book. Most teams don’t run these regularly, because comparative evaluation is closer to A/B testing and only makes sense once you have a working system and multiple variants worth comparing. The real bottleneck is getting a single configuration to work reliably first, which is why we focus on absolute evaluation.

Putting It Together

You now have the building blocks for an LLM application: single-step calls, conversations, retrieval, tools, and agents. You understand the two main evaluation approaches: absolute evaluation (is this good enough?) and comparative evaluation (which is better?). You’ve seen how to structure prompts with templates.

When you run your prototype on real data, you’ll find problems. The model might interpret instructions differently across inputs. Retrieval might return irrelevant documents. Output quality might vary unpredictably. These problems are your starting point for evaluation.

The next chapter shows you how to analyze these problems systematically. You’ll learn how to collect representative examples, categorize different types of failures, and distinguish between two root causes: specification failures (where your prompt was unclear or incomplete) and generalization failures (where the model behaves inconsistently despite clear instructions). This systematic error analysis is the foundation for everything that follows in the book.

About the Authors

Shreya Shankar builds open source systems for AI-powered data processing. She is a final-year PhD at UC Berkeley. Shreya created DocETL, an open-source system for analyzing unstructured text at scale. DocETL has been deployed across journalism, law, medicine, policy, finance, and urban planning. Her research has been published at top computer science venues including VLDB, SIGMOD, and UIST (including a Best Paper award). Before her PhD, Shreya worked as a machine learning and data engineer at startups. She holds a BS in computer science from Stanford University.

Outside of research, Shreya cofounded the AI Evals for Engineers and PMs course with Hamel Husain, which has reached over 3,000 professionals from 500+ companies and maintains a mailing list of 25,000+ readers.

Hamel Husain is a machine learning engineer with 20 years’ experience, including roles at innovative companies like Airbnb and GitHub where he contributed to early LLM research used by OpenAI and led popular open-source ML tools. Currently an independent consultant, Hamel has extensive hands-on experience helping numerous companies (35+ implementations cited) build and evaluate real-world AI products using the techniques described in this book.