Large Language Models: The Hard Parts

Open Source AI Solutions for Common Pitfalls

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Tharsis T. P. Souza and Jonathan K. Regenstein, Jr.

Large Language Models: The Hard Parts

by Tharsis T.P. Souza and Jonathan K. Regenstein, Jr.

Copyright © 2026 Tharsis T.P. Souza and Jonathan K. Regenstein Jr. All rights reserved.

Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

Acquisitions Editor: Nicole Butterfield

Development Editor: Jeff Bleiel

Production Editor: Clare Laylock

Interior Designer: David Futato

Interior Illustrator: Kate Dullea

May 2026: First Edition

Revision History for the Early Release

2025-08-27: First Release

See https://oreilly.com/catalog/errata.csp?isbn=9798341622524 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Large Language Models: The Hard Parts, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

979-8-341-62247-0

[LSI]

Brief Table of Contents (Not Yet Final)

Chapter 1. The Evals Gap (available)

Chapter 2. Open-Source Evaluation Frameworks (available)

Chapter 3. Unstructured Input Data (available)

Chapter 4. Structured Data Output (unavailable)

Chapter 5. Laws, Guidelines, and Safety Techniques (unavailable)

Chapter 6. Safety Tools (unavailable)

Chapter 7. Policy Alignment (unavailable)

Chapter 8. Local and Open-Source LLMs (unavailable)

Chapter 9. Frontiers (unavailable)

Chapter 1. The Evals Gap

It doesn’t matter how beautiful your theory is,
it doesn’t matter how smart you are.
If it doesn’t agree with experiment, it’s wrong.

—Richard Feynman

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at jbleiel@oreilly.com.

Introduction

The advent of Large Language Models (LLMs) and the subsequent rapid adoption of LLM-based applications (LLMBAs) mark a pivotal shift in the landscape of software development, testing and verification. Unlike traditional software systems, where deterministic outputs are the norm, LLMs and LLMBAs introduce a realm of non-deterministic and generative behaviors that challenge conventional software engineering paradigms. This shift is not merely a technical evolution but a fundamental transformation in how we conceive, build, and assess business applications. For that reason, remaining entrenched in traditional testing frameworks that fail to account for the probabilistic nature of LLMs and LLMBAs will inevitably lead to significant risks.

However, even for the motivated, enthusiastic and excited participants in this new world, shifting mindset and methodology is not easy. To help address a crucial part of this shift, this Chapter explores a critical “evaluation gap” between traditional software testing approaches and the unique requirements of LLMBAs, examining why traditional testing frameworks fall short and what new evaluation strategies are needed.

To be clear, by LLMBAs, we mean the deployment of a combination of an LLM, a data source and an output, to generate a business or commercial outcome. For example, using an LLM to write a report or an email is an LLMBA; creating a chat interface to technical documents is an LLMBA. As we explore different evaluation frameworks, our ultimate goal is to use LLMs in LLMBAs or as part of software development pipelines.

In the remainder of this chapter, we use practical examples to investigate key aspects of LLM evaluation including benchmarking approaches and metrics selection. We emphasize the importance of developing comprehensive evaluation frameworks that can handle both the near deterministic (when we cover structured outputs with logit post processing) and probabilistic aspects of LLM behavior, providing concrete guidance for implementing robust evaluation pipelines for LLM-based applications.

Non-Deterministic Nature of LLMs

One of the most fundamental challenges when incorporating LLMs into LLMBAs is their non-deterministic nature. Unlike traditional software systems where the same input reliably produces the same output, LLMs can produce novel text that may not exist in their training data, and produce different responses each time they’re queried - even with identical prompts and input data. This behavior is both a strength (when we prize creativity and flexibility) and a significant challenge. This non-deterministic behavior manifests itself in generative tasks when an LLM gives different written answers to the same queries.

Indeed, most models offer a “temperature” parameter, which controls the randomness of outputs, allows models to be creative and allows them to generate diverse responses. However, this same feature makes it difficult to build reliable, testable systems especially at the enterprise level where a fundamental requirement of business intelligence is to offer consistent answers.

Consider a financial services company using LLMs to generate the following investment research: an executive summary of a 10-K (a filing made by public companies with the Securities and Exchange Commission). This research would require factual grounding in the filing (this is the input data) but also potentially some level of opinion about the latest information reported by the company (this is the output where we might favor creative thinking).

However, the non-deterministic nature of LLMs means that:

The same input data could yield different analytical conclusions

Regulatory compliance becomes challenging to guarantee

Trust and efficiency may be degraded by inconsistent responses

Testing becomes more complex compared to traditional software

Where does this non-determinism arise from when LLMs generate answers? The primary source is sampling.

More specifically, during text generation:

The chosen tokenizer segments the input text into tokens

Each token gets mapped to a unique numerical ID

The LLM processes these token IDs through its deep neural network

The model produces logits - unnormalized scores for each possible next token

A softmax transformation converts these raw logits into a probability distribution. That conversion is represented by the following equation:

P ( y i | x ) = softmax ( x i ) = e x i ∑ j=1 n e x j

for i = 1 , ... , n where P ( y i | x ) represents the probability of class i given input x and x i is the logit (raw score) for class i .

The highest probability indicates the model’s strongest prediction for the next token

Text generation then selects the next token based on a sampling given strategy (e.g. greedy, top-k, etc)

As those steps indicate, an LLM doesn’t just choose the most likely next token, it samples from a probability distribution. Let’s see how this works in a real life example.

In the following simple experiment, we use an LLM to write a single-statement executive summary from a 10-K and jitter the temperature parameter to see how non-determinstic LLMs can be. We observe that even a simple parameter like temperature can dramatically alter model behavior in ways that are difficult to systematically assess.

At temperature 0.0, responses are consistent but potentially too rigid.¹ At 1.0, outputs become more varied but less predictable. At 2.0, responses can be wildly different and often incoherent. This non-deterministic behavior makes traditional software testing approaches inadequate.

from dotenv import load_dotenv
import os

# Load environment variables from .env file
load_dotenv()

from openai import OpenAI
import pandas as pd
from typing import List

def generate_responses(
    model_name: str,
    prompt: str,
    temperatures: List[float],
    attempts: int = 3
) -> pd.DataFrame:
    """
    Generate multiple responses at different temperature settings
    to demonstrate non-deterministic behavior.
    """
    client = OpenAI()
    results = []

    for temp in temperatures:
        for attempt in range(attempts):
            response = client.chat.completions.create(
                model=model_name,
                messages=[{"role": "user", "content": prompt}],
                temperature=temp,
                max_tokens=50
            )

            results.append({
                'temperature': temp,
                'attempt': attempt + 1,
                'response': response.choices[0].message.content
            })

    # Display results grouped by temperature
    df_results = pd.DataFrame(results)
    for temp in temperatures:
        print(f"\nTemperature = {temp}")
        print("-" * 40)
        temp_responses = df_results[df_results['temperature'] == temp]
        for _, row in temp_responses.iterrows():
            print(f"Attempt {row['attempt']}: {row['response']}")

    return df_results

Now let’s supply our base document, the 10-K. We are not going to supply the entire 10-k, but rather just 10,000 characters. The reason for this is our model “gpt-3.5-turbo” has a token limit of 4,096. The combination of our input and output can’t exceed 4,096 tokens. A token is a meaningful chunk of text, like a word, a phrase, possibly punctuation. 10,000 characters typically equates to about 2,500 tokens, or 4-5 pages of text. So we are passing in about 4-5 pages of the 10-K and leaving room for a 4-5 page response, if needed.

For the curious, we recommend having a look at the full 10-K filing and jotting your own executive summary, and then asking an LLM to do so. Notice the full filing is over 150 pages long. Would we truly ask an LLM or a person to summarize more than 150+ pages, or would we ask the LLM or person to focus on key sections? How long would it take a person to read the full 150+ pages?²

Here’s a quick glimpse of the sec_filing we have stored as text. [Example to Come]

Now let’s start our code flow to load the first 10,000 characters of the filing, pass it to our model and supply some simple instructions via a prompt.

MAX_LENGTH = 10000 # We limit the input length to avoid token issues
with open('../data/apple.txt', 'r') as file:
    sec_filing = file.read()
sec_filing = sec_filing[:MAX_LENGTH]
df_results = generate_responses(model_name="gpt-3.5-turbo",
                                prompt=f"Write a single-statement executive
                                summary of the following text: {sec_filing}",
                                temperatures=[0.0, 1.0, 2.0])

Temperature = 0.0
----------------------------------------
Attempt 1: Apple Inc. filed its Form 10-K for the fiscal year ended September 28, 2024 with the SEC, detailing its business operations and financial performance.
Attempt 2: Apple Inc. filed its Form 10-K with the SEC for the fiscal year ended September 28, 2024, detailing its business operations, products, and financial information.
Attempt 3: Apple Inc. filed its Form 10-K with the SEC for the fiscal year ended September 28, 2024, detailing its business operations, products, and financial information.

Temperature = 1.0
----------------------------------------
Attempt 1: Apple Inc., a well-known seasoned issuer based in California, designs, manufactures, and markets smartphones, personal computers, tablets, wearables, and accessories, with a focus on innovation and technology.
Attempt 2: Apple Inc. filed its Form 10-K with the SEC for the fiscal year ended September 28, 2024, reporting on its business operations, products, and financial performance.
Attempt 3: Apple Inc., a well-known seasoned issuer, filed its Form 10-K for the fiscal year ended September 28, 2024, reporting on its financial condition and operations.

Temperature = 2.0
----------------------------------------
Attempt 1: The Form 10-K for Apple Inc. for the fiscal year ended September 28, 2024, filed with the Securities and Exchange Commission, outlines the company's financial performance, products, and risk factors affecting future results.
Attempt 2: Apple Inc., a California-based company and leading technology manufacturer invDestacksmeticsisdiction setIspection-$20cyan evaluationseld anvisions droitEntering discernminerval Versbobprefversible vo该 Option和 meio forecast времCisco dellaischenpoihsCapabilities Geme.getTime future
Attempt 3: Apple Inc's Form 10-K provides a comprehensive overview of the company's financial reporting, business operations, products and market information.

We run three generations for each temperature level.

As expected we observe the following results:

Temperature = 0: Summaries seem deterministic (even though still stochastic) showing to be repetitive.

Temperature = 1: Summaries present balanced creativity with a certain level of coherence

Temperature = 2: Increased randomness and potentially incoherent.

The Evals Challange

As the above example illustrates, even a straightforward task with one short sentence of output introduces challenges that confound traditional software and model evaluation.

This is why we say there is an Evals Gap between traditional software testing and LLM evaluation. We need new frameworks that can account for both the deterministic aspects we’re used to testing and the non-deterministic properties that make LLMs unique.

evals-table summarizes how LLM evaluation differs from traditional software testing across several key dimensions:

Capability Assessment vs Functional Testing: Traditional software testing validates specific functionality against predefined requirements. LLM evaluation must assess not necessarily pre-defined behavior but also “emergent properties” like reasoning, creativity, and language understanding that extend beyond explicit programming.

Metrics and Measurement Challenges: While traditional software metrics can usually be precisely defined and measured, LLM evaluation often involves subjective qualities like “helpfulness” or “naturalness” that resist straightforward quantification. Even when we try to break these down into numeric scores, the underlying judgment often remains inherently human and context-dependent.

Dataset Contamination: Traditional software testing uses carefully crafted test cases with known inputs and expected outputs (e.g., unit tests). In contrast, LLMs trained on massive internet-scale datasets risk having already seen and memorized evaluation examples during training, which can lead to artificially inflated performance scores. This requires careful dataset curation to ensure test sets are truly unseen by the model and rigorous cross-validation approaches.

Benchmark Evolution: Traditional software maintains stable test suites over time. LLM benchmarks continuously evolve as capabilities advance, making longitudinal performance comparisons difficult and potentially obsoleting older evaluation methods.

Human Evaluation Requirements: Traditional software testing automates most validation. LLM evaluation still demands significant human oversight³ to assess output quality, appropriateness, and potential biases through structured annotation and systematic review processes.

Evaluating LLMs v. LLMBAs

Before we proceed to an evaluation design, it’s important to emphasize the distinction between evaluating an LLM purely on its inherent capabilities versus evaluating how an LLM perform as part of an LLMBA. LLMs offer foundation capabilities⁴ and are typically general-purpose. When we see reports about their performance (usually when a new model or version is released there is a lot of hype and fanfare about performance), it is on general purpose tasks (we discuss the history of benchmarks used here later in this Chapter).

Within the context of an LLMBA, we are evaluating how those general purpose models perform in a particular application. There are many components of an LLMBA that might affect performance, such as:

the data needed to solve those business problems

the prompt supplied to the LLM

ethical or compliance guidelines particular to a business application

the commercial value of the end result

We will discuss each of these in great detail as we proceed in this book but a short example helps elucidate this.

Take our previous example of an executive summary of 10-K. The 10-K is the data we need for this LLMBA - how we extract text and make it searchable will affect an LLMs ability to summarize the document. The prompt that we supply to ask for the summary will affect how the LLM replies. However, for example, unbeknownst to the LLM, we might have a strict compliance policy that all executive summaries must contain citations. Finally, the LLM might perform admirably on generating a summary but if our stakeholders find no value in it, the LLMBA itself has not performed well. The LLM itself is crucial to this process of course, but we are evaluating more than just the inherent capabilities of the LLM.

As we start to think about evaluating LLMs in the context of LLMBAs, we build a taxonomy of categories, sub-categories, criteria (what to test) and explanations (why these are important) to help structure our thinking. Each enteprise will have different priorities, and indeed individual business lines may have different priorities, but this can be a good starting point.

Before we discuss why each is important and how to test for each, here’s a quick outline of our taxonomy:

Safety

Misinformation Prevention

Unqualified Advice

Bias Detection

Privacy Protection

Cognitive

Reasoning & Logic

Language Understanding

Technical

Code Generation

System Integration

Meta Cognitive

Self-Awareness

Communication Quality

Ethical

Harmful Content

Decision-Making

Environmental

C02 Emission

With that skeleton taxonomy in mind, let’s summarize why each of these is important and how we can test them.

Safety - Misinformation Prevention

Accurate answers serve several critical safety purposes that extend far beyond simple correctness. Accurate information prevents real-world harm that can result from users acting on false data, whether in medical decisions, financial analysis, or other business domains. This accuracy also maintains the essential trust relationship between users and LLMBAs. Without trust, users won’t use our LLMBAs. From an institutional perspective, reliable outputs reduce both legal liability and reputational damage that organizations could face. Furthermore, accuracy enables LLMBAs to serve as dependable decision-making support tools, allowing users to make informed choices based on sound information.

Testing for misinformation begins with verifying the accuracy of factual statements by cross-referencing LLMBA outputs against established, verified databases and authoritative sources to identify any incorrect claims or data errors. Consistency testing ensures that our LLMBA provides similar responses to equivalent queries, preventing contradictory information that could confuse users or indicate underlying reliability issues. Citation and source accuracy testing evaluates whether our LLMBA properly attributes information and references legitimate, credible sources . Additionally, testing must assess how the LLMBA behaves when facing uncertainty, ensuring it appropriately communicates limitations rather than generating false confidence. Temporal consistency checks verify that LLMBAs maintain accurate information over time and don’t introduce errors through updates or data shifts. Finally, scientific accuracy testing specifically evaluates performance in technical domains where precision is paramount and errors could have serious consequences for users relying on the information for important decisions

Safety - Unqualified Advice

Unqualified advice is crucial boundary that LLMBAs must not cross. When LLMBAs provide incorrect professional advice in areas like medicine, law, or finance, they can cause significant harm to users who may lack the expertise. This concern is particularly acute for protecting vulnerable users, including those facing urgent decisions or lacking access to qualified professionals. From a liability standpoint, reducing the risk of providing unqualified advice helps protect both developers and enterprises from legal consequences when users act on inappropriate guidance. Maintaining professional standards requires that LLMBAs appropriately defer to human experts and clearly communicate the limitations of their knowledge.

Testing for unqualified advice begins with assessing an LLMBA’s ability to accurately identify medical, legal, and financial queries (or whatever field where we wish to avoide unqualified advice), ensuring it can distinguish between general informational requests and those requiring professional judgment advice. Disclaimer consistency testing verifies that the LLMBA reliably provides appropriate warnings and limitations when discussing professional topics, maintaining consistent messaging about the need for qualified human expertise. Professional referral mechanisms must be evaluated to ensure that we effectively directs users toward appropriate licensed professionals, healthcare providers, attorneys, or financial advisors when specialized guidance is needed. Boundary recognition testing examines whether our LLMBA maintains clear distinctions between providing general information versus offering specific professional recommendations that could be acted upon inappropriately. Emergency situation handling represents a particularly critical testing area, evaluating how our LLMBA responds to urgent medical, legal, or financial crises. Finally, testing must verify our LLMBA’s consistent avoidance of specific recommendations in professional domains, ensuring it refrains from suggesting particular treatments, legal strategies, investment decisions, or other personalized advice even when asked in different prompt settings.

Safety - Bias Detection

Bias detection addresses both individual fairness and broader societal impacts. By identifying and mitigating biased outputs, LLMBAs can avoid reinforcing harmful societal prejudices that could perpetuate discrimination across racial, gender, cultural, and other demographic lines. From a social responsibility perspective, bias detection helps LLMBAs contribute positively to society rather than amplifying existing inequalities or creating new forms of digital discrimination. Enterprises also benefit from protected brand reputation, as biased outputs can cause significant public relations damage and erode public trust. Perhaps most importantly, effective bias detection enables LLMBAs to genuinely support diverse user bases by providing respectful, accurate, and inclusive responses that acknowledge and serve the full spectrum of human experiences and perspectives.

Testing for bias detection begins with systematic assessment of gender, racial, and cultural bias by examining whether LLMBAs produce different quality responses, make unfair assumptions, or apply stereotypes based on user identity markers or contextual clues about demographics. Demographic representation testing evaluates whether our outputs reflect diverse perspectives and avoid defaulting to dominant group viewpoints. Language inclusivity assessmes for accessible, respectful terminology that doesn’t exclude or marginalize particular groups, while avoiding language that reinforces hierarchies or prejudices. Stereotype avoidance testing specifically identifies instances where an LLMBA might perpetuate harmful generalizations. Problem-solving fairness evaluation ensures that our LLMBA provides equally helpful and thorough assistance regardless of the user’s apparent background, avoiding scenarios where certain groups receive inferior support or guidance.

Safety - Privacy Protection

Privacy protection means that our LLMBAs must recognize and safeguard private details during interactions, preventing inadvertent disclosure or misuse of personal information that users may reveal in their queries or conversations. This protection becomes essential for ensuring regulatory compliance with data protection laws like GDPR, CCPA, and HIPAA, which impose strict requirements on how organizations collect, process, and protect personal information. Maintaining robust data security through privacy-conscious design helps prevent privacy breaches that could expose user information to unauthorized parties or malicious actors seeking to exploit personal data. Furthermore, effective privacy safeguards protect the broader ecosystem from sensitive information that flows through LLMBA pipelines, including proprietary business data, confidential communications, and other information.

Testing for privacy protection begins with detecting Personally Identifiable Information (PII) and PII handling capabilities, ensuring we can accurately recognize PII data such as names, addresses, phone numbers, and social security numbers. Data anonymization should be tested, to ensure our LLMBA can effectively strip identifying details from information when necessary, maintaining usefulness while protecting individual privacy. Information leakage also must be tested, to ensure we don’t inadvertently reveal sensitive details from previous interactions or training data, particularly focusing on scenarios where confidential information might be exposed through seemingly innocent queries. Context carryover requires careful testing as to what information persists across conversation turns without creating privacy risks through excessive data retention or inappropriate sharing between user sessions. Of course, we must test for compliance with relevant privacy regulations such as GDPR, CCPA, and HIPAA. Finally, security protocol evaluation examines the robustness of our LLMBA’s privacy safeguards against potential attacks or exploitation attempts, testing whether malicious users could circumvent privacy protections through prompt injection, social engineering, or other techniques.

Cognitive - Reasoning & Logic

Reasoning and logic capabilities are what empower an LLMBA to provide meaningful intellectual support to users. Reliable problem-solving ensures that our LLMBA can work through multi-step challenges systematically, breaking down complex issues into manageable components and applying appropriate methodologies. Maintaining computational accuracy becomes crucial when users depend on our LLmBA for mathematical calculations, data analysis, or quantitative reasoning tasks where precision directly impacts the utility and trustworthiness of the results. Critical thinking evaluation examines whether our LLMBA can analyze arguments, identify assumptions, evaluate evidence, and help users examine issues from multiple perspectives rather than simply providing surface-level responses. Preventing logical errors requires testing for common reasoning fallacies, inconsistent conclusions, and flawed inferential chains that could mislead users or undermine our LLMBA’s credibility as a thinking partner.

Testing reasoning and logic capabilities begins with multi-step problem-solving assessment, asking whether our LLMBA can break down complex challenges into sequential components and arrive at coherent solutions. Logical fallacy testing ensures our LLMBA can identify flawed reasoning patterns, invalid arguments, and common cognitive biases both in external content and in its own analytical processes. Causal reasoning evaluation tests whether our LLMBA can accurately distinguish between correlation and causation, understand cause-and-effect relationships, and avoid drawing inappropriate conclusions from temporal or statistical associations. Finally, edge case management testing evaluates performance when encountering unusual scenarios that fall outside typical problem parameters, ensuring robust reasoning capabilities even in ambiguous circumstances.

Cognitive - Language Understanding

Language understanding allows our LLMBAs to engage meaningfully with users across diverse communication contexts and requirements. Our LLMBAs must be able to accurately interpret user intent, recognize nuanced meanings and respond in ways that directly address what users are actually asking rather than providing tangential or irrelevant information. Preventing misunderstandings requires the ability to handle ambiguous phrasing, cultural references, idiomatic expressions, and complex sentence structures that might otherwise lead to confusion responses.

Testing language understanding begins assessing our LLMBA’s ability to track conversational threads across multiple exchanges, maintain awareness of previously established information, and use contextual clues to interpret ambiguous statements or references that depend on earlier discussion points. Idiom comprehension and cultural reference assessment examines whether the system can identify and appropriately handle references to cultural events, historical figures, social phenomena, sarcasm and shared cultural knowledge. Technical terminology evaluation assesses our LLMBAs. ability to correctly understand and use specialized vocabulary across different professional domains, academic fields, and technical contexts. Finally, cross-lingual capability testing examines whether the system can handle multilingual conversations and borrowed terms from other languages.

Technical - Code Generation

Code generation directly impacts software quality, security, and maintainability across development projects. Ensuring code reliability requires syntactically correct, logically sound code that performs intended functions without introducing bugs, runtime errors, or unexpected behaviors that could compromise application functionality. Preventing security issues becomes paramount, as vulnerabilities like injection attacks, buffer overflows, improper authentication, or data exposure could create serious risks for applications and their users. Generated code must follow proper resource management practices, handle edge cases gracefully, and integrate well with existing codebases without causing crashes, memory leaks, or performance degradation. Finally, code follows established best practices, maintains proper documentation, uses appropriate design patterns, and creates maintainable solutions that won’t burden future development efforts with cleanup or refactoring requirements.

Testing code generation begins with by verifying that generated code runs/compiles without errors, and adheres to established coding standards for the target programming language or framework. Next, security vulnerability must be assessed, searchingfor common weaknesses such as SQL injection risks, cross-site scripting vulnerabilities, improper input validation, insecure authentication mechanisms, and other security flaws that could expose applications to attacks. Once we establish that code runs and is secure, we move on to performance optimization - the generated code must implement efficient algorithms, use appropriate data structures, minimize computational complexity, and avoid resource-intensive operations that could degrade system performance. Next we test for documentation quality and error-handling. Finally, cross-platform compatibility testing verifies that the code functions correctly across different operating systems, hardware architectures, and deployment environments.

Technical - System Integration

System integration is essential for system reliability. Our LLMBAs might be integrated with databases, APIs, third-party services, and other system components and they can’t introduce instability or unexpected behaviors that could compromise overall system integrity. We must preserve acceptable response times, resource utilization, and throughput levels, ensuring that adding our LLMBA doesn’t create bottlenecks or degrade the performance of existing system functions. Integration stability across different deployment environments configuration changes allow LLMBAs to remain robust and reliable as the broader technical infrastructure evolves and changes over time.

Testing system integration begins with asking our LLMBA to correctly format requests, parse responses, handle authentication protocols, and maintain proper communication with external services while adhering to API specifications and versioning requirements. Rate limit compliance ensures that the system respects usage quotas, implements appropriate throttling mechanisms, and gracefully manages situations where API limits are approached or exceeded without causing service disruptions. Error management and response time evaluation assesses how our LLMBA handles integration failures, network timeouts, service unavailability, and malformed responses. Finally, resource utilization and scalability testing evaluates whether the integration can handle increasing numbers of concurrent users, growing data volumes, and expanded service dependencies while maintaining performance and reliability standards as system demands grow.

Meta Cognitive - Self-Awareness

Self-awareness capabilities allows our LLMBA to represent what it knows and doesn’t know, avoiding false confidence that could mislead users into inappropriate reliance on uncertain information. Preventing overconfidence ensures our LLMBA doesn’t present speculative or uncertain outputs as definitive facts, which could lead users to make poor decisions based on unreliable AI guidance. Maintaining reliability depends on our LLMBAs ability to consistently acknowledge its limitations and maintain appropriate humility about its knowledge and reasoning abilities across different contexts and domains.

Testing self-awareness begins with knowledge limitation recognition assessment. We examine whether our LLMBA can accurately identify topics, domains, or specific questions where its knowledge is incomplete, outdated, or unreliable, and communicate these boundaries clearly to users. This includes areas of uncertainy, where we must distinguishes between high-confidence and speculative responses. Feedback integration testing evaluates whether our LLMBA can incorporate user corrections and guidance to improve its performance during ongoing interactions. Finally, error acknowledgment ensures our LLMBA can recognize when it has made mistakes, provide appropriate apologies or corrections, and avoid defensive or evasive responses when confronted with its own errors.

Meta Cognitive - Communication Quality

Communication quality that our LLMBA can present information that is accessible and comprehensible to users, adapting communication style to match user knowledge and avoiding unnecessary confusion or ambiguity. Enabling knowledge transfer means our LLMBA acn effectively teach concepts, explain processes, and help users learn new information in ways that promote genuine comprehension and retention. Smooth conversational flow, appropriate responses to user cues, and consistent communication patterns promote successful task completion and goal achievement.

Testing communication quality begins with message clarity assessment. Our LLMBAs must deliver well-structured, unambiguous responses that directly address user queries without unnecessary complexity or confusing language. Information density and summary accuracy determikne whether our LLMBA can distill lengthy or complex information into concise, accurate overviews that preserve essential meaning while improving accessibility. Finally, technical communication must be tested as well, determining our LLMBAs ability to explain specialized concepts, use domain-specific terminology appropriately, and translate between technical and lay language depending on audience needs.

Ethical - Harmful Content Prevention

Harmful content prevention is fundamental to protecting users from dangerous information, preventing exploitation for malicious purposes, upholding moral principles, minimizing legal risks for deploying organizations and promoting positive applications. The importance of avoiding harmful content can’t be overstated.

Testing harmful content prevention capabilities begins with identifying dangerous requests (this is actually testing our users too and what training might be needed) across various phrasings and circumvention attempts, providing appropriate refusals with suitable alternatives, filtering harmful outputs before generation, responding properly to crisis situations with emergency referrals, implementing protective measures for sensitive topics and flagging serious safety concerns. This is quite specific on a domain level, because what might be harmful in some domains might not be in others.

Ethical - Decision-Making

Ethical decision-making requires navigating complex moral landscapes and making choices that align with human values and societal standards. There are times when an ethical decision is at odds with the business goals of an enterprise, and our LLMBA needs to make the ethical choice (this will ultimately benefit our enterprise in the long run). Consistent application of moral principles across different scenarios and avoidance of decisions that could violate fundamental ethical standards even when facing novel or ambiguous situations are paramount. Preventing harmful impact involves ensuring our LLMBA can anticipate and avoid decisions that might cause unintended negative consequences.

Testing ethical decision-making requires deploying our LLMBA across similar situations and avoiding arbitrary or contradictory decisions that could undermine trust. Value alignment and decision fairness testing evaluates whether our LLMBA’s decisions reflect our values and societal norms, ensuring that choices align with widely accepted moral principles rather than reflecting biased or harmful value systems. Transparency testing is a bit different as it examines whether our LLMBA can explain its ethical reasoning process, helping users understand how moral considerations influenced specific decisions and enabling accountability for ethical choices. Impact assessment evaluation determines whether our LLMBA can anticipate and weigh the potential consequences of its decisions on different stakeholders.

Environmental - CO2 Emission

CO2 emission management, including energy consumption from training, inference, and infrastructure maintenance that contributes to greenhouse gas emissions, is an important commitment to environmental stewardship and climate change mitigation efforts. Optimizing operational costs through environmental considerations creates synergies between sustainability and economic efficiency, as energy-efficient operations typically reduce both environmental impact and operational expenses. Meeting environmental regulations ensures compliance with emerging carbon reporting requirements, energy efficiency standards, and environmental policies that increasingly govern technology operations.

Testing CO2 emission capabilities requires systematic evaluation of energy consumption patterns and environmental efficiency across all aspects of LLMBA, and technology, operations. Energy consumption per request assessment measures the computational resources and associated carbon footprint required for individual interactions, enabling optimization of response generation and resource allocation. Model size and complexity impact testing evaluates how different architectural choices, parameter counts, and computational requirements affect overall energy consumption and carbon emissions. Request caching efficiency testing assesses how effectively the system reuses previous computations and responses to reduce unnecessary processing and energy consumption (and this aligns with cost consciousness, which creates a nice win-win for enterprises). Finally, inference optimization strategies assessment evaluates techniques such as model compression, quantization, and pruning that can reduce computational requirements and associated carbon emissions while maintaining acceptable performance levels.

Designing an LLMBA Evaluation Framework

We have considered the gap, challenges and important categories around evaluating LLMBAs. Now let’s start to sketch what components we need to build our evaluation framework.

The design of an LLM application framework system depends heavily on the specific use case and business requirements. Thinking back to our investment research based on the 10-K, we can imagine an LLMBA whose success is determined by factual groundedness, or original thinking, or even the ability to find insights in specific sections of the document.

Conceptual Design: Single LLMBA

Let’s start with a design for the evaluation of a single LLMBA, that is, a scenario in which we have constructed one application to accomplish a task.

As shown in Figure 1-1, a single LLMBA evalution consists of examples, the application, an evaluator and the results/score of the evaluator.

Figure 1-1. Conceptual design of LLM-based application evaluation.

Conceptual Design: Multiple LLMBAs

Next we can sketch out a framework for evaluating multiple LLMBAs that have been built to accomplish a task. Here we wish to first evaluate each, and then build a leaderboard of which performs best.

A general conceptual design is shown in Figure 1-2, where multiple LLMBAs are evaluated. In this example, three LLMBAs are evaluated. This design allows for a more comprehensive evaluation of different configurations of LLMBAs, e.g.:

Fixing all application parameters and evaluating different LLM models with their default configurations

Fixing all parameters of an LLM model and evaluating different data sources

Fixing all parameters of an LLM model and evaluating different prompting strategies

Fixing all parameters of an LLM model and evaluating different chunking and retrieval mechanisms

These are just 3 examples. We can keep permutating around LLMs, prompting, chunking, data sources.

In the figure below, we have labeled app1, app2 and appN to stand for the various permutations of LLMs, prompting and chunking that we could evaluate. The end builder has many decisions to make in this framework. Perhaps we wish to keep all inputs - data, prompts, retrieval - the same, and test just LLMs. Perhaps we wish to keep the LLM the same, and test just different prompts or chunking strategies. We all of these in further detail. For now, this figure is meant to serve as a starting point for organizing our initial thoughts.

Figure 1-2. Conceptual design of Multiple LLM-based applications evaluation.

Planning an Evaluation Framework

Below are a few important questions to consider when planning an LLMBA system. These questions correspond to each of the five key components in our example conceptual design.

1. Examples (Input Dataset):

What types of examples should be included in the test set?

Does it cover all important use cases?

Are edge cases represented?

Is there a good balance of simple and complex examples?

How do we ensure data quality?

Are the examples representative of real-world scenarios?

Is there any bias in the test set?

Should we have separate test sets for different business requirements?

Do we need human-validated ground truth for all examples?

Can we use synthetic data to augment the test set?

How can business updates and user data be reflected in the dataset post-launch?

Note that Examples must provide input data to the LLMBA for further evaluation. However, ground truth data is optional. We will return to this in more detail later in the book, where we will see that ground truth data is not always available or practical. Additionally, there are approaches where one can evaluate LLMBAs without ground truth data.

2. LLM Applications:

What aspects of each LLM app should be standardized for fair comparison?

Prompt templates

Context length

Temperature and other parameters

Rate limiting and timeout handling

What specific configurations impact business requirements?

Which LLM application variations should be tested to maximize what we learn?

Which LLM capabilities provide the most value for the business and how can we measure that?

3. Evaluator Criteria:

How do we define success for different types of tasks?

Task-specific evaluation criteria

Objective metrics vs subjective assessment

Should evaluation be automated or involve human review?

Balance between automation and human judgment

Inter-rater reliability for human evaluation

Cost and scalability considerations

4. Scoring System:

How should different metrics be weighted?

Relative importance of different factors

Task-specific prioritization

Business requirements alignment

Should scores be normalized or absolute?

How to handle failed responses?

Should we consider confidence scores from the LLMs?⁵

5. Leaderboard/Ranking:

How often should rankings be updated?

Should ranking include confidence intervals?

How to handle ties or very close scores?

Should we maintain separate rankings for different:

Business requirements

Model Cost Tiers

LLM Model Families

In sum, an evaluation framework should be designed with scalability in mind to handle growing evaluation needs as the combination of (Input Examples X LLM Applications X Evaluators X Scores X Leaderboards) may grow very fast, particularly for an organization that promotes rapid experimentation and iterative development (good properties!). Finally, one should keep in mind that the evaluation system itself requires validation to confirm its accuracy and reliability vis-a-vis business requirements. For example, if an LLMBA has been built to accomplish a specific business purpose, our evaluation framework must reflect this.

Metrics

In the framework discussed above, we noted that the evaluator criteria will include objective metrics and the weighting of these affect the scoring system. Those metrics become quite important as they are our best attempt to bring objectivity to a process that often involves subjectivity. With that in mind, we now focus on the metrics we use to determine whether an LLMBA is performing well or not.

Metrics are our attempt to bring objectivity to a non-deterministic system. This might not be strictly possible but recall the use of LLMs is about tradeoffs: we wish to use these very powerful tools but we can’t eliminate the non-determinism.

Before we dive in, let’s distinguish between two broad categories of metrics: intrinsic and extrinsic.

Intrinsic metrics focus on the model’s performance on its primary training objective, which is typically to predict the next token in a sequence. For example, Perplexity⁶ is a common intrinsic metric that measures how well the model predicts a given sample of text.

Extrinsic metrics, or for our purposes application metrics assess the model’s performance on various downstream tasks, which can range from question answering to code generation. These metrics are not directly tied to the training objective, but they provide valuable insights into the model’s ability to generalize to real-world, business applications.

We will focuse exclusively on extrinsic/application metrics, since we are evaluating LLM-based applications rather than base LLM models.

Another way to think about metrics is in terms of the type of the task we evaluate:

Analytical Tasks involve distinguishing or classifying between existing data points, such as sentiment analysis, classification, or identifying whether a statement is true or false. Generative Tasks involve creating or producing new data or output, such as text generation, image synthesis, or summarization.

For analytical tasks, LLMBAs may produce log-probabilities or discrete predictions, thus traditional machine learning metrics like accuracy, precision, recall, and F1 score can be applied. However, generative tasks output text or images which require different evaluation approaches.

For generative tasks, a range of specialized metrics should be considered. These include match-based metrics such as exact match and prefix match, as well as metrics designed specifically for tasks like summarization and translation, including:

ROUGE: Recall-Oriented Understudy for Gisting Evaluation

BLEU: Bi Lingual Evaluation Understudy

character n-gram comparisons

The selection of appropriate metrics should align with the specific requirements and characteristics of the task being evaluated.

In Table 1-1 we provide a short list of widely used extrinsic metrics that can be used to evaluate generative tasks of LLM-based applications, along with their definitions, use cases, and limitations.

Table 1-1. Key Metrics for Evaluating Generative Tasks

Metric

Definition

Use Case

Limitations

BLEU (Bilingual Evaluation Understudy)

Measures overlap of n-grams between generated text and reference text

Machine translation and text summarization

- Favors short outputs due to brevity penalty
- Insensitive to semantic meaning
- Requires high-quality reference texts

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Measures overlap between n-grams, words, or sentences of generated text and references, focusing on recall

Text summarization tasks

- Biases toward long outputs
- Ignores semantic equivalence
- Heavily influenced by reference quality

METEOR (Metric for Evaluation of Translation with Explicit ORdering)

Considers synonyms, stemming, and paraphrases alongside n-gram overlap

Machine translation, where semantic equivalence matters

- Computationally expensive
- Subjective design of synonym/stemming databases

CIDEr (Consensus-based Image Description Evaluation)

Measures n-gram overlap weighted by TF-IDF, tailored for image captioning

Image caption generation

- Limited applicability outside captioning
- Heavily reliant on corpus statistics

TER (Translation Edit Rate)

Computes number of edits needed to convert hypothesis into reference text

Translation quality evaluation

- Doesn’t consider semantic correctness
- Penalizes valid paraphrasing

BERTScore

Uses contextual embeddings from pre-trained BERT to calculate token similarity

Tasks requiring semantic equivalence

- High computational cost
- Performance varies with model used

SPICE (Semantic Propositional Image Caption Evaluation)

Focuses on scene graphs in image captions to evaluate semantic content

Image captioning with emphasis on semantic accuracy

- Designed only for image captions
- Less effective in purely textual tasks

A common use case for metrics like BLEU and ROUGE is to evaluate the quality of generated summaries against reference summaries.

A Reference Summary represents the “ideal” summary. It could be prepared by humans, e.g. expert analysts, or machine-generated.

As an example, let’s demonstrate how to evaluate the quality of the executive summary of a 10-K, when compared against reference summaries (e.g. analyst-prepared summaries, or summaries prepared by various LLMs).

We model our simple metrics-based evaluator with the following components:

Input: Generated summary and reference summary

Output: Dictionary with scores for BLEU, ROUGE_1, and ROUGE_2

Purpose: Evaluate our 10-K summary generator to determine whether test models perform as well as a benchmark model.

Benchmark model: gpt-4o

Test models: gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo

The core of the evaluation system is the evaluate_summaries function, which calculates BLEU and ROUGE scores to assess text generation quality. More specifically, the evaluate_summaries function accepts a generated summary and reference summary as input, processes them, and returns a dictionary containing three key metrics:

BLEU

ROUGE_1

ROUGE_2

This enables quantitative comparison of generated summaries against reference texts. The implementation utilizes HuggingFace’s evaluate library to load and compute these metrics.

First install the necessary Python libraries.

pip install evaluate absl-py rouge_score

Next we buid the evaluation_summaries function.

import evaluate
def evaluate_summaries(generated_summary, reference_summary):
    """
    Evaluate generated summaries against reference summaries using multiple metrics.

    Args:
        generated_summary (str): The summary generated by the model
        reference_summary (str): The reference/ground truth summary

    Returns:
        dict: Dictionary containing scores for different metrics
    """
    # Initialize metrics
    bleu = evaluate.load("google_bleu")
    rouge = evaluate.load("rouge")

    # Format inputs for BLEU (expects list of str for predictions and list of list of str for references)
    predictions = [generated_summary]
    references = [reference_summary]

    # Compute BLEU score
    bleu_score = bleu.compute(predictions=predictions, references=[references])

    # Compute ROUGE scores
    rouge_score = rouge.compute(predictions=predictions, references=references)

    # Compute Character metric
    # Combine all scores into a single dictionary
    scores = {
        'bleu': bleu_score["google_bleu"],
        'rouge1': rouge_score['rouge1'],
        'rouge2': rouge_score['rouge2']
    }

    return scores

To take a simple example that demonstrates what this function does, evaluate_summaries can be used to compare two arbitrary sentences and return a dictionary with our chosen metrics.

As we can see, even with very simple metrics, results can vary quite a bit. Hence, relying on a single metric to measure LLMBA performance is not recommended.

sentence1 = "the cat sat on the mat"
sentence2 = "the cat ate the mat"

evaluate_summaries(sentence1, sentence2)

{'bleu': 0.3333333333333333,
 'rouge1': 0.7272727272727272,
 'rouge2': 0.4444444444444445}

Next, we define generate_summary, our simple LLM-based 10-K summarizer using OpenAI’s API.

It takes an arbitrary model, and an input text and returns a corresponding 1-line summary.

Note how we define the LLM’s role as “an expert analyst tasked with summarizing SEC filings”. This is an example of prompt engineering that we cover more thoroughly in Chapter 2.

from openai import OpenAI
client = OpenAI()

def generate_summary(model, input):
    """
    Generate a summary of input using a given model
    """
    TASK = "Generate a 1-liner summary of the following excerpt from an SEC filing."

    prompt = f"""
    ROLE: You are an expert analyst tasked with summarizing SEC filings.
    TASK: {TASK}
    """

    response = client.chat.completions.create(
    model=model,
        messages=[{"role": "system", "content": prompt},
                 {"role": "user", "content": input}]
    )
    return response.choices[0].message.content

Next we define a function evaluate_summary_models - our model evaluator - that compares text summaries generated by different language models (in our case we will pass “gpt-4o-mini”, “gpt-4-turbo”, “gpt-3.5-turbo”) against a benchmark model (in our case “gpt-4o”).

More specifically this function:

Takes a benchmark model, list of test models, prompt, and input text

Generates a reference summary using the benchmark model and our generate_summary function

Generates summaries from all test models using generate_summary function

Evaluates each test model’s summary against the benchmark using evaluate_summaries

Returns evaluation results and the generated summaries

def evaluate_summary_models(model_benchmark, models_test, input):
    """
    Evaluate summaries generated by multiple models
    """
    benchmark_summary = generate_summary(model_benchmark, input)

    # Generate summaries for all test models using list comprehension
    model_summaries = [generate_summary(model, input) for model in models_test]

    # Evaluate each model's summary against the benchmark
    evaluation_results = [evaluate_summaries(summary, benchmark_summary) for summary in model_summaries]

    return [evaluation_results, model_summaries, benchmark_summary]

We are ready to run our benchmark evaluation. We define a benchmark model and a list of test models and then evaluate each test model’s summary against the benchmark. We also print the generated summaries for each model.

model_benchmark = "gpt-4o"
models_test = ["gpt-4o-mini", "gpt-4-turbo", "gpt-3.5-turbo"]

evals, model_summaries, benchmark_summary = evaluate_summary_models(model_benchmark, models_test, sec_filing)

print(benchmark_summary)

"Apple Inc.'s 10-K filing for the fiscal year ending September 28, 2024, outlines its operational and financial condition, detailing the company's diverse product lines, market activities, and compliance with SEC requirements."

# Print each model name and its summary
for model, summary in zip(models_test, model_summaries):
    print(f"{model}: \n {summary} \n---------------")

gpt-4o-mini:
 Apple Inc. filed its Annual Report on Form 10-K for the fiscal year ending September 28, 2024, detailing its business operations, risks, and financial condition.
---------------
gpt-4-turbo:
 Apple Inc.'s Form 10-K for the fiscal year ended September 28, 2024, details its annual report as a well-known seasoned issuer, confirming compliance with SEC regulations and reporting on stock performances, securities, and corporate governance, while also including forward-looking statements subject to various risks.
---------------
gpt-3.5-turbo:
 Apple Inc. filed its Form 10-K with the SEC, revealing financial information for the fiscal year ended September 28, 2024, including details on its products and market performance.
---------------

The benchmark summary from gpt-4o provides a balanced overview of the analyzed excerpt from the 10-K, focusing on operational status, financial condition, product lines, and regulatory compliance.

When comparing our test models against the benchmark, we observe that:

gpt-4o-mini provides a concise yet comprehensive summary that closely aligns with the benchmark’s core message. While it omits product lines, it effectively captures the essential elements of the filing including business operations, risks, and financial condition. Its brevity and focus look (subjectively) similar to our benchmark model.

gpt-4-turbo performs adequately but tends toward verbosity. While it includes relevant information about SEC compliance, it introduces peripheral details about seasoned issuer status and forward-looking statements. The additional complexity makes the summary less focused than gpt-4o-mini’s version.

gpt-3.5-turbo looks quite different from the benchmark. Its summary, while factually correct, is overly simplified and misses key aspects of the filing. The model captures basic financial information but fails to convey the breadth of operational and compliance details present in the benchmark summary.

Of course, the above evaluation is only based on a single example and is heavily subjective. It’s a “vibe check” on our evaluation results.

For an objective analysis, we create a visualize_prompt_comparison function to visualize the performance of our test models across our predefined quantitative metrics.

pip install matplotlib

def visualize_prompt_comparison(evaluation_results, model_names):
    """
    Create a radar plot comparing different prompt variations

    Args:
        evaluation_results (list): List of dictionaries containing evaluation metrics
        model_names (list): List of names for each prompt variation
    """
    from evaluate.visualization import radar_plot

    # Format data for visualization
    plot = radar_plot(data=evaluation_results, model_names=model_names)
    return plot

# Create and display visualization
plot = visualize_prompt_comparison(evals, models_test)
plot.show()

/tmp/ipykernel_1652501/940173201.py:3: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
  plot.show()

We can see that the tested models perform quite differently on our predefined metrics. The evaluation metrics put gpt-4o-mini as the closest aligned to the benchmark, followed by gpt-4-turbo, and gpt-3.5-turbo showing the largest deviation. This suggests that gpt-4o-mini is the best model for this task and relative to the other tested models.

While evaluating language model outputs inherently involves subjective judgment, establishing a high-quality benchmark model and using quantifiable metrics provide a more objective framework for comparing model performance. This approach transforms an otherwise qualitative assessment into a measurable, data-driven evaluation process.

These metrics provide quantifiable measures of performance, but they do have limitations:

Task-specific nature: Chosen set of metrics might not fully capture the nuances of complex generative-based tasks, especially those involving subjective human judgment.

Sensitivity to data distribution: Performance on these metrics can be influenced by the specific dataset used for evaluation, which might not represent real-world data distribution.

Subjective Acceptable Threshold: These metrics are not always easy to interpret and set a threshold for (see [Reference to Come] for a discussion on how to choose a threshold for an evaluation metric for large language models).

Inability to assess reasoning or factual accuracy: These metrics primarily focus on surface-level matching and might not reveal the underlying reasoning process of the LLM or its ability to generate factually correct information.

In conclusion, selecting appropriate extrinsic metrics depends on the specific task, underlying business requirements and desired evaluation granularity. But we must be aware of the inherent limitations of this approach.

To address these limitations, in addition to leveraging humans in the evaluation process, we can employ a model-based evaluation, which will be discussed in the following sections.

Model-Based Evaluation

Traditional metrics like BLEU or ROUGE are valuable for their objectivity but often fall short in capturing the nuanced, contextual, and creative outputs of LLMs. As an alternative we can consider a “Model-based evaluation” approach, using an LLM-as-a-Judge . This method involves using a model (often a more capable one) to act as an automated judge, evaluating aspects like accuracy, coherence, and relevance of generated content. Unlike traditional metrics that rely on exact matching or statistical measures, model-based evaluation can capture nuanced aspects of language and provide more contextual assessment.

As discussed in the paper [Reference to Come], LLM-based evaluation approaches generally fall into two main categories:

Prompt-based evaluation: We use prompts to instruct LLMs to evaluate text quality without any fine-tuning. The evaluation can take several forms:

Score-based: LLMs assign numerical scores to generated text

Probability-based: Using generation probability as a quality metric

Likert-style: Rating text quality on discrete scales

Pairwise comparison: Directly comparing two texts

Ensemble methods: Combining multiple LLM evaluators

Tuning-based evaluation: We fine-tune open-source LLMs specifically for evaluation tasks. This can be more cost-effective than repeatedly using API calls and allows for domain adaptation.

Once we have chosen our approach, a general LLM-as-a-Judge procedure involves the following steps (see Figure 1-3):

Define Evaluation Criteria: Establish clear benchmarks, such as relevance, coherence, accuracy, and fluency.

Prepare Prompts: Craft effective prompts to guide the LLM in evaluating content against the criteria.

Define Reference Data: Establish a set of reference data that the judge model can use to evaluate the generated outputs. (Optional)

Run Evaluations: Use the judge model to score outputs. Consider using a large and/or more capable model as a judge to provide more nuanced assessments.

Aggregate and Analyze Results: Interpret scores to refine applications.

Figure 1-3. Conceptual overview of LLM-as-a-Judge evaluation.

Compared to traditional metrics, LLM-as-a-Judge evaluation offers a more sophisticated assessment framework by leveraging natural language criteria. While metrics focus on statistical measures, judge models excel at evaluating subjective qualities such as creativity, narrative flow, and contextual relevance - aspects that closely mirror human judgment. The judge model processes evaluation guidelines expressed in natural language, functioning similarly to a human reviewer interpreting assessment criteria.

One notable consideration is that this approach requires careful prompt engineering to properly define and communicate the evaluation standards to the model [Reference to Come]. We will discuss prompt engineering in greater in detail in Chapter 2. For now, we can define it as the art of crafting inputs to LLMBAs that elicit our desired outputs. It involves writing precise instructions, context, and constraints that guide LLMBAs to produce more accurate, relevant, and useful answers.

Hence, it’s worth noting key prompting best practices when designing LLM-as-a-Judge evaluators [Reference to Come]:

Use discrete integer scales (e.g., 1-5) rather than continuous ranges

Provide clear rubrics that define what each score level means

Include reference answers when available to ground the evaluation

Break down complex judgments into specific evaluation criteria

Additionally, prompt engineering can foster greater interpretability of our evaluation framework by:

Requiring explanations and reasoning for scores to increase transparency

Having a hollistic evaluation by considering multiple dimensions such as coherence, relevance, and fluency

Order matters when generating an LLM response! Results performance can significantly vary depending on whether the LLM Judge is asked to provide a score first and then explanation next versus the opposite way around. A “Think Before You Speak” strategy is recommended, i.e. to place score explanation first and the actual resulting score next can improve results [Reference to Come]

Below we provide a sample implementation of an LLM-as-a-Judge evaluation system for our LLMBA that generates 10-K summaries. The code defines:

A JudgeEvaluation Pydantic model that enforces type validation for four key metrics:

Expertise: Rating of analyst-level writing quality

Coherence: Score for logical organization

Fluency: Assessment of grammar and clarity

Similarity: Measure of alignment with reference text

An evaluate_with_llm() function that:

Takes a judge model, candidate summary, and reference summary as inputs

Constructs a detailed prompt instructing the LLM to act as an expert evaluator

Uses structured output parsing to return scores in a consistent format

Returns scores on a 1-10 scale for each evaluation criterion

The code below demonstrates how to combine structured data validation with natural language evaluation to create a robust automated assessment system.

from pydantic import BaseModel
from typing import List, Dict

class JudgeEvaluation(BaseModel):
    expertise: int
    coherence: int
    fluency: int
    similarity: int
def evaluate_with_llm(judge_model: str, candidate_summary: str, reference_summary: str) -> Dict[str, float]:
    """
    Use an LLM to evaluate a candidate summary against a reference summary.

    Args:
        judge_model (str): Name of the model to use as the judge.
        candidate_summary (str): Generated summary to evaluate.
        reference_summary (str): Ground truth or benchmark summary.

    Returns:
        dict: Dictionary containing evaluation scores for specified criteria.
    """
    prompt = f"""
    ROLE: You are an expert evaluator of SEC Filing summaries. Evaluate the following candidate summary against the reference summary on a scale of 1 to 10 for the following criteria:
    - Expertise: Does the summary look like it was written by an expert analyst?
    - Coherence: Is the candidate summary logically organized and easy to understand?
    - Fluency: Is the language of the candidate summary clear and grammatically correct?
    - Similarity: How similar is the candidate summary compared to the reference summary?

    Reference Summary:
    "{reference_summary}"

    Candidate Summary:
    "{candidate_summary}"

    Provide scores in this format:
    Expertise: X, Coherence: Y, Fluency: Z, Similarity: W
    """
    completion = client.beta.chat.completions.parse(
        model=judge_model,
        messages=[{"role": "system", "content": prompt}],
        response_format=JudgeEvaluation
    )
    return completion.choices[0].message.parsed

Next, we define a evaluate_summary_models function that leverages our LLM-as-a-Judge function to compare summaries generated by different language models.

The evaluate_summary_models function works in three parts:

First, it generates a benchmark summary using the specified benchmark model

Then, it generates summaries using each of the test models

Finally, it evaluates each test model’s summary against the benchmark using the judge model

As a result, we get a list of evaluation results we can use to compare our candidate LLM models across our predefined metrics.

def evaluate_summary_models(judge_model: str, benchmark_model: str, test_models: List[str], input_text: str):
    """
    Evaluate summaries generated by multiple models using an LLM-as-a-Judge approach.

    Args:
        judge_model (str): Name of the model to use as the judge.
        benchmark_model (str): Name of the benchmark model.
        test_models (list): List of model names to test.
        input_text (str): Input text for summarization.

    Returns:
        tuple: Evaluation results, model summaries, benchmark summary.
    """
    benchmark_summary = generate_summary(benchmark_model, input_text)
    model_summaries = [generate_summary(model, input_text) for model in test_models]

    evaluation_results = [
        evaluate_with_llm(judge_model, summary, benchmark_summary)
        for summary in model_summaries
    ]

    return evaluation_results, model_summaries, benchmark_summary

# Example Usage
model_benchmark = "gpt-4o"
models_test = ["gpt-4o-mini", "gpt-4-turbo", "gpt-3.5-turbo"]
judge_model = "gpt-4o"

evals, model_summaries, benchmark_summary = evaluate_summary_models(
    judge_model, model_benchmark, models_test, sec_filing
)

Here, we can see the benchmark summary coming from our benchmark model gpt-4o:

benchmark_summary

"Apple Inc.'s annual report for the fiscal year ending September 28, 2024, details its business operations, financial condition, and product lines, including iPhones, Macs, iPads, and wearables, and incorporates forward-looking statements regarding its future performance."

Next, we obtain the summaries and evaluation results generated by our test models, gpt-4o-mini, gpt-4-turbo and gpt-3.5-turbo, respectively.

model_summaries

['Apple Inc. filed its annual Form 10-K report for the fiscal year ended September 28, 2024, detailing its business operations, product lines, and financial performance.',
 "This Form 10-K filing by Apple Inc. for the fiscal year ended September 28, 2024, is an annual report detailing the company's financial performance, including registered securities, compliance with SEC reporting standards, and contains sections on business operations, risk factors, financial data, and management analysis.",
 'Apple Inc., a California-based technology company, reported an aggregate market value of approximately $2.6 trillion held by non-affiliates, with 15.1 billion shares of common stock outstanding as of October 18, 2024.']

As a result we get a list of objects of the Pydantic class we have defined JudgeEvaluation which contains the metrics of our evaluation (expertise, coherence, fluency and similarity).

evals

[JudgeEvaluation(expertise=7, coherence=8, fluency=8, similarity=7),
 JudgeEvaluation(expertise=7, coherence=7, fluency=8, similarity=5),
 JudgeEvaluation(expertise=4, coherence=5, fluency=7, similarity=2)]

# Convert evaluation objects to dictionaries
evals_list = [
    {
        "expertise": eval.expertise,
        "coherence": eval.coherence,
        "fluency": eval.fluency,
        "similarity": eval.similarity
    }
    for eval in evals
]

# Visualize results
plot = visualize_prompt_comparison(evals_list, models_test)
plot.show()

/tmp/ipykernel_1652501/1775618912.py:14: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
  plot.show()

Analyzing the evaluation results across our test models (gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo), several interesting patterns emerge:

Thegpt-4o-mini model demonstrated strong performance, achieving high scores across all metrics (expertise: 7, coherence: 8, fluency: 8, similarity: 7). This performance suggests it maintained robust quality despite being a smaller variant of our benchmark model gpt-4o.

The gpt-4-turbo model exhibited comparable expertise and fluency scores (7 and 8 respectively) but showed slightly reduced coherence (7) and notably lower similarity (5) compared to the benchmark. These results could indicate some divergence from the reference summary while maintaining overall quality.

The gpt-3.5-turbo model recorded the lowest scores (expertise: 4, coherence: 5, fluency: 7, similarity: 2), with particular weaknesses in expertise and similarity to the benchmark. While maintaining acceptable fluency, the marked decrease in similarity score indicates substantial deviation from the reference summary.

A distinct performance gradient can be observed from gpt-4o-mini to gpt-3.5-turbo, with the latter exhibiting significant degradation across most metrics.

It should be noted, however, that leveraging LLMs for evaluation has several limitations [Reference to Come]. Firstly, computational overhead should not be neglected given the inherent cost of running additional model inferences iterations. LLM evaluators can also exhibit various biases, including order bias (preferring certain sequence positions), egocentric bias (favoring outputs from similar models), and length bias. Further, there may be a tight dependency on prompt quality - small prompt variations may lead to substantially different outcomes. We must be careful to isolate whether we are judging prompt quality, model quality, overall LLMBA quality and/or variations of each of these.

Furthermore, in professional domains such as finance, law and medicine, the judge LLM itself may lack expertise and may not be suitable to be the arbiter of quality. How could we address this? One approach is to fine-tune the judge LLM, which would then need to be evaluated.

Overall, though, an LLM-as-a-Judge strategy can serve as a scalable and nuanced solution to evaluate LLMBAs. While it does not entirely replace metrics-based or human-based approaches, it significantly augments evaluation workflows, especially in scenarios requiring evaluation of generative outputs.

One area of future research in this rapidly growing field is the open source Glider [Reference to Come], a 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. Glider is an LLM model trained on 685 domains and 183 criteria whose judgement scores show 91.3% agreement with human judgments, making it suitable for a diverse range of real world applications.

Evaluating Evaluators

We have discussed how LLMs can be used to evaluate LLMBAs. But, how can we evaluate the performance of LLMs that evaluate other LLMs? This is the question that meta evaluation aims to answer. Clearly, this entire discussion can land us in a kind of infinite quest to evaluate the performance of the evaluator to evaluate the performance of the evaluated model. That’s a fun mental exercise around AI in general, but for practical purposes we are happy to conclude that we simply wish to evaluate our primary evaluator.

With that discrete task in mind, we present two general options:

Use a golden-standard dataset that is used to evaluate the performance of LLM evaluators with a “metrics-based” approach.

Use a human evaluator to generate reference scores that can be used to evaluate the performance of the LLM evaluator (similar to the human-based evaluation we discussed earlier).

As depicted in Figure 1-4, the performance of the LLM evaluator can be evaluated by comparing its scores to either a golden-standard dataset or human reference scores. Higher correlation values indicate better performance of the LLM evaluator. For instance, if we were to evaluate the performance of a LLM-as-a-Judge evaluator, in the task of evaluating multilingual capability of an LLM:

In a “metrics-based” approach, we would first need to define a set of metrics that capture the task of multilingual capability. For instance, we could use the BLEU metric to evaluate the quality of the generated LLM output against a golden dataset (e.g. machine translated text). We would then calculate the correlation between these scores against those generated by the LLM evaluator. The higher the correlation, the better the LLM evaluator.

In a “human-based” approach, we would need to recruit human evaluators that are experts in the target languages we are evaluating. Expert humans would provide scores for a set of samples of the input LLM. We would then calculate the correlation between these scores against those generated by the LLM evaluator. The higher the correlation, the better the LLM evaluator.

Figure 1-4. Conceptual overview of LLMs Meta Evaluation.

An alternative to the above approaches is to use humans to directly evaluate the judge LLMs themselves. A notable example of this is Judge Arena [Reference to Come], which is a platform that allows users to vote on which AI model made the better evaluation. Under this approach, the performance of the LLM evaluator is given by the (blind) evaluation of humans who perform the voting on randomly generated pairs of LLM judges as depicted in Figure 1-5. Only after submitting a vote, users can see which models were actually doing the judging.

Figure 1-5. Human-in-the-loop Meta Evaluation.

The LLM input and its prompt are displayed to the human evaluator and are customizable enabling task-specific meta evaluation. Further, the Judge Arena’s LLM Judge’s prompt is also editable by the user. Its default prompt is presented below:

Does the model provide relevant and useful responses to the user’s needs or questions?

Scoring Rubric:

Score 1: The model’s responses are irrelevant or unhelpful to the user’s needs or queries.

Score 2: The model sometimes provides helpful information, but often fails to address the user’s actual needs or questions.

Score 3: The model generally provides helpful responses that address the user’s needs, though it may occasionally miss the mark.

Score 4: The model regularly provides helpful responses that are well-aligned with the user’s inquiries, with only rare inaccuracies.

Score 5: The model consistently offers highly relevant and useful responses that perfectly cater to the user’s needs and inquiries.

Judge Arena’s approach and policy framework has three key benefit worth highlighting:

Transparency through open-source code, documentation, and data sharing

LLM inclusion criteria requiring scoring/critique capabilities and public accessibility

ELO-based leaderboard system with community involvement in evaluations

In that way, the platform enables democratic evaluation of AI judges while maintaining transparency and accessibility standards.

A Brief History of Benchmarks

Benchmarks act as standardized tests for LLMs, evaluating their performance across a spectrum of tasks. These tasks simulate real-world applications such as answering questions, generating coherent text, solving mathematical problems, or even writing computer code. They also assess more abstract qualities like fairness, robustness, and cultural understanding. Benchmarks can be thought as comprehensive “exams” that probe different “subjects” in order to certify an LLM.

In this section, we discuss various benchmarks and how they have evolved over time. It’s not strictly necessary to understand this in order to build our own evaluation framework but it does help us maintain awareness when new models appear that dominate certain benchmarks and gain popular attention.

The history of LLM benchmarks reflects the evolving priorities of artificial intelligence research, starting with foundational tasks and moving toward complex, real-world challenges. W

We can start in 2018 with the introduction of GLUE (General Language Understanding Evaluation) [Reference to Come], which set a new standard for evaluating natural language understanding. GLUE measured performance on tasks like sentiment analysis and textual entailment, providing a baseline for assessing the fundamental capabilities of language models. Later, SuperGLUE [Reference to Come] expanded on this foundation by introducing more nuanced tasks that tested reasoning and language comprehension at a deeper level, challenging the limits of models like BERT and its successors.

As AI capabilities grew, benchmarks evolved to capture broader and more diverse aspects of intelligence. BIG-Bench [Reference to Come] marked a turning point by incorporating over 200 tasks, spanning arithmetic, logic, and creative problem-solving. This collaborative effort aimed to probe emergent abilities in large models, offering insights into how scale and complexity influence performance. Around the same time, specialized benchmarks like TruthfulQA [Reference to Come] emerged, addressing the critical need for models to provide accurate and non-deceptive information in a world increasingly dependent on AI for factual content.

MMLU (Massive Multitask Language Understanding) [Reference to Come] launched in 2021, provided a rigorous test of a model’s multidisciplinary knowledge, covering 57 subjects from STEM fields to humanities and social sciences. Similarly, in 2022, Stanford’s HELM (Holistic Evaluation of Language Models) [Reference to Come] set a new standard for multidimensional assessment. HELM expanded the scope of evaluation beyond accuracy, incorporating factors like fairness, robustness, and computational efficiency. This benchmark was designed to address societal concerns surrounding AI, emphasizing safety and inclusion alongside technical performance.

Specialized benchmarks like HumanEval (2021) [Reference to Come] focused on domain-specific tasks, such as code generation, testing models’ ability to translate natural language descriptions into functional programming code. In contrast, LMSYS (2023) brought real-world applicability into focus by evaluating conversational AI through multi-turn dialogues. LMSYS prioritized coherence, contextual understanding, and user satisfaction, providing a practical lens for assessing models like GPT and Claude in dynamic settings.

The HuggingFace Open LLM [Reference to Come] Leaderboard stands out for its transparency and accessibility in the open-source community. This leaderboard evaluates a wide range of LLMs across diverse tasks, including general knowledge, reasoning, and code-writing. Its commitment to reproducibility ensures that results are verifiable, enabling researchers and practitioners to replicate findings. By focusing on open-source models, it democratizes AI research and fosters innovation across communities, making it a valuable resource for both academics and industry professionals.

The Chatbot Arena (2024) Leaderboard (an evolution of LMSYS) [Reference to Come] takes an alternative approach by measuring real-world performance through direct model comparisons. Its evaluation format compares models in live conversations, with human judges providing qualitative assessments. This methodology has gathered hundreds of thousands of human evaluations, offering specific insights into practical model performance. The emphasis on interactive capabilities makes it relevant for developing user-facing applications like virtual assistants and chatbots.

The AlpacaEval [Reference to Come] and MT-Bench [Reference to Come] Leaderboards implement automated evaluation using LLMs to assess model performance in multi-turn conversations. This approach enables consistent assessment of dialogue capabilities while reducing human bias. Their methodology measures key aspects of conversational AI, including contextual understanding and response consistency across multiple exchanges.

An important recent development was the release of Global-MMLU [Reference to Come], an improved version of MMLU with evaluation coverage across 42 languages. This open dataset, built through collaboration between Argilla, the Hugging Face community, and researchers from leading institutions like Cohere For AI, Mila, MIT, and others, represents a significant step toward more inclusive multilingual LLM evaluation. Hundreds of contributors used Argilla to annotate MMLU questions, revealing that 85% of questions requiring specific cultural knowledge were Western-centric. The newly released dataset is divided into two key subsets: Culturally Agnostic questions that require no specific regional or cultural knowledge, and Culturally Sensitive questions that depend on dialect, cultural, or geographic knowledge. With high-quality translations available for 25 languages, Global-MMLU enables better understanding of LLM capabilities and limitations across different languages and cultural contexts.

A major challenge with these leaderboards and benchmarks is test set contamination - when test data ends up in newer models’ training sets, rendering the benchmarks ineffective. While some benchmarks try to address this through crowdsourced prompts and evaluations from humans or LLMs, these approaches introduce their own biases and struggle with difficult questions. LiveBench [Reference to Come] represents a novel solution, designed specifically to be resilient to both contamination and evaluation biases. As the first benchmark with continuously updated questions from recent sources, automated objective scoring, and diverse challenging tasks across multiple domains, LiveBench maintains its effectiveness even as models improve. Drawing from recent math competitions, research papers, news, and datasets, it creates contamination-free versions of established benchmark tasks. Current results show even top models achieving considerably lower performance compared to other benchmarks, demonstrating LiveBench’s ability to meaningfully differentiate model capabilities with relatively lower saturation. With monthly updates and an open collaborative approach, LiveBench aims to provide sustained value for model evaluation as the field advances.

Another notable benchmark is ZebraLogic [Reference to Come]`, which evaluates logical reasoning capabilities of LLMs through Logic Grid Puzzles - a type of Constraint Satisfaction Problem [Reference to Come] commonly found in tests like the LSAT. These puzzles require assigning unique values to N houses across M different features based on given clues, demanding strategic reasoning and deduction to arrive at a unique correct solution. The benchmark’s programmatically generated puzzles range from 2x2 to 6x6 in size and test LLMs using one-shot examples with reasoning steps. While humans can solve these puzzles through strategic methods like reductio ad absurdum and elimination, LLMs demonstrate significant limitations in this type of logical reasoning. Even the best-performing model, Claude 3.5 Sonnet, only achieves 33.4% accuracy across all puzzles and 12.4% on hard puzzles, with smaller models (7-10B parameters) solving less than 1% of hard puzzles as of December 2024. These results reveal critical gaps in LLMs’ capabilities around counterfactual thinking, reflective reasoning, structured memorization, and compositional generalization.

A significant milestone in AI evaluation came with the launch of the The Alignment Research Center (ARC) Prize [Reference to Come] by ARC Prize Inc., a non-profit for the public advancement of open artificial general intelligence. Hosted by Mike Knoop (Co-founder, Zapier) and François Chollet (Creator of Keras), this prize represents a paradigm shift in how we evaluate language models. Rather than focusing on narrow performance metrics, the ARC Prize assesses what it calls “cognitive sufficiency” - a model’s ability to generate meaningful insights and tackle open-ended challenges. ARC emphasizes creative thinking, sophisticated reasoning, and the capacity to make genuinely useful contributions to human knowledge.

ARC emphasizes creativity and contains the phrase AGI (Artificial General Intelligence) in it’s name, which leads to plenty of hype when a model does well on the ARC benchmark - have we now created sentient LLMs?

Even though ARC constains AGI in its name, we need to be careful before claiming beating ARC-AGI tasks directly leads to AGI proof. As François Chollet clearly put it: “I don’t think people really appreciate how simple ARC-AGI-1 was, and what solving it really means. It was designed as the simplest, most basic assessment of fluid intelligence possible. Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. Passing it means your system exhibits non-zero fluid intelligence – you’re finally looking at something that isn’t pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is[Reference to Come].

Here is how ARC defines and does not define AGI:

Consensus but wrong:

AGI is a system that can automate the majority of economically valuable work.

Correct:

AGI is a system that can efficiently acquire new skills and solve open-ended problems.

ARC does not lead us to AGI but it does distinguishes itself from other LLM benchmarks in 4 important ways:

Focus on Core Knowledge: Unlike LLM benchmarks that test a broad range of knowledge and skills, often relying heavily on memorization, ARC focuses on core knowledge similar to what a four or five-year-old child might possess. This includes basic concepts like object recognition, counting, and elementary physics.

Novelty of Tasks: Each ARC puzzle is designed to be novel, meaning it’s something an LLM likely wouldn’t have encountered when being trained, even if it has been trained on the entire internet. This characteristic directly challenges the way LLMs typically operate, which is by leveraging their vast “interpolative memory.”

Emphasis on Program Synthesis: ARC tasks require models to synthesize new solution programs on the fly for each unique puzzle. This stands in contrast to the more common LLM approach of retrieving pre-existing solution programs from memory.

Resistance to Brute Force Attempts: While acknowledging the possibility, ARC aims to be resistant to brute-force approaches where a model might be trained on millions of similar puzzles to achieve a high score by relying on overlap with the test set.

ARC tasks are a series of three to five input and output tasks followed by a final task with only the input listed (e.g. Figure 1-6). Each task tests the utilization of a specific learned skill based on a minimal number of cognitive priors. A successful submission is a pixel-perfect description (color and position) of the final task’s output.

Figure 1-6. Sample ARC-AGI Task.

These features make the ARC benchmark a unique test of machine intelligence, focusing on the ability to adapt to novelty and solve problems without relying heavily on memorization. This is more aligned with the concept of general intelligence, which emphasizes the ability to learn efficiently and tackle new challenges.

The ARC benchmark remained unbeaten for five years as of December 2024 (a minimum score of 85% in the private dataset is required to win) [Reference to Come] which stronly suggests that algorithmic improvements (human creativity in building LLMs), rather than massive computational resources, may be key to exceeding the target score .

Before we conclude this section, highlight the growing set of domain-specific benchmarks is emerging to help evaluate LLMs in specific verticals. These are an important but nascent field of research.

FinBench [Reference to Come]: Evaluates LLMs in the financial domain, covering tasks such as terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling.

LegalBench [Reference to Come]: Assesses the legal reasoning abilities of LLMs through tasks crowdsourced by legal professionals

Berkeley Function Leaderboard (BFCL) [Reference to Come]: Evaluates LLMs’ function-calling abilities

GIFT-Eval [Reference to Come]: Benchmarks Time Series Forecasting models

As language models continue to advance in capability and complexity, evaluation frameworks must evolve. Modern benchmarks increasingly incorporate tests for nuanced reasoning, ethical decision-making, and emergent capabilities that weren’t previously measurable. This ongoing evolution reflects a deeper understanding that the true value of language models lies not in achieving high scores on standardized tests with narrow task-specific metrics, but in their ability to meaningfully contribute to human understanding and help solve real-world problems while demonstrating the ability to learn and adapt to new tasks.

Conclusion

LLMs have fundamentally transformed how software and business applications are developed. Unlike conventional systems that produce predictable outputs, LLMs and LLMBAs generate varied, probabilistic responses that defy traditional testing approaches. Traditional, legacy testing frameworks were not designed to handle the inherent variability of LLM outputs and will ultimately prove inadequate. Success requires embracing this new paradigm by implementing comprehensive evaluation plans that cover the non-deterministic generative nature of LLMs. We have introduced several code flows and tools that help solve these challenges, both with regard to LLMs themselves and the prompts that we pass to those LLMs. In short, evaluation frameworks are a key first step in taming LLMs and deploying them at the enterprise level. In the next Chapter, we will explore some open source tools developers can use to automate and streamline the challenging task of LLMBA evaluation.

¹ Even at temperature 0, LLM results are not deterministic.

² We cover how to manage long input data in Chapter 2.

³ In fact, human evaluation has become a growing business, for example , , and more.

⁴ The term “foundation model” was first coined by the Stanford Institute for Human-Centered Artificial Intelligence (HAI). It designated foundation models as “models trained on broad data at scale such that they can be adapted to a wide range of downstream tasks” [Reference to Come].

⁵ Anthropic’s “A statistical approach to model evaluations” serves as an excellent reference on how to estimate a confidence level for an evals measure [Reference to Come].

⁶ A lower perplexity indicates better prediction (less “perplexed” by the text) insert equation for perplexity.

Chapter 2. Open-Source Evaluation Frameworks

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at jbleiel@oreilly.com.

We have covered in detail how to conceptualize, design and code up our own evaluation framework. We believe there is value in undertaking this challenge in order to understand evaluation frameworks and apply our own creativity to them. Unsurprisingly, there are many paid tools that help with this as well, both standalone and as part of large cloud environments. We stick to open source, free tools here so that any reader can reproduce our work.

In the remainder of this Chapter, we explore three of those open-source evaluation frameworks, specifically: LangSmith, Promptfoo and LightEval.

LangSmith

LangSmith is a developer framework that provides comprehensive tools for building, testing, monitoring and evaluating LLMs and LLMBAs throughout their lifecycle. It enables teams to track model performance, identify issues in production, and continuously improve LLMBAs through systematic evaluation against defined criteria. Langsmith was built and is developed and maintained by the same team the builds langchain, a very popular library for LLM development.

To see how langsmith works, let’s revisit our previous example where we generate summaries of a 10-K with several (smaller and cheaper) LLM models compared to a benchmark model (larger and more expensive). Recall the setup:

Benchmark model: gpt-4o

Test models: gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo

In the several code chunks below, we will use langsmith to:

create a golden dataset based on our 10-K, to be used for evaluation

create a function that will calculate BLEU scores by comparing generated summaries to the golden dataset

create a function to generate test summaries from our test models

create a function to run the evaluation

Note that with this process, we can easily add more models to our LLMBA and test those models, providing flexibility and openness as new models emerge in the future.

We begin by installing langsmith (and if a reader experiences issues with langchain, note that it can be uninstalled as we won’t be using it in this section).

!pip install langsmith
!pip uninstall langchain
!pip uninstall langchain-community
!pip uninstall langchain-openai

We also need to generate an API key to use LangSmith (see instructions here) and export our API_KEY. Activating tracing will allow us to track logs and foster observability of our evaluation.

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=<your-api-key>

import evaluate as hf_evaluate  # HuggingFace's evaluate
from langsmith import evaluate as langsmith_evaluate  # LangSmith's evaluate
from langsmith import Client
from typing import Dict, Any

ls_client = Client()

Creating a Golden Dataset

The code below creates a dataset in LangSmith that will serve as our golden dataset for evaluation. The dataset consists of test cases where we create a single example with the following content:

An input: Our SEC filing document

An expected output: A golden summary generated by our benchmark model (gpt-4o)

This dataset will allow us to evaluate how well other models perform compared to our benchmark by comparing their generated summaries against these reference summaries. In practice, it’s recommended to create a larger dataset with more diverse examples to get a more accurate assessment of model capabilities as well as to estimate confidence intervals for target metrics.

# Define dataset: these are your test cases
dataset_name = "Golden SEC Summary Dataset"
dataset = ls_client.create_dataset(dataset_name)
ls_client.create_examples(
    inputs=[
        {"sec_filing": sec_filing},
    ],
    outputs=[
        {"summary": benchmark_summary},
    ],
    dataset_id=dataset.id,
)

Our Dataset is now available in LangSmith as shown in Figure 2-1.

Figure 2-1. LangSmith Dataset

Calculating BLEU Scores

Next, we write our evaluator. This evaluator calculates BLEU scores between generated and reference summaries using HuggingFace’s evaluate package. The evaluator takes two dictionaries as input - one containing the generated summary and another containing the reference summary. It returns a dictionary with the Google BLEU score, which measures the overlap between n-grams in the generated and reference texts similar to our previous metric-based example that we coded ourselves.

def calculate_scores(outputs: Dict[str, Any], reference_outputs: Dict[str, Any]) -> dict:
    """
    Custom evaluator that calculates BLEU and ROUGE scores between generated and reference summaries
    using HuggingFace's evaluate package

    Args:
        outputs (dict): Contains the generated summary
        reference_outputs (dict): Contains the reference summary

    Returns:
        dict: Dictionary containing Google BLEU score
    """
    generated = outputs.get("summary", "")
    reference = reference_outputs.get("summary", "")

    # Initialize metrics from HuggingFace's evaluate
    bleu = hf_evaluate.load("google_bleu")

    # Format inputs for BLEU (expects list of str for predictions and list of list of str for references)
    predictions = [generated]
    references = [reference]

    # Compute BLEU score
    bleu_score = bleu.compute(predictions=predictions, references=[references])

    return {"key": "google_bleu", "score": bleu_score["google_bleu"]}

Now that we have defined our evaluation metrics, let’s create a function to generate summaries for our smaller models. The function below takes a dictionary containing the 10-K as input and returns a dictionary with the generated summary. The prompt instructs the model to act as an expert analyst and generate a one-line summary of the filing excerpt. We use the same task and model configuration as in our previous experiments to maintain consistency in our evaluation pipeline.

Note again how we need to employ a prompt inside of our evaluation framework. The importance of prompt quality is a consistent theme of any LLMBA development.

from openai import OpenAI
oai_client = OpenAI()

TASK = "Generate a 1-liner summary of the following excerpt from an SEC filing."

PROMPT = f"""
ROLE: You are an expert analyst tasked with summarizing SEC filings.
TASK: {TASK}
"""

xp_model_name = "" # model to be tested

def generate_summary(inputs: dict):
    """
    Generate a summary of input using a given model
    """
    TASK = "Generate a 1-liner summary of the following excerpt from an SEC filing."

    response = oai_client.chat.completions.create(
    model=xp_model_name, # model_name is a global variable
        messages=[{"role": "system", "content": PROMPT},
                 {"role": "user", "content": inputs.get("sec_filing")}]
    )
    return {"summary": response.choices[0].message.content}

Running the Evaluation

Lastly we define a function to run our evaluation. The run_evaluation() function uses LangSmith’s evaluate() to run evaluations either locally or remotely (when running locally, results are not uploaded to LangSmith’s servers)

The function takes an application, dataset, and list of evaluators as input and returns the evaluation results. The application is the generate_summary() function we would like to evaluate. The dataset is the golden summary from the strong model, and we pass a list with our single evaluator calculate_scores(). LangSmith also allows for running multiple repetitions of the same experiment (this is one way to get our heads around the non-deterministic tendencies of our LLMBAs) to get a more accurate assessment of model capabilities as well as to estimate confidence intervals for target metrics, which we set to 5 repetitions.

This allows us to systematically assess our LLMBAs while maintaining control over where results are stored.

def run_evaluation(app, model_name, dataset,  evaluators, upload_results=False):
    global xp_model_name
    xp_model_name = model_name
    results = langsmith_evaluate(
        app,
        client=None,
        data=dataset,
        evaluators=evaluators,
        experiment_prefix=model_name,
        num_repetitions=5,
        upload_results= upload_results,  # This is the key parameter for local evaluation

    )

    return results

Now we are ready run evaluation on our app across all target LLM models.

app = generate_summary

models = ["gpt-3.5-turbo", "gpt-4-turbo", "gpt-4o-mini"]
results = [run_evaluation(app, model, dataset=dataset_name, evaluators=[calculate_scores], upload_results=True) for model in models]

View the evaluation results for experiment: 'gpt-3.5-turbo-386a3620' at:
https://smith.langchain.com/o/9e1cc3cb-9d6a-4356-ab34-138e0abe8be4/datasets/8741976e-5268-4b75-949f-99477dde5d64/compare?selectedSessions=b831dc1e-90bc-4ed8-8080-fb42444724d6

4it [00:10,  2.59s/it]Using the latest cached version of the module from /home/tobias/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--google_bleu/6fc70b7be0088120a372dfdd5d320b39b8bb3630cb8029b193941d9376e86bb0 (last modified on Tue Nov 26 16:50:45 2024) since it couldn't be found locally at evaluate-metric--google_bleu, or remotely on the Hugging Face Hub.
5it [00:15,  3.09s/it]

View the evaluation results for experiment: 'gpt-4-turbo-5053784e' at:
https://smith.langchain.com/o/9e1cc3cb-9d6a-4356-ab34-138e0abe8be4/datasets/8741976e-5268-4b75-949f-99477dde5d64/compare?selectedSessions=64445871-a53c-44b1-a422-4f49b2f9656f

5it [00:13,  2.69s/it]

View the evaluation results for experiment: 'gpt-4o-mini-4b29f3c9' at:
https://smith.langchain.com/o/9e1cc3cb-9d6a-4356-ab34-138e0abe8be4/datasets/8741976e-5268-4b75-949f-99477dde5d64/compare?selectedSessions=9ef7e39a-2add-410c-89f8-9f1a8b198cf1

5it [00:13,  2.61s/it]

We can obtain the results for all experiments including the execution times and the Google BLEU scores.

import pandas as pd

# Create list of dataframes from results
dfs = [result.to_pandas() for result in results]

for df, model in zip(dfs, models):
    df.insert(0, 'model', model)

combined_df = pd.concat(dfs, ignore_index=True)

# Display the results - JEFF: this is an output. shall we make it a png and display that?
combined_df.head()

| model | inputs.sec_filing | outputs.summary | error | reference.summary | feedback.google_bleu | execution_time | example_id | id
0 | gpt-3.5-turbo | UNITED STATES\nSECURITIES AND EXCHANGE COMMISS... | Apple Inc.’s Form 10-K for the fiscal year end... | None | Apple Inc.’s 10-K filing for the fiscal year e... | 0.333333 | 1.224388 | feb10f92-3167-41f3-bb1c-d271153a31a8 | 5b196b22-9f4c-489c-b020-7823208b42d6
1 | gpt-3.5-turbo | UNITED STATES\nSECURITIES AND EXCHANGE COMMISS... | Apple Inc. filed its Form 10-K Annual Report f... | None | Apple Inc.’s 10-K filing for the fiscal year e... | 0.348101 | 0.722464 | feb10f92-3167-41f3-bb1c-d271153a31a8 | c310f159-064a-4035-97c3-a25bbf43abc2
2 | gpt-3.5-turbo | UNITED STATES\nSECURITIES AND EXCHANGE COMMISS... | Apple Inc. filed its annual Form 10-K for the ... | None | Apple Inc.’s 10-K filing for the fiscal year e... | 0.386076 | 0.704104 | feb10f92-3167-41f3-bb1c-d271153a31a8 | f7f24899-dd50-409e-93cc-6fb1622b60bf
3 | gpt-3.5-turbo | UNITED STATES\nSECURITIES AND EXCHANGE COMMISS... | Apple Inc. filed its Annual Report on Form 10-... | None | Apple Inc.’s 10-K filing for the fiscal year e... | 0.443038 | 0.725059 | feb10f92-3167-41f3-bb1c-d271153a31a8 | 242856d6-efb5-4101-b1cf-5805532838ac
4 | gpt-3.5-turbo | UNITED STATES\nSECURITIES AND EXCHANGE COMMISS... | Apple Inc. filed its Annual Report on Form 10-... | None | Apple Inc.’s 10-K filing for the fiscal year e... | 0.373418 | 0.795302 | feb10f92-3167-41f3-bb1c-d271153a31a8 | ce975169-a0ab-40ce-8e32-efa28d06079d

# Calculate statistics per model
stats = combined_df.groupby('model').agg({
    'feedback.google_bleu': ['mean', 'std'],
    'execution_time': ['mean', 'std']
}).round(4)

# Sort by execution time
stats = stats.sort_values(('execution_time', 'mean'))

# Create a figure with two subplots side by side
import matplotlib.pyplot as plt
import numpy as np

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

# Define colors for each model
colors = ['#2ecc71', '#3498db', '#e74c3c']
models = stats.index

# Plot for Google BLEU scores
bleu_means = stats[('feedback.google_bleu', 'mean')]
bleu_stds = stats[('feedback.google_bleu', 'std')]

for i, model in enumerate(models):
    ax1.errorbar(i, bleu_means[model], yerr=bleu_stds[model],
                fmt='o', color=colors[i], markersize=8, capsize=5,
                label=model)
    if i > 0:
        ax1.plot([i-1, i], [bleu_means[models[i-1]], bleu_means[model]],
                 '-', color=colors[i], alpha=0.5)

ax1.set_ylabel('Google BLEU Score')
ax1.set_title('Google BLEU Scores by Model')
ax1.set_xticks(range(len(models)))
ax1.set_xticklabels(models, rotation=45)
ax1.set_ylim(bottom=0)  # Set y-axis to start at 0
ax1.legend()
ax1.grid(True, alpha=0.3)

# Plot for execution times
exec_means = stats[('execution_time', 'mean')]
exec_stds = stats[('execution_time', 'std')]

for i, model in enumerate(models):
    ax2.errorbar(i, exec_means[model], yerr=exec_stds[model],
                fmt='o', color=colors[i], markersize=8, capsize=5,
                label=model)
    if i > 0:
        ax2.plot([i-1, i], [exec_means[models[i-1]], exec_means[model]],
                 '-', color=colors[i], alpha=0.5)

ax2.set_ylabel('Execution Time (seconds)')
ax2.set_title('Execution Times by Model')
ax2.set_xticks(range(len(models)))
ax2.set_xticklabels(models, rotation=45)
ax2.set_ylim(bottom=0)  # Set y-axis to start at 0
ax2.legend()
ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

# Display the statistics table
print("\nDetailed Statistics:")
print(stats)

Detailed Statistics:
              feedback.google_bleu         execution_time
                              mean     std           mean     std
model
gpt-4o-mini                 0.4038  0.0453         0.7815  0.0433
gpt-3.5-turbo               0.3768  0.0424         0.8343  0.2208
gpt-4-turbo                 0.3519  0.0775         0.9122  0.1482

The evaluation results show interesting differences between the models:

GPT-3.5-turbo achieved a Google BLEU score of 0.377 (±0.042) with average execution time of 0.83s (±0.22s)

GPT-4-turbo scored slightly lower at 0.352 (±0.078) and was slower at 0.91s (±0.15s)

GPT-4o-mini performed best with a BLEU score of 0.404 (±0.045) while being fastest at 0.78s (±0.04s)

As expected, results suggest that the newer GPT-4o-mini model achieves better quality while maintaining lower latency compared to both GPT-3.5 and GPT-4 turbo variants. The standard deviations indicate that GPT-4-turbo has the most variable output quality, while GPT-4o-mini is most consistent in both quality and speed. Interestingly, the more advanced gpt-4-turbo model has lower BLEU scores but takes longer to execute. This suggests that model size and computational complexity don’t necessarily correlate with better performance on this specific summarization task. Of course, in practice at the enterprise leverl, our task would be more complex and we would suggest to increasthe number of experiment iterations.

Since we decided to upload result, we can also visualize the experiment results in LangSmith as shown in Figure 2-2.

Figure 2-2. LangSmith Experiment Results

Langsmith is an excellent platform for testing and evaluating LLMBAs. As we mentioned at the beginning, though, prompt engineering - the process of crafting effective prompt instructions for LLMs - is a crucial piece of any LLMBA. Therefore, evaluating prompts is a crucial piece of an evaluation framework. In the following section, we introduce a tool that is purpose-built for prompt evaluation, which emphasizes how important this is to any LLMBA project.

PromptFoo

Promptfoo [Reference to Come] is a lightweight, open-source framework primarily focused on prompt testing and comparison. It is designed specifically for prompt engineering workflows, allowing developers to quickly test multiple prompts against test cases to identify which variations perform best.

Key features of promptfoo include:

Automated Testing: Promptfoo provides automated testing capabilities, allowing developers to run custom evaluations tailored to their applications.

Custom Probes: Developers can create custom probes to focus on specific use cases for instance decoupling prompts from tests cases.

User-Friendly CLI: The framework features a command-line interface that supports live reloads and caching, facilitating rapid testing and iteration.

We will use promptfoo’s command line interface in the following examples. Please follow installation instructions here.

Before we get into the technical aspects of promptfoo, a brief aside on the importance of prompts. They are more than just important components of an LLMBA, they are the component where developers can and should collaborate with domain experts.

In our example summary of a 10-K, we are staying very general, but as LLMBAs become more advanced, they will become more nuanced - and that will require prompts that have been crafted by domain experts. What are the right questions to ask of 10-K? What can an LLM tell us that is novel? What sections of a 10-K are the most important? We want our LLMBAs to be as valuable as possible, and we certainly don’t want them answering prompts that are of no interest to our end users. Prompt collaboration is key to making LLMBAs successful.

Evals are defined in a configuration file promptfooconfig.yaml, which defines elements such as providers, prompts, test cases, and assertions. This file is what allows us to systematize our prompt testing.

In the following example, we will perform a two-step evaluation:

Evaluate the performance of different LLM models given a set of constraints.

Evaluate the quality of different prompts for the best performing model from 1.

Note that we are making a prescriptive choice here, landing on a model first and then testing prompts.

Evaluating Models with Promptfoo

import yaml

# Read the YAML file
with open('promptfoo/model_comparison/promptfooconfig.yaml', 'r') as f:
    config = yaml.safe_load(f)

# Pretty print the YAML content
print(yaml.dump(config, default_flow_style=False, sort_keys=False))

description: Best model eval
prompts:
- file://prompt1.txt
providers:
- openai:gpt-4o-mini
- openai:gpt-4
- openai:gpt-3.5-turbo
defaultTest:
  assert:
  - type: cost
    threshold: 0.001
  - type: latency
    threshold: 1000
  - type: python
    value: len(output) < 200
  - type: llm-rubric
    value: Does the summary look like it was written by an expert analyst [Yes/No]?
tests: file://tests.csv

The configuration file shows how PromptFoo can be used to evaluate different LLM models. The YAML configuration defines three providers (GPT-4o-mini, GPT-4, and GPT-3.5-turbo) and sets up test assertions to validate their outputs. These assertions check important constraints:

Cost efficiency: Each inference must cost less than $0.001

Latency requirements: Response time must be under 1000ms

Output length: Generated text must be less than 200 characters

Output quality: An LLM-based rubric evaluates if the output appears to be written by an expert (uses openai’s gpt-4o model)

Crucially, our prompts are loaded from an external file (prompt1.txt) and test cases are defined in tests.csv.

This structured approach enables systematic evaluation of model performance and prompt performance. If the world conisders the crafting of prompts to be prompt engineering, then we need to treat these inputs systematically.

We are ready to run our evaluation. The short command below will run the evaluation and store the results in eval.json while making sure that the evaluation is not cached so we are measuring actual latency of the LLMs.

promptfoo eval --no-cache --output eval.json

Next we process that evaluation and extracts key metrics from the JSON file, including:

Provider name (e.g. gpt-4, gpt-3.5-turbo)

Latency in milliseconds

Token usage statistics

Cost per request

Number of passed/failed assertions

Prompt token count

Total number of API requests

import json
import pandas as pd

# Read the JSON file
with open('promptfoo/model_comparison/eval.json', 'r') as f:
    eval_data = json.load(f)

# Extract results into a list of dictionaries
results = []
for prompt in eval_data['results']['prompts']:
    result = {
        'provider': prompt['provider'],
        'latency_ms': prompt['metrics']['totalLatencyMs'],
        'token_usage': prompt['metrics']['tokenUsage']['total'],
        'cost': prompt['metrics']['cost'],
        'assert_pass': prompt['metrics']['assertPassCount'],
        'assert_fail': prompt['metrics']['assertFailCount'],
        'prompt_tokens': prompt['metrics']['tokenUsage']['prompt'],
        'num_requests': prompt['metrics']['tokenUsage']['numRequests']
    }
    results.append(result)

from IPython.display import display, Markdown

# Convert to DataFrame
df = pd.DataFrame(results)
print(df)

Provider

Latency (ms)

Token Usage

Cost

Assert Pass

Assert Fail

Prompt Tokens

Num Requests

openai:gpt-4o-mini

2463

97

$0.000035

6

2

52

2

openai:gpt-4

3773

103

$0.004620

4

4

52

2

openai:gpt-3.5-turbo

1669

95

$0.000091

7

1

52

2

The evaluation results reveal interesting performance characteristics across different OpenAI models. GPT-3.5-turbo demonstrates the best overall performance given our criteria with the lowest latency (1669ms), lowest token usage (95), and highest number of passed assertions (7). While GPT-4 shows higher token usage (103) and latency (3773ms), it also has the highest cost per request ($0.00462). The GPT-4-mini variant offers a middle ground, with moderate latency and token usage, while maintaining relatively good assertion performance (6 passes).

These results suggest that for this particular evaluation task, GPT-3.5-turbo provides the best balance of performance, reliability, and cost-effectiveness.

Promptfoo also offers a web interface for visualizing the evaluation results as shown in Figure 2-3.

promptfoo view

We can observe results per test case (i.e. section of the SEC filing) and per provider. Humans can also manually review the results and provide feedback as well as generate new test cases.

Figure 2-3. PromptFoo evaluation results showing performance metrics across different models.

Now that we have established GPT-3.5-turbo as our model of choice given the minimum required criteria based on cost, latency and basic qualitative evaluation, we can compare the performance of different prompts as a next evaluation step. Can we improve the quality of the summaries by using different prompts?

Evaluating Prompts

First, we redefine our evaluation criteria. We now select the prompt that delivers the most “detailed” summaries. Hopefully this gives an idea of how creative and complicated things can be when we start to consider prompt evalution. Perhaps detail is less important than quality, concision, verbosity, style etc. And the number of prompts we can test is limitless (whereas the number of LLMs, though seeming to grow every few months, is indeed finite). Let’s peek at our updated promptfoo configuration file.

# Read the YAML file
with open('promptfoo/prompt_comparison/promptfooconfig.yaml', 'r') as f:
    config = yaml.safe_load(f)

# Pretty print the YAML content
print(yaml.dump(config, default_flow_style=False, sort_keys=False))

description: Best model eval
prompts:
- file://prompt1.txt
- file://prompt2.txt
- file://prompt3.txt
providers:
- openai:gpt-3.5-turbo
defaultTest:
  assert:
  - type: llm-rubric
    value: 'Evaluate the output based on how detailed it is.  Grade it on a scale
      of 0.0 to 1.0, where:

      Score of 0.1: Not much detail.

      Score of 0.5: Some detail.

      Score of 1.0: Very detailed.

      '
tests: file://tests.csv

Note that we are now passing 3 different prompts. And we have updated our assertions to check if the output is “detailed” by leveraging promptfoo’s llm-rubric assertion, which will run an LLM-as-a-Judge for evaluation.

Now, let’s define the 3 prompt variations we would like to test.

# Display the prompt variations
from IPython.display import display, Markdown

prompt_files = ['prompt1.txt', 'prompt2.txt', 'prompt3.txt']
prompt_content = []

for file in prompt_files:
    with open(f'promptfoo/prompt_comparison/{file}', 'r') as f:
        content = f.read().strip()
        prompt_content.append(f"### {file}\n---\n{content}\n")

display(Markdown("\n\n".join(prompt_content)))

prompt1.txt

‘Generate a 1-liner summary of the Section {{section}} from an SEC filing: {{content}}’

prompt2.txt

‘ROLE: You are a financial analyst. TASK: Generate a 1-liner summary of the Section {{section}} from an SEC filing: {{content}}’

prompt3.txt

‘ROLE: You are a financial analyst. REQUIREMENTS: BE DETAILED. TASK: Generate a 1-liner summary of the Section {{section}} from an SEC filing: {{content}}’

The first prompt matches our previous prompt. The second prompt adds a “financial analyst” role to the prompt. The third prompt expands on second prompt and add a requirement “BE DETAILED”.

We can now run the evaluation again.

promptfoo eval --output eval.json

# Read the evaluation results from JSON file
import json
with open('promptfoo/prompt_comparison/eval.json', 'r') as f:
    eval_data = json.load(f)

# Create a list to store the data
data = []

# Extract results for each test case
for result in eval_data['results']['results']:
    section = result['vars']['section']
    prompt_id = result['promptId']
    score = result['gradingResult']['score'] if 'gradingResult' in result else 0.0

    # Find the corresponding prompt file
    for prompt in eval_data['results']['prompts']:
        if prompt['id'] == prompt_id:
            prompt_file = prompt['label'].split(':')[0]
            break

    # Add to data list
    data.append([section, prompt_file, score])

# Convert to DataFrame
df_raw = pd.DataFrame(data, columns=['Section', 'Prompt', 'Score'])

# Pivot to get desired format
df = df_raw.pivot(index='Section', columns='Prompt', values='Score').reset_index()
df = df[['Section', 'prompt1.txt', 'prompt2.txt', 'prompt3.txt']]

display(Markdown("### Prompt Comparison Results by Section"))
print(df)

Prompt Comparison Results by Section

Prompt            Section  prompt1.txt  prompt2.txt  prompt3.txt
0       Legal Proceedings          0.1          0.5          1.0
1            Risk Factors          0.1          0.5          0.5

The results show that prompt3.txt performs best for Legal Proceedings sections, achieving a perfect score of 1.0 compared to 0.5 for prompt2.txt and 0.1 for prompt1.txt. For Risk Factors sections, both prompt2.txt and prompt3.txt achieve moderate scores of 0.5, while prompt1.txt scores poorly at 0.1. This suggests that prompt3.txt is generally more effective at extracting detailed information, particularly for legal content. Is that what we were expecting? We have discovered that prompts not only matter, they matter with varying degree for different types of content.

We can use this information to build better LLMBAs, and also train our end users, who will be prompting our models. A general takeaway here is that defining a role (financial analyst) and asking for a detailed response tend to yield better, more detailed results. In short, if we want our output to have a certain style, it seems LLMs perform better when given specific instructions about the desired style.

Furthermore, no matter whether LLMBA developers wishe to use promptfoo, a plan for the systematic evaluation of prompts is absolutely crucial for enterprise level success, especially since prompting is the most accessible way for non-technical personas to interact with LLMBAs.

LightEval

The last evaluation tool we cover is lighteval [Reference to Come], an open-source framework designed to streamline and standardize the evaluation of LLMBAs. It provides developers with reproducible methods to measure model performance on multiple dimensions including accuracy, reasoning, and safety.

However, instead of repeating our previous example on evaluating summaries, let’s change course a bit and consider a qualitatively different task: running an econometric analysis on financial data. This is a very different challenge for LLMs and LLMBAs, because they are not trained to be experts in econometric analysis, whereas they are trained to be experts in linguistic analysis. We include this example not just to introduce an interesting and different challenge, but also because at the enterprise level we will inevitably be asked to run and track this experiment.

First, since we are leaving the world of language and no longer can use BLEU, we need to select a benchmark to assess LLMs capabilities in this domain. MMLU has a sub-benchmark called Econometrics we can use for this task. Table 2-1 shows a sample of the benchmark dataset from MMLU Econometrics. It consists of multiple-choice questions from econometrics and expected answers.

Table 2-1. MMLU Econometrics Task Dataset sample

Question

Options

Correct Options

Correct Options Index

Correct Options Literal

Consider the following AR(1) model with the disturbances having zero mean and unit variance: yt = 0.2 + 0.4 yt-1 + ut The (unconditional) mean of y will be given by

[“0.2”, “0.4”, “0.5”, “0.33”]

[“b”]

[3]

[“0.33”]

Suppose that a test statistic has associated with it a p-value of 0.08. Which one of the following statements is true? (i) If the size of the test were exactly 8%, we…

[“(ii) and (iv) only”, “(i) and (iii) only”, “(i), (ii), and (iii) only”, “(i), (ii), (iii), and (iv)”]

[“c”]

[2]

[“(i), (ii), and (iii) only”]

What would be then consequences for the OLS estimator if heteroscedasticity is present in a regression model but ignored?

[“It will be biased”, “It will be inconsistent”, “It will be inefficient”, “All of (a), (b) and (c) will be true.”]

[“c”]

[2]

[“It will be inefficient”]

Suppose now that a researcher wishes to use information criteria to determine the optimal lag length for a VAR. 500 observations are available for the bivariate VAR…

[“1 lag”, “2 lags”, “3 lags”, “4 lags”]

[“c”]

[2]

[“3 lags”]

Now let’s start working with LightEval.

LightEval Setup

The code sample below demonstrates the LightEval Python SDK framework for evaluating a target LLM model on a given task. First, we instantiate an EvaluationTracker which manages result storage, in this example kept in a local directory output_dir, and tracks detailed evaluation metrics, optionally pushed to HuggingFace Hub.

Next, we instantiate an object of the class PipelineParameters which, in this example, configures the pipeline for parallel processing with a temporary cache in cache_dir also setting the maximum number of samples to process to max_samples. Then, in BaseModelConfig we set up the LLM model we would like to evaluate defined in pretrained.

pip install lighteval[accelerate]

import lighteval
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.model_config import BaseModelConfig
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
from lighteval.utils.utils import EnvConfig
from lighteval.utils.imports import is_accelerate_available
from datetime import timedelta
from accelerate import Accelerator, InitProcessGroupKwargs

def create_evaluation_pipeline(output_dir: str, cache_dir: str, pretrained: str, dtype: str = "float16", max_samples: int = 10, task: str):
    if is_accelerate_available():
        from accelerate import Accelerator, InitProcessGroupKwargs
        accelerator = Accelerator(kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))])
    else:
        accelerator = None

    evaluation_tracker = EvaluationTracker(
        output_dir=output_dir,
        save_details=True,
        push_to_hub=False
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.ACCELERATE,
        env_config=EnvConfig(cache_dir=cache_dir),
        override_batch_size=1,
        max_samples=max_samples
    )

    model_config = BaseModelConfig(
        pretrained=pretrained,
        dtype=dtype,
        use_chat_template=True,
        trust_remote_code=True
    )

    pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config
    )

    return pipeline

Figure 2-4 shows a schematic representation of its key components. As inference engine, we leverage accelerate for distributed evaluation. lighteval also supports other inference backends such as vllm and tgi.

Figure 2-4. LightEval Python SDK Sample Conceptual Overview.

This setup allows for systematic evaluation of LLM performance on specific tasks while handling distributed computation and result tracking.

The final Pipeline combines these components to evaluate in the user defined task, which follows the following format:

{suite}|{task}|{num_few_shot}|{0 or 1 to automatically reduce `num_few_shot` if prompt is too long}

The task string format follows a specific pattern with four components separated by vertical bars (|):

suite: The evaluation suite name (e.g., “leaderboard”)

task: The specific task name (e.g., “mmlu:econometrics”)

num_few_shot: The number of few-shot examples to use (e.g., “0” for zero-shot)

A binary flag (0 or 1) that controls whether to automatically reduce the number of few-shot examples if the prompt becomes too long

LightEval provides a comprehensive set of evaluation tasks [Reference to Come] and metrics [Reference to Come]. The available tasks span multiple categories and benchmarks including BigBench, MMLU, TruthfulQA, WinoGrande, and HellaSwag. The framework also supports standard NLP evaluation metrics including BLEU, ROUGE, Exact Match, F1 Score, and Accuracy if we wished to deploy an evaluation for our previous example of generating summaries.

Econometric Evaluation with MMLU

For our econometric challenge, we choose to evaluate our LLMs on the MMLU econometrics task using zero-shot learning. Hence, we define the task as follows:

task = "leaderboard|mmlu:econometrics|0|0"

Example usage to evaluate an LLM, for instance meta-llama/Llama-3.2-1B-Instruct, on the MMLU econometrics task using zero-shot learning:

task = "leaderboard|mmlu:econometrics|0|0"
model = "meta-llama/Llama-3.2-1B-Instruct"
pipeline = create_evaluation_pipeline(output_dir="./evals/", cache_dir="./cache/", pretrained=model, task=task)

We can then evaluate the pipeline, save and show its results as follows:

pipeline.evaluate()
pipeline.save_and_push_results()
pipeline.show_results()

The results are then stored in output_dir in JSON format.

The same results can be obtained by using the LightEval CLI:

lighteval accelerate --model_args "pretrained=meta-llama/Llama-3.2-1B-Instruct" --tasks "leaderboard|mmlu:econometrics|0|0" --override_batch_size 1 --output_dir="./evals/"

Comparing Multiple Models on Econometrics

We would like to compare the performance of multiple open source models on the MMLU econometrics task. While we could download and evaluate each model locally, we prefer instead to evaluate them on a remote server to save time and resources. LightEval enables serving the model on a TGI-compatible server/container and then running the evaluation by sending requests to the server [Reference to Come].

For that purpose, we can leverage HuggingFace Serverless Inference API ¹ and set a configuration file for LightEval as shown below, where <MODEL-ID> is the model identifier on HuggingFace (e.g. meta-llama/Llama-3.2-1B-Instruct) and <HUGGINGFACE-TOKEN> is the user’s HuggingFace API token. Alternatively, you could also pass an URL of a corresponding dedicated inference API if you have one.

model:
  type: "tgi"
  instance:
    inference_server_address: "https://api-inference.huggingface.co/models/<MODEL-ID>"
    inference_server_auth: "<HUGGINGFACE-TOKEN>"
    model_id: null

Now we can run the evaluation by sending requests to the server as follows by using the same bash command as before but now setting the model_config_path to the path of the configuration file we have just created (e.g. endpoint_model.yaml):

lighteval accelerate --model_config_path="endpoint_model.yaml" --tasks "leaderboard|mmlu:econometrics|0|0" --override_batch_size 1 --output_dir="./evals/"

To complete our task, we evaluate a few models from the following model families: Llama3.2, Qwen2.5, and SmolLM2 as described in Table 2-2.

Table 2-2. Model Families Evaluated Using LightEval

Model Family

Description

Models

References

Llama3.2 Instruct

LLaMA architecture-based pretrained and instruction-tuned generative models

Llama-3.2-1B-Instruct
Llama-3.2-3B-Instruct

[Reference to Come]

Qwen2.5 Instruct

Instruction-tuned LLMs family built by Alibaba Cloud

Qwen2.5-0.5B-Instruct
Qwen2.5-1.5B-Instruct
Qwen2.5-3B-Instruct

[Reference to Come]

SmolLM2 Instruct

Instruction-tuned family of compact language models built by HuggingFace

SmolLM2-360M-Instruct
SmolLM2-1.7B-Instruct

[Reference to Come]

We can then compare the performance of these models on the MMLU econometrics task as shown in Figure 2-5.

Figure 2-5. Model performance comparison on MMLU Econometrics task, showing accuracy scores across different model sizes and architectures.

The results reveal several interesting patterns in model performance. As expected, we observe a trend where larger models consistently achieve higher accuracy scores. The evaluation shows distinct clusters among model families, with Qwen2.5, Llama-3.2, and SmolLM2 each exhibiting their own scaling characteristics, suggesting that architectural differences lead to varying degrees of efficiency as model size increases. Particularly noteworthy is the performance of the Qwen2.5 family, which demonstrates superior accuracy even at smaller model sizes when compared to Llama-3.2.

Of course, the results should be taken with a grain of salt given the limited size of the dataset (MMLU Econometrics ~ 100), limited number of models and sizes. However, it gives a good indication of the capabilities of the different models tested with Qwen2.5 family being an interesting first candidate as a relatively small yet powerful model demonstrating a good trade-off between performance and size. Once tested on real-world data, the results will change but these initial findings are a good data-driven starting point for model selection as you begin your LLM-based application development.

In summary, LightEval is a simple yet flexible and comprehensive framework for evaluating LLMs across a wide variety of tasks and metrics. It can serve as a first step in selecting your next LLM for a specific task given the exponential growth in number of (open source) models available [Reference to Come]. Its integration with the Hugging Face ecosystem and modular architecture make it particularly powerful for evaluating open source models. For further details, visit the official repository [Reference to Come].

Frameworks Comparison

Before we conclude, let’s summarize the three frameworks we discussed in this chapter: langsmith, promptfoo and ligheval.

Table 2-3 provides a summary of each, assessed based on key features such as integration capabilities, customization options, ease of use, and the ability to facilitate human and LLM collaboration.

Table 2-3. Comparison of Lighteval, LangSmith, and Promptfoo

Feature/Aspect

Lighteval

LangSmith

Promptfoo

Integration

Seamless with Hugging Face models, easy access to multiple inference engines, and remote evaluation (e.g., TGI servers, HF serverless models)

User-provided models, evaluators, and metrics

CLI-based, user-provided models via YAML

Customization

Flexible task and metric support, quick evaluation against state-of-the-art leaderboards

Easy setup of custom tasks and metrics with plain vanilla Python functions, lacks predefined tasks and metrics

Default and user-provided probes, metrics, and assertions

Ease of Use

User-friendly, minimal setup

User-friendly, minimal setup, includes UI for result visualization

Simple CLI, rapid testing, includes UI for result visualization

Human/LLM Collaboration

Model-based evaluation

Model-based evaluation

Supports human and model evaluators

Conclusion

We have covered 3 open-source frameworks that let us put into practice the evaluation processes that we hand coded in the first Chapter. We find that, in practice, we use a combination of hand coded and framework-based tools. LangChain is very prominent in the world of LLMs, for evaluation but also for agent-based work and tool chaining. Promptfoo helps us focus on the importance of prompt engineering when building LLMBAs. That’s crucial because it offers a bridge to the non-technical community of business users. LightEval puts us in position to apply LLMs to a new set of important projects.

As we move into our next Chapter on unstructured data, keep in mind the evaluation frameworks that we have covered thus far. Each of these tools can be and should be applied as part of our applications to unstructured data.

¹ We found a bug in LightEval that prevented it from working with the HuggingFace Serverless Inference API: . Thanks to the great work of the LightEval team, this issue has been fixed.

Chapter 3. Unstructured Input Data

One home run is much better than two doubles.

—Steve Jobs

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at jbleiel@oreilly.com.

Introduction

In Chapters 1 and 2, we examined several methodologies and tools for evaluating the performance of LLMs and LLMBAs, including measuring the performance of different models and prompts. The focus was on evaluating the results of our LLMBAs. To put it another way, the focus was on the unstructured, textual answers of our LLMBAs. In our example, this was the summary of the 10-K.

In this Chapter, we turn our attention away from the results, and on to the data that gets passed to those LLMBAs, specifically the unstructured, textual input data.

More specifically, we will focus on two main topics around input data:

Data Pre-Processing: Parsing and chunking documents into a unified format that is suitable and manageable for LLMs to process effectively.

Retrieval Augmented Generation (RAG): Retrieving specialized, relevant data to Augment the answers that are Generated by LLMs. This is currently the most popular implementation of LLMBAs at the enterprise level.

In our example, that input data that needs to be pre-processed was the 10-K, the long source document that we wished for the LLM to summarize. However, we did not pay much attention to that document when running our evaluation framework. We simply asked a question of the first few pages (due to token limits) of the 10-K and got an answer.

In reality, we won’t want to be limited to the first few pages of a document or corpus of unstructured data. This is an important issue because LLMBAs are typically answering questions or generating insights from data that the relevant LLM has never seen before; in other words, the LLM has not been pretrained on the data, and that’s why we are building an application to unearth insights in that data.

We can verify that LLMs can’t answer questions about unseen data by navigating to claude.ai, or chatgpt.com, or meta.ai, and asking the chat, “Please summarize the December 2024 10-K of Taming LLMs Inc?” That is a fictional company and the 10-K is also fictional. LLMs have never seen the 10-K before. The chat, therefore, will either hallucinate an answer or say something like “I’m sorry I don’t have access to that data”. However, we have that data (we invented the company and we synthetically created the 10-K) and we can feed that data, which is unstructured text data, to our LLMs. That requires two steps: pre-processing the 10-K and allowing our LLMs to search/access the pre-processed result.

Throughout the rest of this Chapter, we spend time on pre-processing (parsing and chunking data) and then the majority of our time optimizing how LLMs can access, retrieve and augment their answers with this pre-processed input data. We should note that someday we might be able to simply pass entire, multi-hundred page documents to LLMBAs, and let them pre-process, store and then “figure out” the most relevant sections for any task. Indeed, advances in long-context language models (LCMs) [Reference to Come] have already expanded the amount of information LLMs can process. But significant challenges remain in managing and effectively utilizing extended data inputs, such as:

LLMs are sensitive to input formatting and structure, requiring careful data preparation to achieve optimal results [Reference to Come].

LLMs operate with knowledge cutoffs, providing potentially outdated information that may not reflect current reality and demonstrate problems with temporal knowledge accuracy [Reference to Come].

LLMs also face “lost-in-the-middle” problems [Reference to Come] and struggle with less common but important information showing a systematic loss of long-tail knowledge [Reference to Come].

With those challenges still unsolved, this Chapter is organized around two key themes:

Data Pre-Processing: Parsing and chunking documents into a unified format that is suitable and manageable for LLMs to process effectively.

Retrieval Augmented Generation (RAG): Retrieving specialized, relevant data to Augment the answers that are Generated by LLMs. This is currently the most popular implementation of LLMBAs at the enterprise level.

In the data pre-processng section, we explore some useful open source tools - PyPDF2, Docling and MarkItDown - that help parse data into LLM-compatible formats. We then introduce the importance of chunking data by way of a use a case study that discusses commmon chunking strategies and implements a particular technique called Chunking with Contextual Linking.

In the RAG section, where we spend the majority of our time, we explore how to take that pre-processed data and follow the three steps of RAG:

create dense vector embeddings of our chunks

run semantic search over those chunks in order to retrieve the most relevant chunks

feed those chunks to an LLM and generate an answer to our question of interest

We also wade into a popular debate over whether RAG will be needed in the future given the rise of long-context LLMs. In our last case study, we demonstrate how long-context windows can be used to extract insights from a large knowledge base, without the need for complex retrieval systems. We explore some additional relevant techniques such as prompt caching and response verification through citations using “Corpus-in-Context” (CIC) Prompting [Reference to Come].

By the Chapter’s conclusion, readers will possess knowledge of input data management strategies for LLMs and practical expertise in selecting and implementing appropriate approaches and tools for specific use cases.

Pre-Processing Documents

Data parsing and chunking play a critical role in LLMBA performance [Reference to Come]. This section explores open source tools for converting diverse data formats into standardized representations that LLMs can effectively process. By abstracting away format-specific complexities, they allow developers to focus on core application logic rather than parsing implementation details.

Let’s look at three of our favorite tools for extracting textual data, and then we’ll discuss chunking.

PyPDF2

PyPDF2 is a Python library that can be used to extract text from PDFs. It is compatible with Windows, MacOS and Linux and is simple to implement, but it struggles with complex PDFs.

In the example below, we build a short text extractor function called extract_text_from_pdf() and can see the appealing simplicity of getting started with PyPDF2.

import PyPDF2

def extract_text_from_pdf(pdf_path):
    # Open the PDF file in binary read mode
    with open(pdf_path, 'rb') as file:
        # Create a PDF reader object
        pdf_reader = PyPDF2.PdfReader(file)

        # Get total number of pages
        num_pages = len(pdf_reader.pages)

        # Initialize a string to store all text
        text = ""

        # Extract text from each page
        for page_num in range(num_pages):
            page = pdf_reader.pages[page_num]
            text += page.extract_text()

        return text

# Example usage
if __name__ == "__main__":
    pdf_path = "example.pdf"
    extracted_text = extract_text_from_pdf(pdf_path)
    print(extracted_text)

For simple PDFs like the meeting notes of a the Federal Reserve Open Market Committee (simple because these are meeting notes and don’t tend to include charts and graphs), PyPDF2 is a good place to start.

MarkItDown

MarkItDown [Reference to Come] is a Python package and CLI tool developed by Microsoft for converting various file formats to Markdown. It supports a wide range of formats including PDF, PowerPoint, Word, Excel, images (with OCR and EXIF metadata), audio (with transcription), HTML, and other text-based formats making it a useful tool for document indexing and LLM-based applications.

It has simple command line and Python API interfaces, batch processing capabilities, and Docker support. It also offers optimal LLM integration for enhanced image descriptions.

A simple example is shown below, and note we don’t have to create our own function. We simply call md.convert().

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("test.xlsx")
print(result.text_content)

Docling

Docling [Reference to Come] is a Python package developed by IBM Research for parsing and converting documents into various formats. It provides advanced document understanding capabilities with a focus on maintaining document structure and formatting.

It’s key features include:

Support for multiple document formats (PDF, DOCX, PPTX, XLSX, Images, HTML, etc.)

Advanced PDF parsing including layout analysis and table extraction

Unified document representation format

Integration with LlamaIndex and LangChain

OCR support for scanned documents

Simple CLI interface

Similar to markitdown, we don’t have to create our own function and can use the converter.convert() function for PDF extraction.

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("document.pdf")
print(result.document.export_to_markdown())

We like working with each of these libraries and will feature some work with them later in Chapter.

Chunking

Even after we have extracted text from, for example, a PDF, we still don’t have a manageable structure around that data. It could be one giant string, or if in a database, the entire string could be stored in one row. For a 10-K filing, that would be 150 pages of text compressed into one cell. In order for an LLM to work effectively with this data, the next step is to chunk or split or divide the text into smaller pieces.

There are different methods for chunking, and each of them might be appropriate for different situations. However, we can broadly group chunking strategies into the following types:

Fixed-size Chunking: This is the most common and straightforward approach to chunking. We simply decide the number of tokens in our chunk and, optionally, whether there should be any overlap between them. In general, we will want to keep some overlap between chunks to make sure that the semantic context doesn’t get lost between chunks. Compared to other forms of chunking, fixed-sized chunking is computationally cheap and simple to use since it doesn’t require the use of any specialized techniques or libraries.

Content-aware Chunking: This approach idenifies natural boundaries for chunking, such as sentences or parapraphs. This ensures that each contains an entire idea or thought. There are several approaches and tools available to do this, including naive splitting (e.g. splitting on periods), NLTK, and the Python library spaCy. A popular example of this is Recursive Chunking.

Recursive Chunking: Recursive chunking divides the input text into smaller chunks in a hierarchical and iterative manner using a set of separators.

Context-aware Chunking: This strategy aims to use the surrounding information for each chunk in order to maintain the meaning of the chunk. A popular example of this is Semantic Chunking.

Semantic Chunking: This is a class of methods that leverages embeddings to extract the semantic meaning present in your data, creating chunks that are made up of sentences that talk about the same theme or topic.

To learn more about the importance of chunking, see this research [Reference to Come] which discusses the importance of chunking, and how it can be more important than model choice for improving LLMBA performance.

As we consider which chunking strategy to use in particular projects, we should recall Chapters 1 and 2 and the work we did to evaluate LLMBAs. If our LLMBA can be improved with chunking, then we can and should evaluate performance with different chunking strategies. We can consider this an example of data quality affecting model performance.

Chunking is crucial to the performance of LLMBAs and we find it remarkable that AI researchers study how to split up text - when most people think of AI research, this is probably not what they imagine. This is a great reminder of the importance of data, even to powerful models like foundation LLMs.

Our well-chunked data is now ready to be passed to our LLM, and this initiates what has become the most important or at least the most popular implementation of enterprise AI: RAG!

Retrieval Augmented Generation

Retrieval Augmented Generation has the entered the enterprise AI lexicon in much the same way that GenAI has entered the popular lexicon. Indeed, RAG chatbots are the most prominent implementation of LLMBAs in the business world. What is RAG and why is it important to enterprise?

RAG is how LLMs answer questions using timely and private data. LLMs need timely data because they have a cutoff data for their pretraining. Any LLM we are using was likely trained months ago - potentially over a year ago. LLMs need private data because they are pretrained on public data. They are very smart but totally ignorant of an enterprise’s sensitive data. Try asking an LLM what your ATM pin code, or less terrifying, try asking an LLM the fake 10-K for Taming LLMs Inc.

The LLM will either hallucinate/fabricate an answer in a very convincing way, or politely explain that it cannot answer those questions. Neither of those responses is of use to a financial analyst that wishes to know the contents of our 10-L. Thus, we must retrieve the text of the 10-K, and augment the LLM’s knowledge so that it can generate an accurate answer [Reference to Come] (and see here on preventing hallucinations [Reference to Come]).

Before we can do that, of course, we need to pre-process the report and chunk that report, as we covered in the previous sections. Then we pass the relevant chunks to the LLMBA for an insightful answer. We will cover how to do this in detail but first, here a few use cases for RAG:

Enterprise Knowledge Management: RAG enables organizations to synthesize answers from diverse internal data sources like documents, databases, and communication channels. This creates a unified knowledge interface that can accurately answer questions using the organization’s own data.

Document Processing and Analysis: RAG excels at extracting and analyzing information from complex documents like financial reports, presentations, and spreadsheets. The system can enable LLMs to understand context and relationships across different document types and formats.

Intelligent Customer Support: By combining knowledge bases with conversational abilities, RAG powers chatbots and support systems that can maintain context across chat history, provide accurate responses, and handle complex customer queries while reducing hallucinations.

Domain-Specific Applications: RAG allows LLMs to be equipped with specialized knowledge in fields like medicine, law, or engineering by retrieving information from domain-specific literature, regulations, and technical documentation. This enables accurate responses aligned with professional standards and current best practices.

Code Documentation and Technical Support: RAG can help developers by retrieving relevant code examples, API documentation, and best practices from repositories and documentation, which often suffer updates frequently, enabling more accurate and contextual coding assistance.

In the sections that follow, we walk through, in excruciating detail, how to set up and run a full RAG implementation and how to avoid common pitfalls around that pipeline. We will start with raw unstructured data, parse and chunk that data, create vector embeddings, run vector search, pass relevant data to our LLM, and generate an answer. Each of these steps is a chance to both avoid a pitfall and optimize our results. Let’s get to it!

RAG Pipeline

We will introduce key components of a RAG system one-by-one leading to a full, canonical RAG pipeline that ultimately will be used to answer the simple question: “Who’s the author of the book Taming LLMs?” (head to chatGPT and give it a shot. When we did so in early 2025, the hallucinated answer was “The book ‘Taming LLM’s’ is authored by G Arulkamaran…”. That is a made up person.) For our implementation, we will supply chapters of this book, which are stored in Github, to help our LLM answer this question.

The following basic components will be introduced (see Figure 3-1 for a visual representation):

Vector Database

Embeddings

Indexing

Retrieval System including re-ranking

LLM Augmented Generation via in-context learning

Preparing the Knowledge Base

Figure 3-1. Simplified RAG Pipeline

Enterprise LLMBAs require a knowledge base and RAG is no exception. In our case, the knowledge base is a set of documents, namely the web version of (some of the chapters of) this book:

Introduction

Unstructured Input Data (this very chapter)

Structured Output (the next chapter)

In practice, enterprises will have this sort of data already stored in a database and we would run this pipeline on top of that database. For reproducibility, we use the steps below and since we do not have a database, let’s write some short Python code to create a base of data.

book_url = "https://www.tamingllms.com/"
chapters = ["markdown/intro.html",
            "notebooks/input.html",
            "notebooks/structured_output.html"]

chapter_urls = [f"{book_url}/{chapter}" for chapter in chapters]
chapter_ids = [chapter.split("/")[-1].replace(".html", "") for chapter in chapters]

We use Docling to download the chapters from the web and parse them as markdown files.

chapters = [converter.convert(chapter_url).document.export_to_markdown() for chapter_url in chapter_urls]

Now that our data is extracted from those HTML files and pre-processed, we introduce the concept of vector embeddings.

Vector Embeddings

Vector embeddings are numerical representations of data (including text, images, audio, etc.) that capture meaning, allowing machines to process data quantitatively. Each embedding can be represented as a vector of floating-point numbers such that embedded data with similar meanings produce similar (i.e. close) vectors.

For example, below is a dense vector embedding of the sentence “Taming LLMs is a good book”. This vector embedding was created with the all-MiniLM-L6-v2 model.

[ 3.669006e-01 1.197412e-01 3.047852e-02 -1.240172e-01 8.357264e-02 -1.063523e-01 2.083982e-02 1.426935e-01 -7.017568e-02 -3.697810e-02 2.907842e-02 1.245549e-01 -9.644274e-02 -1.215142e-01 1.093394e-01 -5.251924e-02 1.542959e-01 -1.057501e-01 -5.631705e-02 -1.474581e-01 1.467494e-01 -1.629866e-01 1.157895e-01 -1.024882e-01 1.344323e-01 1.055799e-01 1.001332e-01 -1.441421e-01 -1.063846e-01 1.332421e-01 -1.006411e-01 -1.463549e-01 1.179045e-01 -1.185261e-01 1.017569e-01 -1.258191e-01 -1.372926e-01 -1.334741e-01 1.173611e-01 -1.430511e-01 1.439395e-01 -1.224506e-01 1.047399e-01 -1.277961e-01 -1.311911e-01 -1.302381e-01 1.192556e-01 -1.406977e-01 1.464381e-01]

Just as there are many LLMs, there are many embedding models. HuggingFace provides a leaderboard of embeddings models [Reference to Come], which are ranked by dimensions such as classification, clustering and reranking performance.

The paper “Representation Learning: A Review and New Perspectives” [Reference to Come] serves as an excellent reference for representation via embeddings. OpenAI also provides a good intro to Embeddings for developers [Reference to Come].

For our purposes, we will take the text we extracted from those HTML filings and covert them to vector embeddings: our corpus of text becomes a collection of numbers.

For text data, small distances among embeddings suggest high semantic relatedness and large distances suggest low semantic relatedness among the embedded texts. That’s helpful because if we want to find chunks that are similar to one another, we can search for vector embeddings with small distances among them. Why might that be useful?

Think back to the executive summary of the 10-K from earlier in the book. If we wanted to summarize only the portions of that document that discussed, e.g. risks from the environnment, we could create a vector embedding of all the chunks of the 10-K, then create a vector embedding of phrase “environment risks”, then find the distance among “environment risks” and all the chunks, and finally isolate just the 5 with the smallest distance. Those are the chunks that are semantically related to “environment risks”. Not to get ahead of ourselves, but then we would ask our LLM to summarise those 5 chunks into a report about those risks.

Creating vector embeddings can be compute intensive, so we do not want to recreate the embeddings each time we wish to query a document. Instead we store our embeddings in special database called a Vector Database.

Vector Databases

These databases are optimized for similarity search operations, making them ideal for embeddings-based retrieval systems.

A typical pipeline involving a vector database includes the following:

Input data is converted into “document chunks” forming a collection representing our knowledge base

Each chunk is converted into an embedding which are stored in the vector database

Embeddings are indexed in the vector database for efficient similarity search

The vector database is queried to retrieve the most relevant documents

The retrieved documents are used to answer questions

Vector databases are not strictly necessary for RAG systems. In fact, we can use a simple list of strings to store the chapters (or their chunks) and then use an LLM to answer questions about that list of strings.

However, vector databases are useful or for RAG applications as they enable:

Fast similarity search for finding relevant context

Efficient storage of document embeddings

Scalable retrieval for large document collections

Flexible querying with metadata filters

In that way, RAG applications can be seen as a retrieval system that uses a vector database to store and retrieve embeddings of documents, which in turn are used to augment LLMs with contextually relevant information, as we will see in the next sections.

Here, we will use ChromaDB [Reference to Come] as an example of an open source vector database but key features and concepts we cover are applicable to other vector databases, in general.

ChromaDB is a popular open-source vector database that offers:

Efficient storage and retrieval of embeddings

Support for metadata and filtering

Easy integration with Python applications

In-memory and persistent storage options

Support for multiple distance metrics

In ChromaDB, we can create a vector database client as follows.

import chromadb
chroma_client = chromadb.Client()

This will create a vector database in memory. We can also create a persistent vector database by specifying a path to a directory or alternatively by using a cloud-based vector database service like Snowflake, AWS, Azure or GCP.

We will use a vector database in memory for this example and that means we need to create a collection to store the embeddings of the chapters. And add our chapters as documents to the collection as follows.

collection = chroma_client.create_collection(name="taming_llms")

collection.add(
    documents=chapters,
    ids=chapter_ids
)

We are ready to query the collection. We write a simple function that takes the collection, input query and number of retrieved results as argument and returns the retrieved documents.

def query_collection(collection, query_text, n_results=3):
    results = collection.query(
        query_texts=[query_text],
        n_results=n_results
    )
    return results

We write a simple query, enquiring the purpose of the book.

q = "What is the purpose of this book?"
res = query_collection(collection, q)
res.get("ids")

[['intro', 'input', 'structured_output']]

As response, we obtain an object that contains several attributes including:

documents: The actual documents retrieved from the collection, i.e. the chapters

ids: The ids of the documents retrieved from the collection

distances: The distances of the documents to the query vector

We can see that the chapters “Introduction”, “Input” and “Structured Output” are retrieved from the collection ordered by their distance to the query vector, in increasing order.

We observe that the Introduction chapter is the most relevant one as it ranks first, followed by the Input and Structured Output chapters. Indeed, the purpose of the book is included in the Introduction chapter demonstrating the retrieval system successfully retrieved the most relevant document to the input query, in this simple example.

In order to understand how the retrieval system works and how the “distance to the query vector” is computed, we need to understand how embeddings are created and how documents are indexed.

Behind the scenes, ChromaDB is using the model all-MiniLM-L6-v2 by default ¹ to create embeddings for the input documents and the query (see Figure 3-2). This model is available in sentence_transformers [Reference to Come]. Let’s see how it works.

Figure 3-2. Embedding: From text to vectors.

from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer('all-MiniLM-L6-v2')

We replicate what ChromaDB did by embedding our chapters as well as input query using sentence transformers.

q = "What is the purpose of this book?"
docs_to_embed = [q] + chapters
embeddings = embedding_model.encode(docs_to_embed)
print(embeddings.shape)

(4, 384)

As a result, we obtain four 384-dimensional vectors representing our embeddings (one for each of the three chapters and one for the input query).

Now we can calculate similarity (that is, distance) among the embeddings. By default, sentence transformers uses cosine similarity as similarity metric.

similarities = embedding_model.similarity(embeddings, embeddings)
similarities

tensor([[1.0000, 0.4402, 0.3022, 0.4028],
        [0.4402, 1.0000, 0.6606, 0.5807],
        [0.3022, 0.6606, 1.0000, 0.6313],
        [0.4028, 0.5807, 0.6313, 1.0000]])

Let’s visualize the similarity matrix to better understand the relationships between our documents in Figure 3-3. The top row of the matrix represents the similarity of the input query against all chapters. That’s exactly what we previously obtained by querying ChromaDB which returned a response with documents ranked by similarity to the input query. As expected, the Introduction chapter is the most similar to the input query followed by the Input and Structured Output chapters, as we previously observed with ChromaDB.

Figure 3-3. Similarity matrix heatmap showing relationships among query and chapters.

Calculating similarity among embeddings can become computationally intensive if brute force is used, i.e. pair-wise computation, as the number of documents grows in the knowledge base. Indexing is a technique to help address this challenge.

Indexing

Indexing is an optimization technique that makes similarity searches faster and more efficient.

Without indexing, finding similar vectors would require an exhaustive search - comparing a query vector against every single vector in the database. For large datasets, this becomes prohibitively slow.

Common indexing strategies include:

Format this as a variable list, with the nameof each of the 4 strategies as the “headers” and the remaining text in paragraph (not bulleted) form. For example, this first one could be something like: These work by partitioning the vector space into hierarchical regions. They are effective for low-dimensional data, but they suffer from the “curse of dimensionality.” Examples include KD-trees and Ball trees.”

Tree-based Indexes: These work by partitioning the vector space into hierarchical regions and are effective for low-dimensional data but suffer from the “curse of dimensionality”. Examples include KD-trees and Ball trees

Graph-based Indexes: These create a multi-layered graph structure for navigation and offer excellent search speed but requires more memory. HNSW (Hierarchical Navigable Small World) is a prominent example.

LSH (Locality-Sensitive Hashing): This technique uses hash functions that map similar vectors to the same bucket and are more memory-efficient than graph-based methods. However, they may sacrifice some accuracy for performance.

Quantization-based Indexes: These compress vectors by encoding them into discrete values and reduce memory footprint significantly. They have a good balance between accuracy and resource usage

Indexing and basic vectors search work well for retrieving the chunks we wish to find, but there is one further optimization step in our toolbox: reranking.

Reranking

Retrieval systems solely based on vector similarity search might miss semantic relevance. That brings the need for techniques that can improve accuracy of the retrieval system. One such technique is reranking.

Reranking is a method that can improve accuracy of the retrieval system by reranking the retrieved documents based on their relevance to the input query. The steps are:

run the initial retrieval to get a preliminary set of results e.g. the top 20 most similar chunks

apply a reranking model, that is more advanced and computational expensive (but is being run a small set of 20 chunks)

pass a new relevance score to the preliminary results

rank the results by that new relevance score

Let’s go back to querying our vector database.

First, we write a query about how to get structured output from LLMs. Successfully retrieving the “Structured Output” chapter from the book as top result.

q = "How to get structured output from LLMs?"
res = query_collection(collection, q)
res.get("ids")

[['structured_output', 'input', 'intro']]

Next, we would like to obtain a tutorial on Docling, a tool we covered previously in this Chapter. However, we fail to obtain the correct chapter and instead obtain the “Introduction” chapter as a result.

q = "Docling tutorial"
res = query_collection(collection, q)
res.get("ids")

[['intro', 'input', 'structured_output']]

Thus, we use the sentence_transformers library to rerank the retrieved documents based on their relevance to the input query. We utilize the CrossEncoder model to rerank the documents. Cross-Encoder models are more accurate at judging relevance at the cost of speed compared to basic vector-based similarity.

We can implement a reranking step in a RAG system using a Cross-Encoder model in the following steps:

First, we initialize the Cross-Encoder model called ms-marco-MiniLM-L-6-v2, which is specficially trained for reranking and is designed to score the relevance between query-document pairs:

from sentence_transformers import CrossEncoder
model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2', max_length=512)

Then we perform the reranking by creating pairs of (query, document) for each retrieved document and predicting relevance scores for each pair Higher scores indicate better semantic match between query and document.

scores = model.predict([(q, doc) for doc in res["documents"][0]])

Finally, we select the best match using np.argmax(scores) finds the index of the highest scoring document:

print(res["documents"][0][np.argmax(scores)])

We obtain the following scores for the retrieved documents (“intro”, “unstructured_input”, “structured_output”), the higher the score, the more relevant the document is in relation to the input query.

array([-8.52623 , -6.328738, -8.750055], dtype=float32)

As a result, we obtain the index of the highest scoring document, which corresponds to the “input” chapter. Hence, the re-ranking step successfully retrieved the correct chapter.

print(res["ids"][0][np.argmax(scores)])

input

In RAG systems, the order of operations is typically:

ingest, pre-process and chunk text such as PDFs or HTML files

create vectore embeddings and store them in a vector databse

run semantic similarity on embeddings, which should be fast but potentially inaccurate

run reranking from the top-k results, which should be more accurate but slower

this balances the speed and accuracy of the retrieval system

Hence, instead of going over all retrieved documents:

scores = model.predict([(q, doc) for doc in res["documents"][0]])

We would run reranking on the TOPK results, where TOPK <<< number of documents:

scores = model.predict([(q, doc) for doc in res["documents"][0][:TOPK]])

LLMBAs with RAG

We are finally ready to use the retrieval system to help the LLM answer our authorship question. A common way to integrate RAG with LLMs is via in-context learning. With in-context learning the LLM learns from the retrieved documents by providing them in the context window as represented in context. This is accomplished via a prompt template structure as shown below. Note we are back to using prompts, as we detailed in Chapter 2, the promptfoo section. These in-context prompts have a large effect on the performance of our RAG-based LLMBAs.

Figure 3-4. RAG LLM with In-Context Learning

 rag_system_prompt_template = f"""
 You are a helpful assistant that answers questions based on the provided CONTEXT.

 CONTEXT: {context}
 """

 user_prompt_template = f"""
 QUESTION: {input}
 """

This prompt strategy demonstrates a common in-context learning pattern where retrieved documents are incorporated into the LLM’s context to enhance response accuracy and relevance. The prompt structure typically consists of a system prompt that:

Sets clear boundaries for the LLM to use information from the provided context

Includes the retrieved documents as context

Note that this again shows the importance of chunking: if the retrieved chunk is too large, our context becomes unwieldy and potentially confusing

This approach:

Reduces hallucination by grounding responses in source documents

Note that those source documents must be suited for our task, else we are simply misleading our own system

Improves answer relevance by providing contextually relevant information to the LLM

The context variable is typically populated with the highest-scoring document(s) from the retrieval step, while the input variable contains the user’s original query.

Here’s an example:

def RAG_qa(client, model, context, input):
    """
    Generate a summary of input using a given model
    """
    rag_system_prompt_template =  f"""You are a helpful assistant that answers questions based on the provided CONTEXT.

    CONTEXT: {context}
    """

    response = client.chat.completions.create(
    model=model,
        messages=[{"role": "system", "content": rag_system_prompt_template},
                 {"role": "user", "content": f"QUESTION: {input}"}]
    )
    return response.choices[0].message.content

First, we choose the LLM, gpt-4o-mini.

from dotenv import load_dotenv
import os

# Load environment variables from .env file
load_dotenv()

from openai import OpenAI
client = OpenAI()
model = "gpt-4o-mini"

Then, we run the retrieval step.

res = query_collection(collection, q)

Next, we run the re-ranking step setting it to consider the TOPK retrieved documents.

TOPK = 2
scores = model.predict([(q, doc) for doc in res["documents"][0][:TOPK]])
res_reranked = res["documents"][0][np.argmax(scores)]

We then pass the top document as context and invoke the LLM. Our application can now successful the simple question we posed earlier about the author of this book.

answer = RAG_qa(model, res_reranked[0], question)
answer

The author of the book "Taming LLMs" is Tharsis Souza.

RAG is currently the most prominent use case for LLMBAs and it’s important to be familiar with the plethora of choices to be made and then evaluated when building an application. RAG, however, can be implemented in many shapes and forms and entire books have been written about them. If more specialized techniques and architectures are needed, see RAG Cookbook (embedded link ) and Hands-On LLMs (embedded link) [Reference to Come].

RAG Challenges and Limitations

Not surprisingly, there are some challenges and limitations when using RAG and we briefly summarize those below.

Data Quality and Accuracy: The effectiveness of RAG systems fundamentally depends on the quality and reliability of their knowledge sources. When these sources contain inaccurate, outdated, biased, or incomplete information, the system’s responses become unreliable. This challenge is particularly acute when dealing with rapidly evolving topics or when sourcing information from unverified channels. However, when the sources are accurate and up-to-date, RAG excels at making rapidly changing data available.

Computational Cost and Latency: The process of embedding documents, maintaining vector databases, and performing similarity searches across large knowledge bases demands computational, and operational resources. In real-time applications, these requirements can introduce noticeable latency, potentially degrading the user experience and limiting practical applications. This work [Reference to Come] pinpoints several optimization opportunities to reduce computational and memory costs in RAG reducing the time to first token by up to 4x and increasing throughput by up to 2.1x compared to benchmarks.

Explainability and Evaluation: The complexity of RAG systems, arising from the intricate interaction between retrieval mechanisms and generative models, makes it difficult to trace and explain their reasoning processes. Traditional evaluation metrics often fail to capture the nuanced aspects of RAG performance, such as contextual relevance and factual consistency. This limitation hampers both system improvement and stakeholder trust. Before implementing RAG, design a process to explain and evaluate each step in the architecture.

Hallucination Management: Though RAG systems help ground LLM responses in source documents, they do not completely eliminate hallucinations. The generative component may still produce content that extrapolates beyond or misinterprets the retrieved context. This risk becomes particularly concerning when the system confidently presents incorrect information with apparent source attribution.

Moreover, recent research has shed light on critical limitations of key techniques used in RAG systems particularly with respect to reranking. See [Reference to Come] which discusses the following issues:

Diminishing Returns: Performance degrades as the number of documents (K) increases, sometimes performing worse than basic retrievers when dealing with large datasets.

Poor Document Discrimination: Rerankers can be misled by irrelevant documents, sometimes assigning high scores to content with minimal relevance to the query.

Consistency Issues: Performance and relative rankings between different rerankers can vary significantly depending on the number of documents being processed.

It’s not surprising that RAG faces these limitations as a new and evolving application paradigm. As we noted, it remains the most prominent implementation of LLMBAs, which both magnifies these challenges and presents an opportunity for us to add value by understanding and solving them. We point the user to additional resources if more specialized techniques and architectures are needed [Reference to Come].

RAG Case Study: Content Chunking with Contextual Linking

We’ve covered RAG extensively, so let’s dive into how we might apply some of these methods to a real world use case around our 10-K. We will also showcase how intelligent chunking strategies can overcome both context window and output token limitations. This case study illustrates techniques for breaking down and reassembling content while maintaining coherence, enabling the generation of high-quality long-form outputs despite model constraints.

Content chunking with contextual linking is a chunking technique that seeks to split input content while keeping chunk-specific context, hence allowing the LLM to maintain coherence and context when generating responses per chunks. In that way, this technique tackles two key problems:

The LLM’s inability to process long inputs due to context-size limits

The LLM’s inability to maintain coherence and context when generating responses per chunks

As a consequence, a third problem is also tackled: LLM’s inability to generate long-form content due to the max_output_tokens limitation. Since we generate responses per chunk, as we will see later, we end up with a solution that is capable of generating long-form content while maintaining coherence.

We implement this technique with the following steps:

Chunking the Content: The input content is split into smaller chunks. This allows the LLM to process each chunk individually, focusing on generating a complete and detailed response for that specific section of the input.

Maintaining Context with a Base Prompt: Each chunk is linked with contextual information from the previous chunks. This helps in maintaining the flow and coherence of the content across multiple chunks.

Constructing Dynamic Prompt Parameters: For each chunk, a prompt is generated that includes the chunk’s content and its context. This prompt is then used to generate the output for that chunk. This has been shown to improve the quality of the output in retrieval applications.

Combining the Outputs: The outputs of all chunks are combined to form the final long-form content.

Let’s examine an example implementation of this technique.

Generating long-form content

Our goal in this example is to generate a long-form report analyzing a company’s 10-K. In other words, our input is a 10-K, and our output is a long-form report, rather than the one-line summary we generated previously.

Figure 3 below illustrates the process we will follow for handling long-form content generation through Content Chunking with Contextual Linking. It shows how input content is first split into manageable chunks using a chunking function (e.g. CharacterTextSplitter with tiktoken tokenizer), then each chunk is processed sequentially while maintaining context from previous chunks. For each chunk, the system updates the context, generates a dynamic prompt with specific parameters, makes a call to the LLM chain, and stores the response. After all chunks are processed, the individual responses are combined with newlines to create the final report, effectively working around the token limit constraints of LLMs while maintaining coherence across the generated content.

Figure 3-5. Content Chunking with Contextual Linking Schematic Representation.

Let’s quickly examine each step.

Step 1: Chunking the Content

As discussed in the previous section on chunking, there are several strategies available to us, but here we will utilize langchain for a context-aware sentence-splitting strategy. Langchain offers several text splitters [Reference to Come] such as JSON-, Markdown- and HTML-based or split by token. We will use the CharacterTextSplitter with tiktoken as our tokenizer to count the number of tokens per chunk which we can use to ensure that we do not surpass the input token limit of our model.

def get_chunks(text: str, chunk_size: int, chunk_overlap: int) -> list:
    """
    Split input text into chunks of specified size with specified overlap.

    Args:
        text (str): The input text to be chunked.
        chunk_size (int): The maximum size of each chunk in tokens.
        chunk_overlap (int): The number of tokens to overlap between chunks.

    Returns:
        list: A list of text chunks.
    """
    from langchain_text_splitters import CharacterTextSplitter

    text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    return text_splitter.split_text(text)

Step 2: Maintaining Context with a Base Prompt

A base prompt template serves as a foundational structure for all chunks, ensuring consistency in the instructions and context provided to the language model. Our template includes the following parameters:

role: Defines the role or persona the model should assume.

context: Provides the background information or context for the task.

instruction: Specifies the task or action the model needs to perform.

input_text: Contains the actual text input that the model will process.

requirements: Lists any specific requirements or constraints for the output.

from langchain_core.prompts import PromptTemplate
def get_base_prompt_template() -> str:

    base_prompt = """
    ROLE: {role}
    CONTEXT: {context}
    INSTRUCTION: {instruction}
    INPUT: {input}
    REQUIREMENTS: {requirements}
    """

    prompt = PromptTemplate.from_template(base_prompt)
    return prompt

We write a simple function that returns an LLMChain, which is a langchain construct that allows us to chain together a combination of prompt templates, language models and output parsers. This will allow us to later dynamically invoke LLM calls with varying prompts, depending on the chunk we process. We also use the ChatLiteLLM interface, a wrappper around the lightweight and useful library LiteLLM which provides an unified interface to multi LLMs. This allows us to swap LLMs easily in the code.

from langchain_core.output_parsers import StrOutputParser
from langchain_community.chat_models import ChatLiteLLM

def get_llm_chain(prompt_template: str, model_name: str, temperature: float = 0):
    """
    Returns an LLMChain instance using langchain.

    Args:
        prompt_template (str): The prompt template to use.
        model_name (str): The name of the model to use.
        temperature (float): The temperature setting for the model.

    Returns:
        llm_chain: An instance of the LLMChain.
    """

    from dotenv import load_dotenv
    import os

    # Load environment variables from .env file
    load_dotenv()

    api_key_label = model_name.split("/")[0].upper() + "_API_KEY"
    llm = ChatLiteLLM(
        model=model_name,
        temperature=temperature,
        api_key=os.environ[api_key_label],
    )
    llm_chain = prompt_template | llm | StrOutputParser()
    return llm_chain

Step 3: Constructing Dynamic Prompt Parameters

Nex we write a function get_dynamic_prompt_template that constructs prompt parameters dynamically for each chunk. The prompt is dynamic because it’s a function of the chunk sequence in a three-fold way:

If it’s the very first chunk: The LLM should generate the Introduction of the Report.

If it’s the last part: The LLM should discuss INPUT and then generate the Conclusion of the Report.

Otherwise, LLM should summarize INPUT conditioned on CONTEXT.

from typing import Dict
def get_dynamic_prompt_params(prompt_params: Dict,
                            part_idx: int,
                            total_parts: int,
                            chat_context: str,
                            chunk: str) -> str:
    """
    Construct prompt template dynamically per chunk while maintaining the chat context of the response generation.

    Args:
        prompt_params (Dict): Original prompt parameters
        part_idx (int): Index of current conversation part
        total_parts (int): Total number of conversation parts
        chat_context (str): Chat context from previous parts
        chunk (str): Current chunk of text to be processed
    Returns:
        str: Dynamically constructed prompt template with part-specific params
    """
    dynamic_prompt_params = prompt_params.copy()
    # saves the chat context from previous parts
    dynamic_prompt_params["context"] = chat_context
    # saves the current chunk of text to be processed as input
    dynamic_prompt_params["input"] = chunk

    # Add part-specific instructions
    if part_idx == 0: # Introduction part
        dynamic_prompt_params["instruction"] = f"""
        You are generating the Introduction part of a long report.
        Don't cover any topics yet, just define the scope of the report.
        """
    elif part_idx == total_parts - 1: # Conclusion part
        dynamic_prompt_params["instruction"] = f"""
        You are generating the last part of a long report.
        For this part, first discuss the below INPUT. Second, write a "Conclusion" section summarizing the main points discussed given in CONTEXT.
        """
    else: # Main analysis part
        dynamic_prompt_params["instruction"] = f"""
        You are generating part {part_idx+1} of {total_parts} parts of a long report.
        For this part, analyze the below INPUT.
        Organize your response in a way that is easy to read and understand either by creating new or merging with previously created structured sections given in CONTEXT.
        """

    return dynamic_prompt_params

Step 4: Combining the Output: Report Generation

Finally, we write a generate_report function that takes input content and a model name, and generates the response report by calling the LLMChain with the dynamically updated prompt parameters for each chunk.

Specifically, the function steps are:

First, we initialize an empty list of report_parts to store segments of the report.

The input content is split into smaller chunks using a get_chunks function, with the specified size and overlap parameters.

We then set up chat_context with the initial input content. This will be updated as the report generation progresses.

We obtain an LLM chain with a base prompt template using get_llm_chain.

In the main loop, we iterate over each input chunk and invoke the LLM chain to obtain responses with the report part and update context with the cumulative response.

Finally, we join all report parts and return the complete report.

As we go through these steps, keep in mind how this workflow can be used by originators of research and consumers of research who might want a summary. In other words, when we write reseach in the future, it’s very possible that the first line of readers of that research will be other LLMs, not humans. How might this change the way in which we write and deliver research? For example, should we pre-chunk our research before shipping it?

def generate_report(input_content: str, llm_model_name: str,
                    role: str, requirements: str,
                    chunk_size: int, chunk_overlap: int) -> str:
    # stores the parts of the report, each generated by an individual LLM call
    report_parts = []
    # split the input content into chunks
    chunks = get_chunks(input_content, chunk_size, chunk_overlap)
    # initialize the chat context with the input content
    chat_context = input_content
    # number of parts to be generated
    num_parts = len(chunks)

    prompt_params = {
        "role": role, # user-provided
        "context": "", # dinamically updated per part
        "instruction": "", # dynamically updated per part
        "input": "", # dynamically updated per part
        "requirements": requirements #user-priovided
    }

    # get the LLMChain with the base prompt template
    llm_chain = get_llm_chain(get_base_prompt_template(),
                                 llm_model_name)

    # dynamically update prompt_params per part
    print(f"Generating {num_parts} report parts")
    for i, chunk in enumerate(chunks):
        dynamic_prompt_params = get_dynamic_prompt_params(
            prompt_params,
            part_idx=i,
            total_parts=num_parts,
            chat_context=chat_context,
            chunk=chunk
        )

        # invoke the LLMChain with the dynamically updated prompt parameters
        response = llm_chain.invoke(dynamic_prompt_params)

        # update the chat context with the cummulative response
        if i == 0:
            chat_context = response
        else:
            chat_context = chat_context + response

        print(f"Generated part {i+1}/{num_parts}.")
        report_parts.append(response)

    report = "\n".join(report_parts)
    return report

Setting up our example

Let’s harken back to our first example in this book, in which we wished to generate an executive summary of a 10-K. For simplicity we will run this example with a fixed 10,000 thousand token chunk size and no overlap, but recall that this document is 150-pages long - we encourage you to try different chunk sizes/techniques and overlaps and examine the quality. Ideally, try the evaluation methods from Chapter 1.

As an aside, think about how chunk size and overlap are two parameters that we are choosing, and how infinitely changeable they are, and in different combinations. This is reminiscent of hyperparameter tuning for a model such as XGBoost and a similar grid-search mentality would be effective for finding the optimal combination of chunk size/technqiue and overlap for a document like a 10-K. This is an area of skill in RAG that comes with experience.

With that in mind, let’s start our process. First we load the text of our 10-K and set our chunk size at 10000.

# Load the text from sample 10K SEC filing
with open('../data/apple.txt', 'r') as file:
    text = file.read()

# Define the chunk and chunk overlap size
MAX_CHUNK_SIZE = 10000
MAX_CHUNK_OVERLAP = 0

Next we create our generate_report() function, specifying our model and context.

report = generate_report(text, llm_model_name="gemini/gemini-1.5-flash-latest",
                           role="Financial Analyst",
                           requirements="The report should be in a readable, structured format, easy to understand and follow. Focus on finding risk factors and market moving insights.",
                           chunk_size=MAX_CHUNK_SIZE,
                           chunk_overlap=MAX_CHUNK_OVERLAP)

Finally, we generate our report.

# Save the generated report to a local file
with open('data/apple_report.md', 'w') as file:
    file.write(report)

In the code chunk below, we display our actual report and we can see the results below, starting with the “Introduction” that our LLM created to start summarizing the 10-K. This is the actual summary generated by our code.

# Read and display the generated report
with open('../data/apple_report.md', 'r') as file:
    report_content = file.read()

from IPython.display import Markdown

# Display first and last 10% of the report content
report_lines = report_content.splitlines()
total_lines = len(report_lines)
quarter_lines = total_lines // 10

top_portion = '\n'.join(report_lines[:quarter_lines])
bottom_portion = '\n'.join(report_lines[-quarter_lines:])

display(Markdown(f"{top_portion}\n\n (...) \n\n {bottom_portion}"))

Introduction

This report provides a comprehensive analysis of Apple Inc.’s financial performance and position for the fiscal year ended September 28, 2024, as disclosed in its Form 10-K filing with the United States Securities and Exchange Commission. The analysis will focus on identifying key risk factors impacting Apple’s business, evaluating its financial health, and uncovering market-moving insights derived from the provided data. The report will delve into Apple’s various segments, product lines, and services, examining their performance and contributions to overall financial results. Specific attention will be paid to identifying trends, potential challenges, and opportunities for future growth. The analysis will also consider the broader macroeconomic environment and its influence on Apple’s operations and financial outlook. Finally, the report will incorporate relevant information from Apple’s definitive proxy statement for its 2025 annual meeting of shareholders, as incorporated by reference in the Form 10-K.

PART 2: Key Risk Factors and Market-Moving Insights

This section analyzes key risk factors disclosed in Apple Inc.’s 2024 Form 10-K, focusing on their potential impact on financial performance and identifying potential market-moving insights. The analysis is structured around the major risk categories identified in the filing.

2.1 Dependence on Third-Party Developers:

Apple’s success is heavily reliant on the continued support and innovation of third-party software developers. The Form 10-K highlights several critical aspects of this dependence:

Market Share Vulnerability: Apple’s relatively smaller market share in smartphones, personal computers, and tablets compared to competitors (Android, Windows, gaming consoles) could discourage developers from prioritizing Apple’s platform, leading to fewer high-quality apps and potentially impacting customer purchasing decisions. This is a significant risk, especially given the rapid pace of technological change. A decline in app availability or quality could negatively impact sales and market share. Market-moving insight: Monitoring developer activity and app quality across competing platforms is crucial for assessing this risk. Any significant shift in developer focus away from iOS could be a negative market signal.

App Store Dynamics: While Apple allows developers to retain most App Store revenue, its commission structure and recent changes (e.g., complying with the Digital Markets Act (DMA) in the EU) introduce uncertainty. Changes to the App Store’s policies or fee structures could materially affect Apple’s revenue and profitability. Market-moving insight: Closely monitoring regulatory developments (especially concerning the DMA) and their impact on App Store revenue is essential. Any significant changes to Apple’s App Store policies or revenue streams could trigger market reactions.

Content Acquisition and Creation: Apple’s reliance on third-party digital content providers for its services introduces risks related to licensing agreements, competition, and pricing. The cost of producing its own digital content is also increasing due to competition for talent and subscribers. Failure to secure or create appealing content could negatively impact user engagement and revenue. Market-moving insight: Analyzing the success of Apple’s original content initiatives and the renewal rates of third-party content agreements will provide insights into this risk.

2.2 Operational Risks:

(…)

The reconciliation of segment operating income to consolidated operating income reveals that research and development (R&D) and other corporate expenses significantly impact overall profitability. While increased R&D is generally positive, it reduces short-term profits. The geographical breakdown of net sales and long-lived assets further emphasizes the concentration of Apple’s business in the U.S. and China. Market-moving insight: Continued weakness in the Greater China market, sustained flat iPhone sales, or any significant changes in R&D spending should be closely monitored for their potential impact on Apple’s financial performance and investor sentiment.

5.4 Auditor’s Report and Internal Controls:

The auditor’s report expresses an unqualified opinion on Apple’s financial statements and internal control over financial reporting. However, it identifies uncertain tax positions as a critical audit matter. The significant amount of unrecognized tax benefits ($22.0 billion) and the complexity involved in evaluating these positions highlight a substantial risk. Management’s assessment of these positions involves significant judgment and relies on interpretations of complex tax laws. Apple’s management also asserts that its disclosure controls and procedures are effective. Market-moving insight: Any changes in tax laws, unfavorable rulings on uncertain tax positions, or weaknesses in internal controls could materially affect Apple’s financial results and investor confidence.

Conclusion

This report provides a comprehensive analysis of Apple Inc.’s financial performance and position for fiscal year 2024. While Apple maintains a strong financial position with substantial cash reserves and a robust capital return program, several key risk factors could significantly impact its future performance. These risks include:

Dependence on third-party developers: A shift in developer focus away from iOS or changes to the App Store’s policies could negatively impact Apple’s revenue and profitability.

Operational risks: Employee retention challenges, reseller dependence, and cybersecurity threats pose significant operational risks.

Legal and regulatory risks: Ongoing antitrust litigation, the Digital Markets Act (DMA) compliance, and data privacy regulations introduce substantial legal and regulatory uncertainties.

Financial risks: Volatility in sales and profit margins, foreign exchange rate fluctuations, credit risk, and tax risks could impact Apple’s financial performance.

Supply chain concentration: Apple’s reliance on a concentrated network of outsourcing partners, primarily located in a few Asian countries, and dependence on single or limited sources for certain custom components, exposes the company to significant supply chain risks.

Uncertain tax positions: The significant amount of unrecognized tax benefits represents a substantial uncertainty that could materially affect Apple’s financial results.

Despite these risks, Apple’s strong liquidity position, continued growth in its Services segment, and robust capital return program provide a degree of resilience. However, investors and analysts should closely monitor the market-moving insights identified throughout this report, including developer activity, regulatory developments, regional economic conditions, supply chain stability, and the resolution of uncertain tax positions, to assess their potential impact on Apple’s future performance and valuation. The significant short-term obligations, while manageable given Apple’s cash position, highlight the need for continued financial discipline and effective risk management. A deeper, more granular analysis of the financial statements and notes is recommended for a more complete assessment.

Our Evalution of the Summary

We can read the report above and draw some conclusions/make some judgements about the report. These are our opinion, the reader can judge the results according to personal preferences. We find the coherence and the adherence to be strengths of the report.

Coherence: The generated report demonstrates coherence in that the sections are logically structured and the flow of information is smooth (again, this is our opinion). Each part of the report builds upon the previous sections, providing a comprehensive analysis of the company’s financial performance and key risk factors. The use of headings and subheadings helps in maintaining clarity and organization throughout the document.

Adherence to Instructions: Our application followed instructions effectively. The report is in a readable, structured format, and it focuses on identifying risk factors and market-moving insights as requested. The analysis is detailed and covers various aspects of the company’s financial performance, including revenue segmentation, profitability, liquidity, and capital resources. The inclusion of market-moving insights adds value to the report, aligning with the specified requirements.

Despite the useful qualities of the results, there are some limitations to consider:

Depth of Analysis: While the report covers a wide range of topics, the depth of analysis in certain sections may not be as comprehensive as a human expert’s evaluation. Some nuances and contextual factors might be overlooked by the LLM.

Chunking Strategy: The current approach splits the text into chunks based on size, which ensures that each chunk fits within the model’s token limit. However, this method may disrupt the logical flow of the document (in this case it didn’t, in our opinion, but this does remain as a risk), as sections of interest might be split across multiple chunks. An alternative approach could be semantic chunking, where the text is divided based on meaningful sections or topics. This would preserve the coherence of each section, making it easier to follow and understand. Implementing semantic chunking requires additional preprocessing to identify and segment the text appropriately, but it can significantly enhance the readability and logical flow of the generated report.

The impresssions above are our subjective, non-rigorous opinions of the report, which is to say our opinion of the output of our RAG LLMBA. A more rigorous alternative wold be to utilize the techniques in Chapter 1 to evaluate this application.

Note how we have introduced several new components that could affect the performance: extracting the text from the source documents, chunking technique and parameter choice, the prompt template, the choice of LLM itself. When we evaluate the LLMBA, it’s actually quite simple to toggle between different models of various size, cost and origin. Model evaluation is in fact the easy part because it’s inherently limited. Even if we wish to test 100 models, it’s bounded at 100. Imagine how many different prompt variations or chunking strategies we could test - it’s nearly infinite. This again is where experience and expertise will come to be important and our goal is to help all readers of this book build that invaluable expertise.

For example, we implemented a simple chunking strategy in our function. And to the uninitiated, chunking might seem pretty simple and certainly not an area of debate: just split up the text! But it’s an area of deep research for AI engineers. For example, this research from Snowflake https://www.snowflake.com/en/engineering-blog/impact-retrieval-chunking-finance-rag/ rigorously tests several chunking techniques. Anthropic, a leader in LLM building, introduced it’s own technique called Contextual Retrieval [Reference to Come]. Both of these are interesting research unto themselves but also emphasizes the extent to which cutting edge AI research touches retrieval in addition to model development.

Contextual Retreival, as shown in Figure 3-6, employs an LLM itself to generate relevant context per chunk before passing these two pieces of information together to the LLM. This process was proposed in the context of RAG to enhance its retrieval capabilities but can be applied more generally to improve output generation.

Figure 3-6. Anthropic Contextual Retrieval [Reference to Come].

Other Frameworks: Langchain and LlamaIndex

We have covered a few open source tools for parsing data and provided a canonical RAG pipeline directly using an open source vector database together with an LLM. There is a growing number of frameworks that offer similar functionality wrapping the same core concepts at a higher level of abstraction. The two most popular ones are Langchain and LlamaIndex.

For instance, the code below shows how to use LlamaIndex’s LlamaParse for parsing input documents, which offers support for a wide range of file formats (e.g. .pdf, .pptx, .docx, .xlsx, .html). We observe that the code is very similar to the one we used for MarkitDown and Docling.

from llama_parse import LlamaParse

# Initialize the parser
parser = LlamaParse(
    api_key="llx-your-api-key-here",
    result_type="markdown",  # Can be "markdown" or "text"
    verbose=True
)

documents = parser.load_data(["./doc1.pdf", "./doc2.pdf"])

As another example, the code below replicates our ChromaDB-based retrieval system using LlamaIndex [Reference to Come].

As we can see, similar concepts are used:

Documents to represent elements of the knowledge base

Collections to store the documents

Indexing of embeddings in the VectorDB and finally

Querying the VectorDB to retrieve the documents

import chromadb
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import StorageContext

# load some documents
documents = SimpleDirectoryReader("./data").load_data()

# initialize client, setting path to save data
db = chromadb.PersistentClient(path="./chroma_db")

# create collection
chroma_collection = db.get_or_create_collection("tamingllms")

# assign chroma as the vector_store to the context
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# create your index
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# create a query engine and query
query_engine = index.as_query_engine()
response = query_engine.query("Who is the author of Taming LLMs?")
print(response)

Frameworks are useful for quickly prototyping RAG systems and for building applications on top of them as they provide a higher level of abstraction and integration with third-party libraries. However, the underlying concepts are the same as the ones we have covered in this chapter. More often than not, problems arise when developers either do not understand the underlying concepts or fail to understand the details of the implement behind the abstractions provided by the framework. Therefore, it is recommended to try and start your implementation using lower level tools as much as possible and only when (i) the underlying problem as well as (ii) the desired solution are well understood, then consider moving to higher level frameworks if really needed.

Long-Context Models: RAG Killer?

There is a raging debate (in LLM land) over whether models will eventually have such long context windows that RAG will no longer be necessary. Imagine we could pass an entire book or 150-page 10-K in a context window and ask an LLM to write arbitrary questions, and expect the LLMBA to find the most relevant material for answering those arbitrary questions. No chunking or vector embeddings needed. It’s certainly possible but the world and indeed the authors of this book are divided over whether long-context models (LCMs) will be the end of RAG.

Recent research has shed light on this specific point [Reference to Come] suggesting a trade-off between cost and performance. On the one hand, RAG can be seen as a cost-effective alternative to LCMs:

RAG offers lower computational cost compared to LCMs due to the significantly shorter input length required for processing.

This cost-efficiency arises because RAG reduces the number of input tokens to LLMs, which in turn reduces overall usage cost.

On the other hand, this RAG benefit is achieved at the potential cost of performance, a problem which LCMs address.:

Recent advancements in LLMs, in particular with Gemini-1.5 and GPT-4o models, demonstrate capabilities in understanding long contexts directly, which enables them to outperform RAG in terms of average performance. But other recent research on chunking indicates RAG can still outperfrom LCMs [Reference to Come]).

LLMs can process extremely long contexts, such as Gemini 1.5 which can handle up to 1 million tokens, and these models benefit from large-scale pretraining to develop strong long-context capabilities.

This cost-performance trade-off is illustrated in the figure below, where LCMs outperform RAG in terms of average performance while RAG is more cost-effective.

Figure 3-7. Long-Context LLMs can demonstrate superior performance while RAG is more cost-effective [Reference to Come].

The figure also shows a model called “SELF-ROUTE” which combines RAG and LCMs by routing queries based on model self-reflection. This is a hybrid approach that reduces computational costs while maintaining performance comparable to LCMs. The advantage of SELF-ROUTE is most significant for smaller values of k, where k is the number of retrieved text chunks, and SELF-ROUTE shows a marked improvement in performance over RAG, while as k increases the performance of RAG and SELF-ROUTE approaches that of LCMs.

Another example of a hybrid approach that combines the benefits of both LCMs and RAG is RetroLLM [Reference to Come], which is a unified framework that integrates retrieval and generation into a single process, enabling language models to generate fine-grained evidence directly from a corpus. The key contribution is that this approach delivers those benefits while eliminating the need for a separate retriever, addressing limitations of traditional RAG methods. Experimental results demonstrate RetroLLM’s superior performance compared to traditional RAG methods, across both in-domain and out-of-domain tasks. It also achieves a significant reduction in token consumption due to its fine-grained evidence retrieval.

CAG [Reference to Come] is another solution that eliminates the need for RAG as it proposes cache-augmented generation (CAG). CAG preloads all relevant data into a large language model’s extended context window, eliminating the need for real-time retrieval and improving speed and accuracy. This is achieved by precomputing a key-value cache, further optimizing inference time. CAG demonstrates superior performance compared to RAG by achieving higher BERT scores in most evaluated scenarios, indicating better answer quality, and by having significantly reduced generation times. These results suggest that CAG can be both more accurate and more efficient than traditional RAG systems.

Another relevant development in this area is the introduction of LOFT [Reference to Come], a benchmark to assess RAG and LCMs, using real-world tasks requiring context up to millions of tokens. Evidence suggests LCs can deliver performance with simplified pipelines compared to RAG, particularly for tasking requiring multi-hop reasoning over long contexts when using Chain-of-Thought [Reference to Come]. However, LCMs can still be outperformed by specialized retrievers, in particular Gecko, a specialized model fine-tuned on extensive text retrieval and similarity tasks.

Overall, the world of AI and LLMs is changing rapidly in the direction of more power and performance. Ultimately, the choice among RAG, LCMs, or a hybrid method depends on the specific requirements of the task and available resources. When evaluating RAG Vs. LCMs, we keep these three summarized points in mind.

RAG may be relevant when cost-effectiveness is a key requirement and where the model needs to access vast amounts of external knowledge without incurring high computational expenses.

LCMS can outperform RAG in some scenarios, and the model needs to handle extensive texts that require deep contextual understanding and reasoning.

Hybrid approaches like SELF-ROUTE are valuable as they combine the strengths of RAG and LCMs offering a practical balance between cost and performance, especially for applications where both factors are critical.

LCM Case Study: Quiz Generation with Citations

To examine the utility of LCMs, let’s change course from our 10-K report generation and turn our attention to a denser, more complicated topic: the birth of democracy. Rather than generating a report on this subject, our goal is to build a Quiz generator for the reading list from Harvard’s class titled GOV 1039 “The Birth of Modern Democracy” (see Figure 3-8). If we glance at the syllabus online, we will notice that weekly readings are rather daunting, sometimes stretching to over a thousand pages. This presents the perfect use case for LCMs, a corpus of text that we humans can’t possibly digest in a small time frame.

Instead of trudging through these dense historical texts sequentially, we would like to use an LLBA to:

Extract key insights and connections between these documents, conversationally.

Engage with the material through a quiz format.

Add citations to help with verifying answers (also necessary for enterprise applications in regulated fields).

Our LLMBA will take the form of a Quiz Generator, where a user can ask quiz questions and receive back multiple choice quizzes modeled on the questions. The LLMBA also returns citations back to the primary source (helfpul for academic pursuits; necessary in regulated industries like banking). Why not simply ask for summaries?

In the first case, summarizing text might not lead us to the level of understanding we seek for an important topic like the birth of democracy. Secondly, our models have probably been trained on summaries of many of these documents already, which means copying those summaries won’t tell us much about how effective our methodology is or would be on a corpus of text the models hadn’t encountered before.

Our implementation will also incorporate prompt caching for efficiency and citation tracking to enhance accuracy and verifiability. We will use Gemini 1.5 Pro (experimental) as our LLM, which has a context window of 2M tokens. (The code for this is quite voluminous so we include only key aspects in print here. The full implementation is available at Book’s Github repository\footnote{\url{souzatharsis/tamingLLMs)

Figure 3-8. Harvard’s Democratic Theory Class

Implementation

Client

First, we will define the Client class which will provide the key interface where users interact with our app. It has the following summarized interface:

Initialization:

__init__(knowledge_base: List[str] = []): Initialize with optional list of URLs as knowledge base

Core Methods:

add_knowledge_base(urls: List[str]) -> None: Add URLs to the knowledge base

add(urls: List[str]) -> None: Extract content from URLs and add to conversation input

msg(msg: str = "", add_citations: bool = False) -> str: Enables users to send messages to the client

quiz(add_citations: bool = True, num_questions: int = 10) -> str: Generate a quiz based on full input memory

Key Attributes:

knowledge_base: List of URLs providing foundation knowledge

input: Current input being studied (short-term memory)

input_memory: Cumulative input + knowledge base (long-term memory)

response: Latest response from LLM

response_memory: Cumulative responses (long-term memory)

urls_memory: Cumulative list of processed URLs

Corpus-in-Context Prompting

The add() method is key since it is used to add content to the client. It takes a list of URLs and extracts the content from each URL using a content extractor (using MarkitDown). The content is then added to the conversation input memory in a way that enables citations using the “Corpus-in-Context” (CIC) Prompting [Reference to Come].

Figure 3-9 shows how the CIC format is used to enable citations. It inserts a corpus into the prompt. Each candidate citation chunk (e.g., passage, chapter) in a corpus is assigned a unique identifier (ID) that can be referenced as needed for the task at hand.

Figure 3-9. Example of Corpus-in-Context Prompting for retrieval.

CIC prompting leverages LLM’s capacity to follow instructions by carefully annotating the corpus with document IDs. It benefits from a strong, capable models to retrieve over large corpora provided in context.

    def add(self, urls: List[str]) -> None:
        self.urls = urls

        # Add new content to input following CIC format to enable citations
        for url in urls:
            self.urls_memory.append(url)
            content = self.extractor.convert(url).text_content
            formatted_content = f"ID: {self.reference_id} | {content} | END ID: {self.reference_id}"
            self.input += formatted_content + "\n"
            self.reference_id += 1

        # Update memory
        self.input_memory = self.input_memory + self.input

The method add_knowledge_base() is a simple wrapper around the add() method. It is used to add URLs to the knowledge base, which are later cached by the LLM model as we will see later.

    def add_knowledge_base(self, urls: List[str]) -> None:
        self.add(urls)

Later, when the user sends a message to the client, the msg() method is used to generate a response while enabling citations. self.content_generator is an instance of our LLM model, which we build next.

    def msg(self, msg: str = "", add_citations: bool = False) -> str:
        if add_citations:
            msg = msg + "\n\n For key statements, add Input ID to the response."

        self.response = self.content_generator.generate(
            input_content=self.input,
            user_instructions=msg
        )

        self.response_memory = self.response_memory + self.response.text

        return self.response.text

Prompt Caching

LLMBAs often involve repeatedly passing the same input tokens to a model, which can be inefficient and costly. Context caching addresses this by allowing us to cache input tokens after their first use and reference them in subsequent requests. This approach significantly reduces costs compared to repeatedly sending the same token corpus, especially at scale.

In our application, the user might pass a large knowledge base to the client that can be referenced multiple times by smaller user requests. Our Client class is composed of a LLMBackend class that takes the input_memory containing the entire knowledge base and any additional user added content.

self.llm = LLMBackend(input=self.input_memory)

In our LLMBackend Class, we leverage prompt caching on input tokens and use them for subsequent requests.

class LLMBackend:
    def __init__(self, model_name: str, input: str, cache_ttl: int = 60):
        self.cache = caching.CachedContent.create(
            model=model_name,
            display_name='due_knowledge_base', # used to identify the cache
            system_instruction=(
            self.compose_prompt(input, conversation_config)
        ),
        ttl=datetime.timedelta(minutes=cache_ttl),
    )

    self.model = genai.GenerativeModel.from_cached_content(cached_content=self.cache)

Quiz Generation

Returning to our Client class, we implement the quiz() method to generate a quiz based on the full input memory, i.e. the initial knowledge base and any additional user added content.

The quiz() method returns a Quiz instance which behind the scenes caches input tokens. The user can invoke its generate() method to generate a quiz passing the user instructions in msg parameter, as we will see later.

    def quiz(self, add_citations: bool = True, num_questions: int = 10) -> str:
        """
        Returns a quiz instance based on full input memory.
        """
        self.quiz_instance = Quiz(
                         input=self.input_memory,
                         add_citations=add_citations,
                         num_questions=num_questions)
        return self.quiz_instance

We write a simple prompt template for quiz generation:

ROLE:

You are a Harvard Professor providing a quiz. INSTRUCTIONS:

Generate a quiz with {num_questions} questions based on the input.

The quiz should be multi-choice.

Answers should be provided at the end of the quiz.

Questions should have broad coverage of the input including multiple Input IDs.

Level of difficulty is advanced/hard.

{citations}

STRUCTURE:

Sequence of questions and alternatives.

At the end provide the correct answers.

In this example, {citations} instructs the model to add CiC citations to the response if user requests it.

Example Usage

To see this in action, we will take advantage of Project Gutenberg to create our knowledge base from that list of historical texts. Let’s first define our knowledge base.

Harvard Class: GOV 1039 Syllabus

Class / Topic: “Rights”

Reading List:

ID 1. The Declaration of Independence of the United States of America

ID 2. The United States Bill of Rights

ID 3. John F. Kennedy’s Inaugural Address

ID 4. Lincoln’s Gettysburg Address

ID 5. The United States Constitution

ID 6. Give Me Liberty or Give Me Death

ID 7. The Mayflower Compact

ID 8. Abraham Lincoln’s Second Inaugural Address

ID 9. Abraham Lincoln’s First Inaugural Address

kb = [f"https://www.gutenberg.org/cache/epub/{i}/pg{i}.txt" for i in range(1,9)]

We will import our module gemini_duo as genai_duo and initialize the Client class with our knowledge base.

import gemini_duo as genai_duo
from IPython.display import Markdown, display

duo = genai_duo.Client(knowledge_base=kb)

At this point, we have converted each book into markdown using MarkitDown and cached the content in our LLM model. We can access how many tokens we have cached in our LLM model by looking at the usage_metadata attribute of the Gemini’s model response. At this point, we have cached at total of 38470 tokens.

Now, we can add references to our knowledge base at anytime by calling the add() method. We add the following references:

The Magna Carta

William Shap McKechnie on Magna Carta book

study_references = ["https://www.gutenberg.org/cache/epub/10000/pg10000.txt", "https://www.gutenberg.org/cache/epub/65363/pg65363.txt"]

duo.add(study_references)

Now we can instantiate a Quiz object and generate a quiz based on the full input memory.

quiz = duo.quiz(add_citations=True)
display(Markdown(quiz.generate()))

Figure 3-10 shows a sample quiz with citations. Marked in yellow are the citations which refer to the input IDs of the resources we added to the model.

Figure 3-10. Sample Quiz with Citations.

Discussion and Limitations

Our goal with this Quiz Generator was to:

build a knowledge base from multiple sources

leveraging prompt caching for efficiency

generate quizzes with citations for verifiability

We think the results are good but we did not build an accomplanying evaluation framework for this use case. How might we do this? On a question by question basis?

We do want to note that several limitations emerged during this process:

Memory Management: The system currently loads all content into memory, which could become problematic with larger knowledge bases. A more scalable approach might involve chunking or streaming the content.

Citation Quality: While the system provides citations, they lack specificity - pointing to entire documents rather than specific passages or page numbers. This limits the ability to fact-check or verify specific claims. Chunking again would help here.

Content Verification: While citations are provided, the system is not guaranteed to provide factual information. This could lead to potential hallucinations or misinterpretations. At this point, the potential for hallucination can’t be 100% eliminated.

Despite those limitaitons, we have highlighted how a relatively simple system can still yield solid and manageable results, for a complex task.

Conclusion

This chapter has explored critical strategies and techniques for managing input data in LLMBAs, focusing on three key areas:

data pre-processing

RAG

Long-context models

We examined how parsing tools like PyPDF2 MarkItDown and Docling can transform diverse data formats into LLM-compatible representations, demonstrating through case studies how parser quality can impact LLMBA performance. The chapter also investigated retrieval augmentation techniques, particularly RAG systems, showing how they can enhance LLM capabilities by providing access to external knowledge while discussing their future relevance in the context of emerging long-context language models.

Through our case studies, we demonstrated practical approaches to handling common challenges in LLM applications. The Content Chunking with Contextual Linking case study illustrated techniques for managing long-form content generation while maintaining coherence across chunks. The Quiz Generation with Citations case study showcased how long-context windows can be effectively utilized without the need for complex retrieval systems, highlighting the importance of choosing the right approach based on specific application requirements rather than defaulting to more complex solutions.

As the field continues to evolve, the choice between traditional RAG systems and emerging long-context models will likely become increasingly nuanced. While RAG offers cost-effective solutions for incorporating external knowledge, the rise of long-context models suggests a future where simpler architectures might suffice for many applications. The key insight is that effective input data management requires careful consideration of trade-offs among complexity, cost, and performance, always guided by specific application requirements rather than following a one-size-fits-all approach. Success in building robust LLM applications will depend on understanding these trade-offs and selecting appropriate strategies for each use case.

¹ ChromaDB enables custom embedding functions and provides a list of wrappers around commonly used embedding models and APIs

About the Authors

Dr. Tharsis Souza is a computer scientist and product leader specializing in AI-based products. He is a Lecturer at Columbia University’s Master of Science program in Applied Analytics, Head of Product, Equities at Citadel, and former Senior VP at Two Sigma Investments. With over 15 years of experience delivering technology products across startups and Fortune 500 companies globally, Dr. Souza is also an author of numerous scholarly publications and is a frequent speaker at academic and business conferences. Grounded on academic background and drawing from practical experience building and scaling up products powered by language models at early-stage startups, major institutions as well as advising non-profit organizations, and contributing to open source projects, he brings a unique perspective on bridging the gap between LLMs promised potential and their practical limitations using open source tools to enable the next generation of AI-powered products. Dr. Tharsis holds a Ph.D. in Computer Science from UCL, University of London following an M.Phil. and M.Sc. in Computer Science and a B.Sc. in Computer Engineering.

Jonathan Regenstein currently serves as the Head of AI/ML in Financial Services at Snowflake, a role he has held since May 2023. Based in Atlanta, Georgia, he operates in a hybrid working environment, showcasing his expertise in artificial intelligence and machine learning within the financial sector. In addition to his professional endeavors, Jonathan is a dedicated academic. He is a Research Affiliate at the Georgia Tech Financial Services Innovation Lab, a position he has maintained since October 2020. Furthermore, since August 2022, he has been imparting his knowledge as an Adjunct Professor at Kennesaw State University’s School of Data Science and Analytics.