AI Agents: The Definitive Guide

Design, Deployment, and Evaluation

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Nicole Koenigstein

AI Agents: The Definitive Guide

by Nicole Koenigstein

Copyright © 2027 Nicole Koenigstein. All rights reserved.

Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

Acquisitions Editor: Nicole Butterfield

Development Editor: Michele Cronin

Production Editor: Elizabeth Faerm

Copyeditor: TO COME

Proofreader: TO COME

Indexer: TO COME

Cover Designer: Susan Brown

Cover Illustrator: TO COME

Interior Designer: David Futato

Interior Illustrator: Kate Dullea

November 2026: First Edition

Revision History for the Early Release

2025-09-12: First Release

2025-11-24: Second Release

2025-12-15: Third Release

See https://oreilly.com/catalog/errata.csp?isbn=9798341666931 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. AI Agents: The Definitive Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

979-8-341-66689-4

[LSI]

Brief Table of Contents (Not Yet Final)

Chapter 1: From LLMs to Agents: The Foundational Blueprint (available)

Chapter 2: Architectures and Patterns: Planning, Reactivity, and Multi-Agent-Systems (available)

Chapter 3: Advanced Planning, Reasoning, and Scalable Execution in Agents (available)

Chapter 4: Models Behind the Agents: Capabilities and Optimization (available)

Chapter 5: Interfaces and Tooling: System Integration and Execution (unavailable)

Chapter 6: Deploying Agents in Real Products (unavailable)

Chapter 7: Benchmarking LLMs and Agentic Systems for Production (unavailable)

Chapter 8: Multimodal and Domain-Specific Benchmarks (unavailable)

Chapter 9: Safety, Guardrails, and Risk Mitigation (unavailable)

Chapter 10: Memory, Autonomy, and Long-Horizon Use (unavailable)

Chapter 11: Cost Estimation and Efficiency Optimization (unavailable)

Chapter 12: Security and Guardrails (unavailable)

Chapter 1. From LLMs to Agents: The Foundational Blueprint

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at mcronin@oreilly.com.

One of the defining traits of human intelligence is the way we combine inner reasoning with concrete actions, and this maps surprisingly well to how large language model (LLM) agents operate when paired with tools. Take the process of building a coding project. As a human developer you begin with a prompt, the client’s request. First comes reasoning, where you sketch out a plan of how to approach it. Then comes action, such as searching for documentation, writing functions, or debugging errors. Feedback enters the loop when tests fail, a peer review points out gaps, or the client tests the application. Each step is not static but iterative, with reasoning adapting to new insights and actions changing in response.

The same applies to a single LLM agent. Given a task, the agent first reasons about what is missing or what step comes next, then takes actions by calling tools to retrieve data, run code, or check results. Like a developer’s feedback loop, the environment provides signals such as errors, gaps, or confirmations that guide the next iteration. External support, such as code libraries or documentation, maps to tools like retrieval systems or web searches, which refine the output beyond the model’s own raw ability.

An LLM agent is a large language model embedded in a loop of reasoning, acting, and feedback, where it can call external tools and adapt its behavior based on results. Unlike a standalone LLM, which is limited to static text generation, an agent operates as a decision-making entity within a workflow. In this book, I will use AI agents, agents, and LLM agents interchangeably. Strictly speaking, an AI agent or agent does not need to involve a language model at all. For example, in pure reinforcement learning agents are trained directly through trial and error interaction with an environment. The benefit of using an LLM is that it brings strong generalization, reasoning, and natural language capabilities, which makes agent behavior more flexible and broadly applicable beyond narrowly defined environments.

Just as individual skills eventually meet their limits, a single agent can only take a workflow so far before complexity demands coordination. If you think about it, in a small company, a single contributor might handle everything end-to-end: designing, coding, testing, and deploying. But as the project grows, the team needs to expand. One person might focus on backend, another on frontend, another on testing. To keep everything aligned, a tech lead or VP of engineering steps in, coordinating the workload, assigning tasks, and integrating results. The difference between a lone contributor and a coordinated team maps directly onto the difference between a single agent and a multi-agent system (MAS). Just as a tech lead coordinates multiple contributors, a supervisor agent can orchestrate multiple specialized agents to collaborate on solving a complex goal.

It is also important to recognize that neither humans nor agents operate with complete autonomy. Developers are guided by coding standards, project requirements, and organizational processes, and their freedom is bounded by these structures. Agents are similarly constrained by their workflows, operating only within the scope of the tools, permissions, and safeguards they are provided. Far from being a weakness, this lack of full autonomy is what makes both systems viable. For humans, structure ensures code quality, maintainability, and client satisfaction. For agents, constraints ensure safety, reliability, and alignment with the goals set by their developers and users.

This parallel highlights why extending pure LLMs with tools makes sense. A language model on its own is like a developer cut off from resources like StackOverflow, IDEs, or GitHub, capable of reasoning but unable to gather new data, verify results, or improve their task output without outside help. Just as software teams scale their capabilities by adding better tools, resources, and leadership, LLMs extend their capability when embedded in agentic workflows and even multiple specialized agents within a MAS. Without these upgrades, both remain confined to the limits of their initial training and quickly become overwhelmed when faced with complex tasks. With tools, feedback, and a divide and conquer approach, LLMs can iteratively refine, adapt, and solve complex tasks that would otherwise exceed their standalone capacity.

I assume in this book that you have at least a fundamental knowledge of LLMs. Maybe you’ve read the book Hands-On Large Language Models, or a similar work. You don’t need to know how to deploy LLMs to make your agents work, since I’ll guide you through those steps in later chapters. However, I expect you have solid coding skills in Python, meaning you know what classes and functions are, and that you are familiar with core ML and deep learning concepts such as neural networks, training loops, and backpropagation. A working knowledge of linear algebra and calculus is also helpful, since they underpin much of modern deep learning. It is also useful if you understand how transformers process sequences and have at least a basic sense of how embeddings work or vector stores support retrieval.

If you haven’t touched these topics in a while, DON’T PANIC! Just like The Hitchhiker’s Guide to the Galaxy advises not to panic, I’ll say the same here in my guide. That said, every concept will be grounded in code, so your knowledge becomes usable again without requiring you to dust off old textbooks. And no, before you ask, the answer to everything about AI agents is not 42, but I promise you’ll get your answers throughout this book.

Code for the book

All code examples from this book are available in the GitHub accompanying repository. The repository is organized by chapter, so you can easily find the code that belongs to each section.

If you would like a closer look at transformers across different domains and how embeddings are used for text and other modalities, my book Transformers: The Definitive Guide can serve as a complementary source. It also explains the fundamentals of reinforcement learning, which will help you build a stronger foundation for understanding the advanced agent concepts introduced later in this book.

Therefore, this book is written for readers who want to go beyond curiosity, which is why understanding these fundamentals is important. This includes data scientists, software and ML engineers who are building real systems in production, and technical leaders who need to understand the trade-offs, costs, and guardrails involved. This book is not about explaining LLMs from scratch, nor is it about abstract promises, or thought experiments. It is about the practical realities of building and running AI agents that can perform and be useful outside a PowerPoint presentation or a lab demo.

This chapter explains how LLMs evolve from static prompts to dynamic agentic systems, why tool use is essential, and what being stateful and autonomous truly means in the design of modern AI agents.

Finite and Hierarchical State Machines: The Base Paradigm of Agents

LangGraph, CrewAI, and similar frameworks build on the concept of state machines. At a high level, a state machine is a computational model that exists in one of a finite number of states at any given time. It transitions between states in response to specific events or inputs, with the next state determined by the current state and the input.

This section is meant to help you build a mental model of agents by showing how state machines form the foundation of modern AI agent frameworks. In simple terms, think of it as structured transitions, a pattern long used in compilers, GUIs, embedded systems, and networking protocols, now applied to reasoning and tool use in LLM-based agents.

My intention in grounding your understanding first in finite state machines (FSMs) and hierarchical state machines (HSMs) is more than a loose analogy. I want to give you a blueprint for building LLM agents that are robust, reliable, and adaptable. It will empower you to not only understand the concepts but to apply them effectively as you learn about more complex topics later in the book. However, if you haven’t worked with state machines recently, don’t worry. You’ll see how the patterns translate directly in the code examples.

Finite State Machines

A finite state machine has a small vocabulary in which it can act upon and modes your system can be in, and the allowed moves between them. Each move is triggered by an event, optionally guarded by a predicate, and may perform actions that update a state. Start and end are distinguished states with special meaning. The core vocabulary is:

State

A compact snapshot that captures what the system knows at a given moment. For an agent, this might be the message list, routing hints, or progress markers.

Event

Something that happens since the last decision, such as a tool being invoked or returning a result.

Guard

A check that decides which transition to take, based on the current state and latest event.

Action

Work performed during a transition, like invoking a tool, appending a message, saving a checkpoint, or requesting human input.

Termination

A condition that signals the process has reached the end.

Figure 1-1 illustrates a finite state machine for tool use. Each transition is defined by an event, a guard, and an action.

Figure 1-1. High level overview of a finite state machine for tool use.

An FSM is ideal when behavior alternates among a few stable modes and when recoverability matters, since you can checkpoint on every node and resume after a crash from the last completed state.

Thinking in terms of states lets you decompose a complex task into distinct steps. For example, instead of a single prompt for “write a blog post”, you might define states such as topic ideation, outline generation, drafting, editing, and SEO optimization. Each state can link to specific tools your agent will use.

Transitions make you specify the conditions under which one task hands off to the next. For instance, what output from the outline generation state triggers the drafting state. This structure counters unstructured, one-shot prompts that often lead to inconsistent results with LLMs alone.

As soon as you add more modes, such as planning, reflection, approval gates, and retries, an FSM can force duplication or create spaghetti routing. You need to keep control of the edges, since they are decision points: each one encodes the event, guard, and action that drives functionality. This is where hierarchy helps.

Hierarchical State Machines

Hierarchical state machines let states contain other states. Superstates capture shared entry/exit behavior and shared guards. Substates inherit those rules and add their own. History nodes remember which substate you were in when you left, so you can resume exactly there later. These new additions to the state machine are explained below.

Superstate

A state that groups a set of child states and their common policies. Entering the superstate runs shared logic once; exiting it runs shared cleanup once.

Substate

A concrete mode that lives inside a superstate. It has its own edges but inherits the superstate’s guards and actions.

History

A “remember where I was” marker. Shallow history resumes at the last active child; deep history resumes inside nested grandchildren. In agent terms, this is a checkpoint for a portion of the graph.

Parallel region

Two or more child regions of a superstate that advance independently. Useful for fanning out tool calls or agent roles and then joining.

Figure 1-2 shows a hierarchical state machine with a WORKING superstate. PLAN, ACT, and REFLECT are substates. H is a shallow history marker that records the last active substate so execution can resume there if the superstate is re-entered.

Figure 1-2. Overview of a hierarchical state machine with history marker hat records the last active substate.

HSMs are particularly relevant to more advanced MAS. An HSM allows for “states within states”, which reduces duplication and makes policies obvious. For example, you can attach rate limits, safety filters, or circuit breakers to a WORKING superstate, so planning, acting, and reflecting all inherit the same safeguards. You define a guard or router once and reuse it across nodes. Consider a research agent. Instead of treating research as a single monolithic step, you can break it into a sub-machine with states such as searching for keywords, gathering URLs, scraping data, and summarizing findings. Each of these states can even map to a dedicated agent with its own capabilities, contributing to the larger team of agents. This nested structure helps manage complexity, keeps policies consistent, and makes the overall system easier to reason about and maintain.

Mapping FSM/HSM to Agent Frameworks

LangGraph already speaks the language of state machines and hierarchy. You implement it with subgraphs and shared keys. The router or decision-making agent that selects the next action (such as calling the next agent or tool) is analogous to the event handler in a classic state machine. It’s the part of the system that takes the current state and the LLM’s output (the event or payload) and determines the next logical step. The direct mapping is shown below.

State schema

The parent graph’s TypedDict defines the shared “bus”. Subgraphs declare which keys they read and write. Overlapping keys are the superstate’s interface.

class State(TypedDict):
messages: List[BaseMessage]
working_last: str | None

Shared bus across parent and subgraphs.

Optional shallow history marker.

Nodes and subgraphs

A subgraph is an HSM superstate. Its internal nodes are substates. Entering the subgraph runs its entry node; leaving it runs its exit edge back to the parent.

def plan_node(state: State) -> State:
    ai = AIMessage(content="Plan next step")
    return {"messages": [ai], "working_last": "plan"}

def act_node(state: State) -> State:
    ai = AIMessage(content="Act on plan")
    return {"messages": [ai], "working_last": "act"}

subgraph_builder = StateGraph(State)
subgraph_builder.add_node("PLAN", plan_node)
subgraph_builder.add_node("ACT", act_node)
subgraph_builder.add_edge(START, "PLAN")
subgraph = subgraph_builder.compile()

Subgraph is the superstate.

Guards and conditional edges

Parent-level guards enforce global policy (budget, safety). Child-level guards route among PLAN/ACT/REFLECT. Because guards are plain Python, they remain deterministic and auditable.

def route_within_working(state: State):
    last = state.get("working_last")
    return "ACT" if last == "plan" else END

subgraph_builder.add_conditional_edges("PLAN",
                route_within_working, {"ACT": "ACT", END: END})

parent = StateGraph(State)
parent.add_node("WORKING", subgraph)

def global_guard(state: State):
    tail = "".join([
            getattr(m, "content", "") for m in state["messages"][-3:]
            ]).lower()
    return END if "stop" in tail else "WORKING"

parent.add_edge(START, "WORKING")
parent.add_conditional_edges("WORKING", global_guard,
                            {"WORKING": "WORKING", END: END})

Add conditional edge inside subgraph.

Parent graph with a global guard.

Example policy: end if user said stop.

History and checkpointing

MemorySaver gives you shallow and deep history for free: resuming a thread recreates the subgraph at the last completed node. If you want explicit “return to where I was inside WORKING,” persist a small marker (for example, state["working_last"] = “reflect”) and branch on it when re-entering.

def reflect_node(state: State) -> State:
    ai = AIMessage(content="Reflect on result")
    return {"messages": [ai], "working_last": "reflect"}

checkpointer = MemorySaver()
app = parent.compile(checkpointer=checkpointer)

cfg = {"configurable": {"thread_id": "session-1"}}
app.invoke({"messages": [HumanMessage(content="start")],
                        "working_last": None}, config=cfg)

Substates set working_last on exit.

Compile with checkpointing.

Run in a thread. MemorySaver restores last completed node on resume.

FSMs give you a way to structure tasks into stable, recoverable steps. HSMs extend this by reducing duplication and centralizing policy when workflows grow more complex. Both patterns prepare you for the next step: turning static LLM predictions into dynamic, stateful agents.

Foundations: From Static Models to Dynamic Agents

As you saw in the previous section, adding states and control flows is essential for automation. The same principle applies to agents. To move from static LLM usage to agentic systems, you need explicit state and control flow. State turns one off tool calls into a memory bearing process. Control flow adds guards and transitions that govern when to plan, when to act, and when to stop.

The reason to add this to LLMs is simple: LLMs are static predictors. They generate the next token given a prompt, drawing only on patterns encoded in their training data. This makes them powerful reasoners but leaves them confined to what they already know. They cannot update knowledge, verify claims, or interact with an environment.

Agency emerges when reasoning is coupled with action. Reasoning is the internal process of planning or deciding the next step. Action extends this by invoking tools, retrieving data, executing code, or calling external services. Together they form an iterative loop: reason, act, receive feedback, and adjust. This loop is the foundation of agentic behavior.

Table 1-1 compares static LLMs to agentic systems. These contrasts show not only why agents are necessary, but also how their design shifts traditional machine learning workflows into dynamic, tool augmented systems.

Table 1-1. Comparison of Static LLMs and Agentic Systems

Static LLM (Traditional Use)
Agentic System

Single-pass token generator

Iterative reasoning–action–feedback system

Confined to training data

Can retrieve, verify, and update knowledge

No memory, stateless

Stateful with short- and long-term memory

Linear prompt–response

Iterative and adaptive workflow

No tool use

Tool-augmented (retrieval, code, APIs)

Fixed interpretation of prompt

Can refine or reinterpret goals dynamically

No external validation

Actively checks, corrects, and improves output

Figure 1-3 illustrates how modules turn a static LLM into a dynamic agent. The user request enters the core, planning and memory shape its reasoning, and tools enable concrete action. The cycle of reasoning, acting, and adapting emerges from the coordination of these elements.

Figure 1-3. Baseline agent architecture: an LLM alone can only predict text, but with planning, memory, and tools it becomes a usable agentic system.

Planning modules guide the agent’s reasoning, ranging from simple chain-of-thought ¹ traces to more advanced approaches such as trees of thought ² or self-critique. Memory modules provide continuity, allowing the agent to recall past steps, reuse prior knowledge, or persist information across sessions. Tools connect the agent to its environment, whether through search, retrieval, code execution, or custom APIs.

Moreover, a key difference between traditional LLM usage and agentic systems is whether the workflow is stateless or stateful. A stateless call treats the model like a black box: it takes a prompt, predicts the next tokens, and returns a result. Nothing is remembered, no actions are taken, and no feedback loop exists. Example 1-1 illustrates a stateless LLM call.

Example 1-1. Stateless LLM call with LangChain

llm = ChatOpenAI(model="gpt-5-mini")
response = llm.invoke("What are AI agents?")
print(response.content)

In addition, the moment you need facts from outside training data, or you want the model to act (search, compute), you introduce tools.

In the following code (Example 1-2 - Example 1-4) is a minimal stateless tool-using run: first you bind two tools to the LLM, and then run a tiny loop that executes any tool calls the model needs to fulfill its task. First, you need to define your tools. Example 1-2 shows an implementation of tools.

Example 1-2. Defining tools for a stateless run

@tool("internet_search")
def internet_search(query: str) -> str:
    """Search Google via SerpAPI for up to date information."""
    serp_api_key = os.environ["SERPAPI_API_KEY"]
    params = {"engine": "google", "gl": "us", "hl": "en"}
    search = SerpAPIWrapper(params=params, serpapi_api_key=serp_api_key)
    return search.run(query)

@tool("calculator")
def calculator(expression: str) -> str:
    """Evaluate a single line mathematical expression with numexpr."""
    local_dict = {"pi": math.pi, "e": math.e}
    out = numexpr.evaluate(
        expression.strip(),
        global_dict={},
        local_dict=local_dict,
    )
    return str(out)

tools = [internet_search, calculator]
tool_map: Dict[str, Any] = {t.name: t for t in tools}

LangChain’s @tool decorator uses the function name, typed signature, and docstring to build the schema the model sees.

The docstring is mandatory: if it is missing, LangChain will raise an error when constructing the tool schema or binding tools to the model.

Constrain evaluation: deny globals and expose only the constants you want. This keeps the calculator deterministic and safe.

With LangChain’s @tool decorator, the function signature and docstring are used to build the schema the model sees. You need to keep these precise, consistent and as explanatory as possible for the task-to-be-done efficiently. The docstring acts as the tool description and is a hard requirement from LangChain, if it’s missing, you get an error. In addition, if the docstring is vague the model is more likely to produce invalid calls.

Now, to give the LLM access to the tool, you just bind the tools to the LLM as shown in Example 1-3.

Example 1-3. Bind tools to LLM

llm = ChatOpenAI(model="gpt-4o", temperature=0,
                max_tokens=800).bind_tools(tools, tool_choice="auto")

Bind tools to the model. tool_choice="auto” lets the model decide whether and which tool to call.

Modern LLMs are built for agentic tasks

Modern LLMs are not only capable of calling tools, they are increasingly trained and optimized specifically for agentic tasks. For example, models such as Kimi K2 and Llama 4 are explicitly tuned for coding, tool use, and powering agentic systems. These models go beyond answering questions in a chat window. They are instruction tuned to act, invoking APIs, running code, or retrieving data as part of their core design.

With tools bound, you can run a minimal stateless tool loop. The code in Example 1-4 executes any tool calls the model requests, feeds observations back, and forces a short wrap up to ensure the final answer is returned even if the last step ended on a tool observation.

Example 1-4. Stateless single run with tool loop

def run_once(prompt: str, max_steps: int = 8) -> str:
    messages: List[HumanMessage | AIMessage | ToolMessage] = [
    HumanMessage(content=prompt)]
    last_ai: AIMessage | None = None

    for _ in range(max_steps):
        ai: AIMessage = llm.invoke(messages)
        messages.append(ai)
        last_ai = ai

        calls = getattr(ai, "tool_calls", None) or []
        if not calls:
            return messages[-1].content

        for call in calls:
            name = call["name"]
            args = call.get("args", {}) or {}
            result = tool_map[name].invoke(args)
            messages.append(ToolMessage(
                content=str(result),
                name=name,
                tool_call_id=call.get("id")
            ))

    messages.append(HumanMessage(content="""
Finish now. Give a short final answer in this exact format:

Current temperature:
Square of current temperature:
""".strip()))
    final_ai: AIMessage = llm.invoke(messages)
    return final_ai.content

print(run_once("""Two step task.

Step 1: Use internet_search to get the current air temperature in New York City
today. Show the exact query you used, the top source title and snippet, and
extract a numeric temperature in Celsius. Return this temperature as feedback
for Step 2.

Step 2: Using the Celsius value from Step 1, compute its square with calculator.
Show the exact expression you used and the numeric result.

Important: Give a short final answer in this format:
Current temperature:
Square of current temperature:"""))

Local short term memory for this run only. This makes the workflow stateless across runs.

A simple step bound protects against runaway loops: LLMs cycling with no termination.

One model step. The assistant may choose to call tools.

Read structured tool calls.

Attach the observation with the matching tool_call_id so the model can correlate results to calls.

Tool cycles can end on an observation. This final instruction guarantees a closing assistant message in your requested format.

Giving the LLM a State

The first step in turning an LLM into an agent is to make it stateful. Give your LLM a state that records where it’s in the workflow. Based on input and logic, it transitions to the next state, and each state has an associated function or behavior. In LangGraph, this can be any Python type, but is typically a TypedDict or Pydantic BaseModel. The following outlines LangGraph’s key concepts to build stateful, adaptable AI agents.

State

A shared data structure that represents the current snapshot of an application. It can be any Python type, but is typically a TypedDict or Pydantic BaseModel.

Nodes

Functions that encode the agent’s logic. They take the current state, perform computation or side effects, and return an updated state.

Edges

Rules that select the next node based on the current state. They can be conditional branches or fixed transitions and should also detect when a finish condition is met.

Command

An object that combines control flow and state updates to support multi-actor communication. A node can both update the state and choose the next node by returning a Command.

So far, the tools you created earlier have only been used in stateless loops. Each run was independent: the model could call tools, get answers, and wrap up, but as soon as the run ended, all context was lost. Now, if you want to combine LangGraph’s concepts to build stateful agents, you need to embed the tools into a stateful workflow. Instead of discarding messages after each turn, you keep them in a structured state. This makes the system agentic:

The LLM node reasons about the next step.

The tool node executes external actions.

Edges route control flow based on whether tools were requested.

Memory checkpoints persist the conversation, so the agent can reuse earlier results in later turns.

This transforms tool use from being a one-off helper into a continuous reasoning–acting–adapting cycle. In the next code listings (Example 1-5 – Example 1-13), you’ll build such a stateful agent with LangGraph, starting from the same tools and LLM binding introduced in Example 1-3 - Example 1-4.

Example 1-5 shows how a state object can be implemented in LangGraph to carry the conversation and coordinate control flow. You can reuse the tools and the bound llm from Example 1-3 and add only what is new: a typed state, an LLM node that appends assistant messages, a tool node that executes calls, and a router that decides whether to continue or stop based on tool calls.

Example 1-5. Build a minimal LangGraph with state, nodes, and routing

class AgentState(TypedDict):
    messages: Annotated[List[BaseMessage], add_messages]

def llm_node(state: AgentState) -> AgentState:
    ai = llm.invoke(state["messages"])
    return {"messages": [ai]}

tool_node = ToolNode(tools=tools)

graph = StateGraph(AgentState)
graph.add_node("llm", llm_node)
graph.add_node("tools", tool_node)
graph.add_edge(START, "llm")

def route(state: AgentState):
    last = state["messages"][-1]
    calls = getattr(last, "tool_calls", None) or []
    return "tools" if calls else END

graph.add_conditional_edges("llm", route, {"tools": "tools", END: END})
graph.add_edge("tools", "llm")

The state carries the conversation as a message list. add_messages handles safe merging across node updates.

Core reasoning node. It takes the current state and returns a new assistant message.

Prebuilt node to execute tool calls emitted by the assistant.

Graph container for nodes and edges.

Router. If the last assistant turn requested tools, go to tools, otherwise finish.

With the graph defined, you need to make it runnable and persistent. Example 1-6 compiles the graph with an in-memory checkpointer and assigns a thread id. The checkpointer restores prior messages on each turn so the agent can reuse earlier results. The thread id separates branches, so you can run parallel conversations without interference.

Example 1-6. Compile with in-memory checkpoints and configure a thread

checkpointer = MemorySaver()
app = graph.compile(checkpointer=checkpointer)

cfg = {"configurable": {"thread_id": "nyc-weather-session"}}

Lightweight, per-thread checkpointing that persists state across turns.

Produces a runnable app that knows how to save and load state.

A thread id identifies one conversation branch. New ids create new branches.

Once you have a runnable app and checkpoints in place, the next step is to make its execution observable. You want to see what the graph is doing, which messages are flowing, and how tool outputs attach. Example 1-7 introduces a compact helper that formats messages into a single readable line, making it easier to follow traces without overwhelming logs.

Example 1-7. Trace execution and inspect state and memory

def _short(msg: BaseMessage, max_len: int = 140) -> str:
    """Compact one-line view of a message."""
    role = type(msg).__name__.replace("Message", "").lower()
    content = getattr(msg, "content", "")
    if isinstance(content, list):
        # some tool outputs can be list payloads
        try:
            content = json.dumps(content)
        except Exception:
            content = str(content)
    text = str(content).replace("\n", " ").strip()
    if len(text) > max_len:
        text = text[: max_len - 3] + "..."
    if hasattr(msg, "tool_calls") and getattr(msg, "tool_calls"):
        tnames = [tc.get("name", "tool") for tc in msg.tool_calls]
        return f"{role}: tool_calls -> {tnames}"
    if isinstance(msg, ToolMessage):
        return f"{role}({msg.name}): {text}"
    return f"{role}: {text}"

Include tool name or function call info when available.

Example 1-8 prints a concise view of messages and routing hints. Together they let you verify that tool outputs are attached correctly and that state transitions behave as intended.

Example 1-8. Trace execution and inspect state and memory

def print_state_snapshot(app, config, title: str):
    """Print current graph state and memory for a given thread."""
    snap = app.get_state(config)
    values = snap.values or {}
    msgs: List[BaseMessage] = values.get("messages", [])
    print(f"\n=== {title} | state snapshot ===")
    print(f"messages: {len(msgs)} total")
    for i, m in enumerate(msgs[-5:], start=max(0, len(msgs)-5) + 1):
        print(f"  {i:>3}: {_short(m)}")

    nxt = getattr(snap, "next", None)
    tasks = getattr(snap, "tasks", None)
    if nxt:
        print(f"next nodes: {list(nxt)}")
    if tasks:
        print(f"queued tasks: {tasks}")
    print("memory: in-memory checkpoint present for this thread")

Read the latest state for a thread.

Print a concise tail of messages to keep logs readable.

Show routing info and queued tasks if present.

Example 1-9 streams node updates, so you can see each step the graph takes.

Example 1-9. Trace execution and inspect state and memory

def run_with_tracing(app, input_state: AgentState, config, title: str):
    """Run the graph while printing per-node updates and final memory."""
    print(f"\n=== {title} | execution trace ===")
    final = None
    for event in app.stream(input_state, config=config, stream_mode="updates"):
        for node, upd in event.items():
            keys = list(upd.keys())
            print(f"[enter {node}] updated: {keys}")
            # if messages updated, print the last one briefly
            msgs = upd.get("messages") or []
            if msgs:
                print(f"  {_short(msgs[-1])}")
            print(f"[leave {node}]")
            final = upd
    print_state_snapshot(app, config, title=f"{title} | after run")
    snap = app.get_state(config)
    msgs = snap.values.get("messages", [])
    return msgs[-1].content if msgs else ""

Stream node updates to see the exact flow through the graph.

upd is a dict like {"messages": [<new msg>]} or tool results.

Show final assistant message from app.get_state.

Example 1-10 asks for the current air temperature in New York City in Celsius. The trace shows the graph entering the LLM node, initiating a tool call if needed, and returning a final assistant message. The state snapshot confirms that the result is saved in memory for this thread.

Example 1-10. Turn 1: get the current NYC air temperature in Celsius

turn1_answer = run_with_tracing(
    app,
    {"messages": [HumanMessage(content="Get the current air temperature in New York
    City in Celsius.")]},
    config={**cfg, "recursion_limit": 20},
    title="TURN 1",
)
print("\nTURN 1 (final assistant):\n", turn1_answer)

This results in this model output:

TURN 1 (final assistant):
The current air temperature in New York City is 18°C.

You continue in the same thread. Example 1-11 asks to square the previously retrieved temperature. Because the thread id is unchanged, the app restores the earlier messages, allowing the assistant to use the prior result without you repeating it in the prompt.

Example 1-11. Turn 2: square that temperature in the same thread

turn2_answer = run_with_tracing(
    app,
    {"messages": [HumanMessage(content="Now compute the square of that
    temperature.")]},
    config={**cfg, "recursion_limit": 20},
    title="TURN 2",
)
print("\nTURN 2 (final assistant):\n", turn2_answer)

Now the square will be computed and the model returns:

TURN 2 (final assistant):
The square of the temperature, 18°C, is 324.

Agents often need to explore alternatives without losing progress. Example 1-12 starts a separate branch by using a different thread id and requests a different follow-up. This creates an isolated conversation that can diverge from the main path while preserving the main thread’s memory intact.

Example 1-12. Branch a parallel thread for a different follow-up

branch_answer = run_with_tracing(
    app,
    {"messages": [HumanMessage(content="Instead of squaring, convert it to
    Fahrenheit and report both.")]},
    config=cfg_branch,
    title="BRANCH",
)
print("\nBRANCH (final assistant):\n", branch_answer)

Finally, you can inspect what each branch remembers. Example 1-13 prints compact snapshots for the main thread and the branch. Comparing them side by side confirms that state is persisted within a thread and isolated across threads, which is the foundation for reliable multi-step and multi-branch agent workflows.

Example 1-13. View snapshots for both threads

print_state_snapshot(app, cfg, title="MAIN THREAD memory view")
print_state_snapshot(app, cfg_branch, title="BRANCH THREAD memory view")

With this small implementation, you now see exactly what the graph remembers, how it routed, and what the latest assistant state is for each branch.

Deus Ex Machina: What is Possibly Right Now?

Once you view agents as HSMs with states, the autonomy spectrum becomes a matter of how much choice you place inside a superstate. Routers are one-edge FSMs with a single guard. Tool-calling agents are HSMs with a WORKING superstate that loops among PLAN, ACT, and REFLECT. MAS compose multiple subgraphs under a supervisor superstate, sometimes with parallel regions. The system still runs inside explicit states and guards, perceived “autonomy” is just a richer choice inside well-bounded regions.

In ancient Greek theater, a deus ex machina was the sudden appearance of a god to resolve an unsolvable conflict or problem. It is tempting to imagine AI agents in the same way: a machine that suddenly acts with full autonomy and intelligence solving all your problems. But that’s not what’s possible today, agents are engineered systems. Their strength comes not from hidden magic, but from carefully designed workflows, well-defined tools, and the boundaries that keep them safe and usable. That said, agents are still constrained by the tools you give them, by safeguards you enforce, and by the workflows you allow them to operate within. These constraints are what make agents viable today: safe to deploy, predictable in behavior, and aligned with their intended goals.

What current systems demonstrate is a form of orchestrated autonomy. As you saw in the LangGraph examples earlier (Example 1-5), agents can branch conditionally, call tools, or decide whether to continue a workflow. This already looks very different from a static prompt–response call, but it is still bounded. The orchestration logic is fixed; within it, the LLM can make choices, but it cannot redefine its architecture. Table 1-2 gives a short overview what agents can do so far.

Table 1-2. Current autonomy in agents: what’s possible vs. what remains out of reach

What agents can do
What agents can’t do yet

Route between predefined paths (e.g., router node)

Redesign or rewire their own control graph

Select which tools or sub-agents to call

Generate and validate new graph topologies dynamically

Decide if more steps are needed before completion

Continuously assess tool effectiveness or create new tools autonomously

Revise their own prompt or toolset within bounds

Guarantee alignment and coherence when shifting strategies mid-execution

Write and run small pieces of code to determine next steps

Operate as fully self-directed systems independent of orchestration frameworks

The agent adapts at runtime, but only within the framework you design. It helps to see this as a spectrum. At one end are routers that make a single decision from a fixed set of options. In the middle are tool-calling agents that support multi-step reasoning, reflection, and memory. At the far end are MAS, where specialized agents collaborate under a supervisor, distributing decision-making across roles. Figure 1-4 shows the spectrum of autonomy, from simple routing decisions to the idea of fully self-directed execution. In practice, nearly all real-world systems fall into the middle: they are agentic systems, not truly autonomous agents. Their autonomy is orchestrated rather than self-evolving, decisions happen within a framework of predefined states, transitions, and toolsets, even if the agent can adaptively choose among them at runtime.

Figure 1-4. In orchestrated autonomy, decision points are predefined in the workflow. In full autonomy, the LLM could generate new tools, actions, or decision paths to reach its goal.

Agents can simulate flexibility within frameworks such as LangGraph using reflection, planning, and structured outputs. However, we do not yet have systems that can freely redefine their own architecture or reason about their goals in a robust, and general way.

The AI Scientist: What Limited Autonomy Can Already Do

The AI Scientist v2 ³ is an example of what orchestrated autonomy already makes possible. Built on a multi-agent workflow that integrates tree search, experiment management, and vision–language feedback, it can generate research hypotheses, write code to test them, run experiments in parallel while debugging and refining, visualize results, and ultimately draft a scientific manuscript. In 2025 the system submitted three papers to an ICLR workshop, one of which passed peer review and was formally accepted, the first known case of a fully AI generated scientific paper hitting that bar.

Yet the achievement came with clear limitations. The accepted paper was at workshop level, not a main conference track. From the three submissions, two were rejected, and even the accepted work showed methodological gaps, unclear justifications, shallow analysis, and dataset limitations. Although reviewers praised its clarity and rigor, they stressed that it lacked the depth and innovation required for top tier publication. The system also stumbled on well known LLM pitfalls, including citation errors and misleading figure descriptions. So, while impressive, that the system can refine and debug autonomously, it doesn’t generate fundamentally new scientific paradigms autonomously yet.

This case shows both sides of current autonomy. Within the boundaries of engineered orchestration, agentic systems can already close the loop from idea to experiment to code and even peer reviewed publication. Beyond those boundaries, toward consistently producing groundbreaking science or operating with unconstrained creativity, today’s agents remain limited and human oversight remains important.

As you now understand, today’s agents are not deus ex machina solutions but carefully orchestrated systems that extend LLMs with structure, tools, and feedback loops. Their autonomy is bounded by design, which is precisely what makes them practical, safe, and deployable. While still limited, they are already impressive and capable of reasoning, acting, and adapting in ways that go far beyond static LLMs.

Conclusion

In this chapter, you learned how LLMs evolve from static predictors into dynamic agents through the addition of state, control flow, and tool use. By embedding reasoning in a loop of action and feedback, LLMs gain the ability to adapt to results and extend their capabilities far beyond what is possible with prompt–response interactions alone.

Finite and hierarchical state machines provided the conceptual foundation for agent design. FSMs let you structure workflows into recoverable steps, while HSMs reduced duplication and centralized policies for complex, multi-step tasks. These abstractions map directly to modern frameworks such as LangGraph, where state, nodes, and routing rules define how agents reason and act over time.

You now can also place autonomy in a better context. You understand that today’s agents operate within well-bounded workflows, choosing among predefined paths, invoking tools, and reflecting on results, don’t redesign their own architecture. This orchestrated autonomy is not a weakness but a strength, since constraints make agents safe, predictable, and practical to deploy. The example of the AI Scientist v2 showed both the promise and the limitations of this approach, demonstrating how agents can already generate and test hypotheses but still require oversight to ensure rigor and depth.

With this foundation in place, you are ready to move into the next chapter, where you’ll explore the architectures and patterns that bring planning, reactivity, reflection, and human–agent interaction to life.

¹ Jason Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.”, (2023).

² Shunyu Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models.”, (2023).

³ Yutaro Yamada et al. “The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search.”, (2025).

Chapter 2. Architectures and Patterns: Planning, Reactivity, and Multi-Agent-Systems

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at mcronin@oreilly.com.

You know by now that AI agents are not magic. They are engineered systems whose power comes from the strategic configuration of their architecture. By combining reasoning loops, tools, stateful design, and adaptive control, LLMs can move beyond static prompts and evolve into MAS that plan, react, and reflect in real time. This chapter explores the architectures that make this possible, from structured reasoning methods such as chain of thought, tree of thought, and ReAct to architectural paradigms like supervisor and hierarchical agents. Along the way you will gain a deeper understanding of reasoning flows and see how human-in-the-loop mechanisms including approval gates, corrections, and interruptions keep these systems reliable, aligned, and secure.

The importance of an agent’s architecture becomes obvious when you compare how a simple single-step agent handles a task versus how a multi-step agent does. A single-step agent receives the request, plans everything internally, executes in one go, and returns the result without ever revisiting intermediate steps. A multi-step agent, in contrast, alternates between reasoning and action in an iterative loop, adapting its plan based on partial results or new observations as it works. This flexibility is what allows agents to refine outputs dynamically and cope with uncertainty or feedback from their environment. The contrast is illustrated in Figure 2-1, with Table 2-1 showing how each approach unfolds step by step on the same task.

Table 2-1. Example: Single-step vs. Multi-step (ReAct) execution for the task “Summarize the document and create a bar chart of key statistics”

Step
Single-step agent
Multi-step (ReAct) agent

1

Reads the request once

Reads the request and forms an initial thought on the first action (extract statistics)

2

Plans the entire sequence internally

Action: execute extraction; observation: store extracted data in memory

3

Generates summary and chart in a single pass

Thought: plan next action to generate the summary from extracted data

4

Returns outputs without revisiting intermediate results

Action: generate summary; Observation: store output in memory

5

No mid-execution adjustments

Thought: plan final action to create the bar chart; action: create chart; observation: review output

6

—

If intermediate results are incomplete, repeat the loop until satisfied, then return final outputs

While the table shows the difference step by step, Figure 2-1 illustrates the overall workflows side by side. The single-step agent follows a linear pipeline from prompt to result, whereas the multi-step agent embeds reasoning, actions, and memory into an iterative loop that continues until the problem is solved or the computational budget is exhausted.

Figure 2-1. Single-step (left) versus ReAct (right) agent workflows.

Structured Reasoning and Action

Giving agents distinct personas, sometimes called anthropomorphizing, can sharpen their reasoning and make their contributions more focused. A planner agent framed as a methodical strategist will approach problems differently than a critic agent framed as an evaluator. When each agent is given a clear role and mindset, their reasoning chains tend to be more coherent and their interactions in MAS more purposeful. This alignment improves collaboration, since each agent knows what kind of knowledge it’s expected to contribute.

Yet defining a role is only half the picture. To make agents more effective you might also need to shape how the agent thinks. This is where structured reasoning methods such as chain of thought, tree of thought and ReAct come into play.

Letting Your AI Agents Think Out Loud

Chain-of-thought prompting (CoT) ¹ is inspired by the human thought process, this technique first solves intermediate steps before getting to the final answer. While CoT gets often introduced as prompting trick, in the context of AI agents CoT can serve as added reasoning steps that guides control flow. By forcing the model to generate intermediate steps, you gain better transparency over the reasoning process. It often improves reliability and can help smaller models perform tasks that would otherwise require a larger model. This keeps cost lower while providing clearer insights into how the agent arrives at its decisions. Table 2-2 provides you ideas for prompts for common scenarios that you can adapt to your needs.

Table 2-2. Examples: Chain of Thought prompts by scenario

Scenario
CoT prompt example

Planning

“Plan step by step to create a one-week blog schedule. Explain the reasoning behind each choice.”

Data analysis

“Begin by describing the dataset. Next highlight patterns or trends. Conclude with a clear summary of insights.”

Math problem

“Break the problem into smaller pieces. Solve each piece with explanation. Then combine the results for the final answer.”

Research task

“List the central questions first. Investigate them one by one. Conclude with a synthesized summary.”

Debugging

“Start by listing possible causes. Test each cause in turn. Eliminate what does not fit and explain why until you reach the fix.”

Decision-making

“List all available options. Consider pros and cons step by step. Choose the option that best fits the criteria and explain why.”

To make this more explicit how you might want to implement this into your workflow, let’s look at an agent for data analysis. Since the core of a CoT agent is its ability to reason in a series of logical, intermediate steps, this is the perfect example. You can instruct the agent to think step by step, and have it begin by describing first the dataset and next have it highlight patterns or trends. Example 2-1 shows a generic system prompt you could use for such an agent. I omit the rest of the code for brevity, but you can find the complete working implementation in the book’s repository.

Example 2-1. CoT prompt

SYSTEM_INSTRUCTIONS = """You are a careful data analysis assistant. Think step by
step and be explicit.
Begin by describing the dataset.
Next highlight patterns or trends.
Conclude with a clear summary of insights.
When you need to compute metrics or create a chart, call the python_repl tool with
code that:
1. Loads the CSV from the dataset path
2. Prints descriptive statistics
3. If a target column exists, creates one bar chart of mean values grouped by the
target and saves it to a file
4. Prints key results
Return FINAL ANSWER only after you have executed your analysis."""

Now to use the prompt with your agent, you just need to create a single agent as show in Example 2-2. Note that I use the AgentExecutor from LangChain together with LangGraph for brevity in Example 2-2. But in the accompanying notebook you’ll also find a more granular LangGraph wiring, where the routing between the model and tools is made explicit.

Both approaches yield the same result. However, using the AgentExecutor is often preferable in production, since it keeps your code concise, while the granular graph is valuable for debugging and for understanding how messages, tool calls, and control flow work under the hood.

Example 2-2. CoT Agent

def create_agent(llm: ChatOpenAI, tools: list, system_prompt: str) -> AgentExecutor:
    prompt = ChatPromptTemplate.from_messages(
        [
            ("system", system_prompt + """Work autonomously using the available
            tools."""),
            MessagesPlaceholder(variable_name="messages"),
            MessagesPlaceholder(variable_name="agent_scratchpad"),
        ]
    )
    agent = create_openai_functions_agent(llm, tools, prompt)
    executor = AgentExecutor(
        agent=agent,
        tools=tools,
        return_intermediate_steps=True,
        handle_parsing_errors=True,
        verbose=False,
    )
    return executor

When I ran the code, the agent produced a correct description of the demo dataset, an accurate analysis of the patterns and trends, as well as a bar chart and a summary of its insights. In addition, with the provided code, you can inspect the intermediate tool steps, for example:

(AgentActionMessageLog(tool='python_repl', tool_input={'code': "import pandas
as pd\n\ndf = pd.read_csv('demo.csv')\n\ndf.describe()"})

or

AgentActionMessageLog(tool='python_repl', tool_input={'code': 'df.describe()'})

This example illustrates how CoT enables the agent to make its reasoning process visible and verifiable. If you use the code from this example, you basically force the agent to externalize its intermediate steps, and with that you not only gain confidence in the results but also create a transparent audit trail of how conclusions are reached.

Generating a Tree of Possibilities

Now think of a scenario where you have a more complex task, but you don’t want to use a reasoning model or a full multi-agent architecture. Still, a more exploratory and strategic approach is needed. In such cases, you can use tree of thoughts prompting (ToT) ². ToT expands CoT reasoning into a branching search process, where multiple reasoning paths are explored and evaluated before deciding on the best next step.

In the code examples (Example 2-3 - Example 2-5), I’ll show you how to implement ToT. The key idea is to let a smaller, single AI agent generate multiple potential paths (a “tree”), have another agent evaluate and select the best one, and then execute that chosen path with yet another model. The simplified ToT method offers a pragmatic middle ground between a simple ReAct loop (see “How to Make Your Agent Take Action and Reflect”) and the full computational complexity of more tree-search-based algorithms, which I will cover in the next chapter.

ToT as an engineering shortcut

A lightweight ToT setup where a smaller model proposes branches and a stronger model evaluates can sometimes replace the need for a full multi-agent architecture. This simplifies design and saves both money and time by avoiding repeated calls to large reasoning models when a smaller setup already provides reliable results.

You begin by creating a “tree” of possibilities. The propose_options node in Example 2-3 is the first step. Instead of producing a single, linear output, it uses a smaller LLM to generate three distinct approaches for a blog post. These three options represent branching paths, or “thoughts,” that the agent can explore. This is the core principle of ToT: moving beyond a single-chain process toward multi-path exploration.

Example 2-3. Thought Generation

def propose_options(state: BlogState) -> BlogState:
    """Generator (small): propose 3 creative approaches (ToT-style branching)."""
    proposer = gen_llm.with_structured_output(OptionsPayload)
    prompt = (
        "Generate exactly 3 distinct approaches for a developer-focused blog on:\n"
        f"{state['topic']}\n\nFor each option, provide: title, audience, angle, "
        "a 5-bullet outline, and a concise rationale."
    )
    payload: OptionsPayload = proposer.invoke(prompt)
    state["options_json"] = payload.model_dump_json()
    state["messages"].append(AIMessage(content=state["options_json"]))
    return state

You can then introduce the agent to reflect and select (Example 2-4), which prunes the tree. Here, a more powerful LLM evaluates the three proposed options. The model can be instructed to critique each option based on criteria such as clarity and originality, and then select the best one. This step is crucial in ToT, as it ensures the agent does not simply proceed with the first idea but instead critically assesses the alternatives and chooses the most promising path to pursue.

Example 2-4. Pruning Node

def reflect_and_select(state: BlogState) -> BlogState:
    """Judge (stronger): critique options and select best one."""
    chooser = judge_llm.with_structured_output(ChoicePayload)
    eval_prompt = (
        "Evaluate the 3 approaches for clarity, originality, developer relevance, "
        "and feasibility under time constraints. Pick ONE by index 0..2 and justify."
        f"Options JSON:\n{state['options_json']}"
    )
    choice: ChoicePayload = chooser.invoke(eval_prompt)
    opts = json.loads(state["options_json"])["options"]
    idx = max(0, min(choice.choice_index, len(opts)-1))
    choice.choice_index = idx
    state["choice_json"] = choice.model_dump_json()
    state["messages"].append(AIMessage(content=state["choice_json"]))
    return state

As before, I omit the rest of the code for the other nodes for brevity, but you can find the complete implementation in the book’s repository. What I want to show here is how to tie everything together in LangGraph and which types of LLMs I suggest using. One important paradigm here is the handoff. A handoff defines how control shifts between agents: who speaks next, who acts next, and under what conditions. In LangGraph, nodes encode the logic of the agents, and edges represent the handoffs. By wiring multiple nodes together, you are effectively building a MAS within a single graph. The shared state is the memory and context that all nodes operate on. How this wiring looks in practice for ToT is illustrated in Example 2-5.

Example 2-5. Creating the graph for ToT

graph = StateGraph(BlogState)
graph.add_node("propose", propose_options)
graph.add_node("reflect", reflect_and_select)
graph.add_node("research", research_with_tools)
graph.add_node("draft", draft_outline_and_intro)

graph.add_edge(START, "propose")
graph.add_edge("propose", "reflect")
graph.add_edge("reflect", "research")
graph.add_edge("research", "draft")
graph.add_edge("draft", END)

checkpointer = MemorySaver()
app = graph.compile(checkpointer=checkpointer)

Small creative generator LLM.

Stronger judge/reflector LLM.

Tool-using researcher.

Small writer model.

When you run the code, you’ll see different topics proposed, including reference links. To optimize the results further, you could implement a human-in-the-loop (“Designing Human-in-the-Loop Workflows”) interaction to review and approve the actions.

It’s tempting to think of CoT and ToT as just clever prompting hacks, but you’ve seen they can be powerful engineering tools. Figure 2-2 contrasts both approaches: on the left a single agent reasoning with CoT, on the right a multi-agent setup where different roles collaborate to explore, reflect, and refine mimicking ToT.

Figure 2-2. Control-flow diagram comparing CoT and ToT.

Together, they show how structured reasoning can range from a linear chain of intermediate steps to a branching process of exploration and selection. Both highlight that when agents are guided to think in stages, whether as a single voice or as a coordinated team, the outcome is more deliberate and reliable.

How to Make Your Agent Take Action and Reflect

You’ve seen in the previous section that CoT and ToT can be helpful for reasoning, but reasoning alone isn’t enough. An agent that only reasons never leaves the page, and an agent that only acts can’t learn from its decisions. To combine the two, you need ReAct, which stands for reasoning and action.

Mathematically, you can think of this as mapping a context ct to an action at within an action space A^=A∪L. Here A is the set of task-specific actions that impact the environment, while L is the set of language-based reasoning traces or thoughts. Each action step therefore produces not just a task action at, but also a reasoning trace a^t.

These traces are folded back into the agent’s state so that the context evolves as:

ct+1=(ct,a^t)

In addition, you follow the policy:

π(at|ct)

Here, ct=(o1,a1,…,ot−1,at−1,ot). In other words, the agent carries forward not only what it has done, but also why it chose that path. This recursive structure is what enables reflection: the agent’s future decisions are informed by both its history of actions and the reasoning it has accumulated. The flow diagram in Figure 2-3 illustrates a minimal agent loop implemented in LangGraph.

Figure 2-3. Control-flow diagram of a ReAct agent. This loop is the operational form of the policy π(at|ct).

The process begins at __start__, this hands control to the agent, and then branches depending on the outcome:

If the agent decides it has reached a conclusion, it produces an end signal and moves to __end__.

If more reasoning or action is required, the agent issues a continue signal, possibly calling tools to gather additional information or perform actions.

After interacting with tools, control flows back to the agent, allowing it to integrate results and reflect before deciding whether to continue or stop.

This loop is the essence of ReAct: combining reasoning, action, and reflection in an iterative cycle. ReAct is more than “the model thinks and then acts”. The reasoning trace itself becomes part of the evolving state. That is, the ReAct agent alternates between reasoning traces and tool outputs, and those traces feed back into context.

While this example is in the context of a single agent, the same structure extends naturally into larger MAS. Each agent can maintain its own cycle of reasoning and acting, with reasoning traces shared or exchanged across agents. This makes ReAct not only a tool for individual decision-making, but also a foundation for collaboration and reflection in more complex agentic setups.

Let’s make the math and the diagram concrete, and let me show you how this actually maps into code. The following listings turn the ReAct equations into code. Note, as before, I’ll just show the important parts here, but the full running implementation will be in the accompanying notebook.

Before we jump into the listings, remember the intent: we carry a growing conversation state ct while the model alternates between language traces L and task actions A. Each step appends either a thought, an action, or a tool observation back into the state. The code mirrors this loop. Therefore, you first create the state for the graph as in Example 2-6.

Example 2-6. Create ReAct state

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], add_messages]

The state of the agent.

Context ct is the message stream.

Now you give the model two powers in Example 2-7. It can produce language tokens that act as thoughts L, and it can request structured tool calls A. Binding the tools makes these actions first-class outputs that the runtime can dispatch.

Example 2-7. Init LLM

model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
model = model.bind_tools(TOOLS)

Binding tools to the model augments the action space to A^=A∪L.

Example 2-8 is the policy step. It reads the entire message history, applies the system rules, and produces exactly one thing: either a thought, or a tool call request, or both. You don’t branch here, you simply return what the model decided.

Example 2-8. Call the model

def call_model(state: AgentState, config: RunnableConfig) -> Dict[str, Any]:
    response = model.invoke([SYSTEM_PROMPT] + state["messages"], config)
    return {"messages": [response]}

Policy π(ct)→a^t is model.invoke on the current context.

The reducer appends new messages, implementing ct+1=(ct,a^t).

In Example 2-9 you translate intent into effect. Every requested tool call is executed deterministically, and its result is wrapped as a message. By pushing the observation into the same message stream, you let the next policy step reason over fresh evidence without special cases.

Example 2-9. Execute tool calls

def tool_node(state: AgentState) -> Dict[str, Any]:

    last = state["messages"][-1]
    outputs: list[ToolMessage] = []
    for tc in getattr(last, "tool_calls", []) or []:
        name = tc.get("name")
        args = tc.get("args") or {}
        tool_fn = TOOLS_BY_NAME.get(name)
        if not tool_fn:
            result = json.dumps({"error": f"Unknown tool {name}"})
        else:
            result = tool_fn.invoke(args)
        outputs.append(
            ToolMessage(
                content=result,
                name=name,
                tool_call_id=tc.get("id"),
            )
        )
    return {"messages": outputs}

If the model chose an action at∈A, execute the tool.

Tool output is the observation that informs the next step.

The stopping rule in Example 2-10 is simple. If the model asked for a tool, it must complete that effect before asking it again. If it didn’t, you’d assume it produced a final answer. This keeps the loop predictable and avoids orphaned actions.

Example 2-10. Halting condition

def should_continue(state: AgentState) -> str:
    last = state["messages"][-1]
    if getattr(last, "tool_calls", None):
        return "continue"
    return "end"

Halting test reads the last AI message to decide continue or end.

Now you connect the pieces into a small state machine, as shown in Example 2-11. The edges encode the control flow that mirrors the ReAct diagram. The policy node runs first. If it emits an action, we route to the tools node and then back to policy. If it emits only a thought or a final answer, we terminate.

Example 2-11. Build workflow graph

workflow = StateGraph(AgentState)
workflow.add_node("agent", call_model)
workflow.add_node("tools", tool_node)
workflow.set_entry_point("agent")

workflow.add_conditional_edges(
    "agent",
    should_continue,
    {"continue": "tools", "end": END},
)

workflow.add_edge("tools", "agent")
graph = workflow.compile()

At runtime, the graph advances one edge at a time. The conditional edge decides whether the next hop is tools or END. The fixed edge from tools back to agent completes the think-act loop.

To see this in action, you prompt a question, I asked how the weather will be in Zurich. The LLM first decides which tool to call to answer best, then the tool returns evidence, and the model writes a final answer grounded in that evidence. Because every step is recorded in the message stream, the reasoning is retrievable:

> Entering new ReAct trace...
Question: What is the weather in Zurich today?

Final Answer: What is the weather in Zurich today?

> Finished trace.

Reasoning Summary: selecting a tool based on the query
Action:
```
{
  "action": "internet_search",
  "action_input": {
    "query": "Zurich weather today"
  }
}
```

Observation:
```
{
  "title": "Zurich, Zurich, Switzerland Weather Forecast",
  "snippet": "Hourly Weather \u00b7 1 PM 74\u00b0. rain drop 0% \u00b7 2 PM
  80\u00b0. rain drop 0% \u00b7 3 PM 82\u00b0. rain drop 0% \u00b7 4 PM
  83\u00b0. rain drop 0% \u00b7 5 PM 84\u00b0. rain drop 0% \u00b7 6 PM ...
```

Final Answer: Today in Zurich, the weather is mostly sunny with a high
of 82°F (about 28°C) and a low of 62°F (about 17°C). Winds are light
from the southwest at 6 mph.

For more details, you can check the full forecast
[here](https://www.accuweather.com/en/ch/zurich/316622/weather-forecast/316622).

> Finished trace.

Now you understand how powerful ReAct can be. It’s not just that the agent alternates between thinking and acting, but that the reasoning itself becomes part of the state. This recursive process enables agents to adapt, refine their strategies, and explain their choices. In practice, it allows even relatively small models to perform complex tasks more reliably, while making their decision-making process transparent.

Together, CoT, ToT, and ReAct illustrate how different reasoning strategies can shape the intelligence of your agents. Each approach has its strengths: CoT offers transparency and reliability, ToT explores multiple alternatives before committing, and ReAct closes the loop by combining reasoning with tool use. Table 2-3 gives you a consolidated overview of these reasoning patterns along with tips for when they are most useful.

Table 2-3. Pattern summary: Structured reasoning methods (CoT, ToT, ReAct)

Method
Use Case
Benefits
Tradeoffs

Chain of Thought (CoT)

Step-by-step reasoning for math, planning, analysis

Improves reliability and transparency; enables smaller models on harder tasks

Slower, verbose outputs; sometimes redundant

Tree of Thought (ToT)

Exploratory tasks with multiple solution paths (creative writing, strategy search)

Explores alternatives; reduces “first-answer bias”

Higher compute cost; requires orchestration (selection and pruning)

ReAct

Tasks needing both reasoning and tool use (research, coding, APIs)

Integrates reasoning + action; creates audit trail; adaptive to feedback

More complex state management; risk of over-calling tools

This highlights how these methods can become powerful building blocks once you integrate them into your overall agentic workflow.

Surely You’re Joking, AI Agent!

Sometimes an agent’s behavior can be entertaining. It might call the wrong tool, loop endlessly, or produce an oddly creative response. In those moments you might laugh and say to yourself addressing the AI agent: “You’re joking, right?” But when the same unpredictability reaches into access control, financial operations, or customer data, it’s no longer a joke. An action taken at the wrong moment or on the wrong system can have serious consequences. This is why human-in-the-loop (HITL) exists. HITL allows the agent to pause at critical points, ask for your approval, and continue only once it’s safe to proceed.

HITL is not only about catching errors before they happen. It also provides accountability and transparency. By inserting checkpoints into an agent’s workflow, you make sure irreversible actions are reviewed, decisions are traceable, and sensitive operations stay under human control. Far from slowing the system down, this oversight makes agents more trustworthy and easier to adopt in production settings.

Designing Human-in-the-Loop Workflows

There are a couple of different interruptions that you can perform with a HITL workflow. The most common ones are shown below.

Approve or reject

Pause the graph before a critical step, such as an API call, to review and approve the action. If the action is rejected, you can prevent the graph from executing the step, and potentially take an alternative action. This pattern often involve routing the graph based on the human’s input.

Review and edit

Pause the graph to review and edit the graph state. This is useful for correcting mistakes or updating the state with additional information.

Review tool calls

If a tool call could affect external systems, inspect its arguments before execution. For example, you might confirm booking details before a travel reservation is made.

Validate human input

Explicitly request human input at a particular step in the graph. This is useful for collecting additional information or context to inform the agent’s decision-making process or for supporting multi-turn conversations.

LangGraph offers an extensive documentation on how to apply these patterns in your own applications. In addition, the book’s repository includes complete, runnable examples of six different patterns. You can select among them through a main() function, as shown in Example 2-12.

Example 2-12. Main function for HITL pattern demos

def main():
    menu = """
Pick a demo
1. Human feedback loop for writing
2. Approval gate before API call
3. Review and edit state
4. Parallel interrupts with resume map
5. Tool call review in a tiny ReAct loop
6. Static interrupts for debugging
q. Quit
> """

In the following code examples (Example 2-13 - Example 2-20) I’ll explain how to implement the patterns covered above and also cover the main points you’d want to watch out for when you implement them.

Command in LangGraph

Combines control flow (edges) and state updates (nodes). It facilitates multi-actor (or multi-agent) communication. For example, a node can update its state and decide the next node to visit. LangGraph enables this by returning a Command object from node functions.

Use approve or reject when you want to gate a side-effecting step like an HTTP call.

Example 2-13 asks the LLM to produce a request plan and stores it in proposed_request.

Example 2-13. Propose request with LLM

def b_propose(state: BState) -> BState:
    prompt = "Return only JSON with keys url and params for GET to
              https://httpbin.org/get using q and limit."
    text = LLM.invoke(prompt).content
    m = re.search(r"\{.*\}", text, re.S)
    data = {"url": "https://httpbin.org/get", "params":
           {"q": "fallback", "limit": 1}}
    if m:
        try:
            data = json.loads(m.group(0))
        except Exception:
            pass
    return {"proposed_request": data}

Ask the LLM to generate a structured plan.

Fallback default request if parsing fails.

Try to parse JSON out of the model’s output.

Store result in state under proposed_request.

Example 2-14 calls interrupt(…​) to pause and request human approval or a revision.

Example 2-14. Approval gate for request

def b_gate(state: BState) -> Command[Literal["b_call", "b_propose"]]:
    v = interrupt({
        "question": "Approve or revise request",
        "proposed_request": state["proposed_request"],
        "schema": {
            "type": "object",
            "properties": {
                "action": {"enum": ["approve", "revise"]},
                "update": {"type": "object"}
            },
            "required": ["action"]
        }
    })
    action = v.get("action")
    if action == "approve":
        return Command(goto="b_call", update={"decision": "approved"})
    upd = v.get("update") or {}
    new_req = state["proposed_request"].copy()
    if "url" in upd:
        new_req["url"] = upd["url"]
    if isinstance(upd.get("params"), dict):
        new_req.setdefault("params", {}).update(upd["params"])
    return Command(goto="b_gate", update={"proposed_request": new_req,
    "decision": "revised"})

Pause execution and wait for human decision.

Present the proposed request for inspection.

Enforce structured approval/revision schema.

On approval, route to the call step.

On revision, loop back to the gate with updated request.

On resume you pass Command(resume={"action": "approve"}) to continue, or {"action": "revise", "update": {...}} to merge edits and loop back to the gate.

Example 2-15 runs the HTTP call only after approval and stores api_result. You can reject to prevent execution, and routing depends on human input.

Example 2-15. Execute approved HTTP call

def b_call(state: BState) -> BState:
    import requests
    r = requests.get(state["proposed_request"]["url"],
    params=state["proposed_request"]["params"], timeout=10)
    return {"api_result": {"status_code": r.status_code, "url": r.url}}

Run the external request only after approval.

Store result under api_result in state.

You use review and edit to correct or improve model output by writing directly to the state. Here, Example 2-16 generates a short summary with the LLM and stores it in summary.

Example 2-16. Generate summary text

def c_write(state: CState) -> CState:
    text = LLM.invoke("Write 2 sentences about why human
                     in the loop matters for agents").content
    return {"summary": text}

Ask the LLM to produce a candidate summary.

Store summary in state for later review.

Example 2-17 pauses with interrupt(...) that includes a JSON schema asking for edited_text.

Example 2-17. Interrupt for human edit

def c_edit(state: CState) -> CState:
    res = interrupt({
        "task": "Edit the summary text",
        "summary": state["summary"],
        "schema": {
            "type": "object",
            "properties": {"edited_text": {"type": "string"}},
            "required": ["edited_text"]
        }
    })
    return {"summary": res["edited_text"]}

Pause and request edit input from human.

Provide the generated summary for inspection.

Restrict edits to a string field.

Overwrite summary with human revision.

You resume with Command(resume={"edited_text": "your revision"}). The node restarts and returns your edit instead of pausing, then the graph updates summary with the human version. Meaning, you literally pause to review then overwrite the graph state with the edited content.

You use review tool calls when a tool may affect external systems and needs inspection of arguments. Example 2-18 wraps any tool with an interrupt([...]). The interrupt payload asks for accept, edit, or respond.

Example 2-18. Wrap tool with review gate

def add_hitl(tool_obj: BaseTool | Any) -> BaseTool:
    if not isinstance(tool_obj, BaseTool):
        tool_obj = tool(tool_obj)

    @tool(tool_obj.name,description=tool_obj.description,
    args_schema=tool_obj.args_schema)
    def wrapped(**tool_input):
        request = [{
            "action_request": {"action": tool_obj.name, "args": tool_input},
            "config": {"allow_accept": True, "allow_edit":
                       True, "allow_respond": True},
            "description": "Review this tool call"
        }]
        response = interrupt(request)[0]
        if response["type"] == "accept":
            return tool_obj.invoke(tool_input)
        if response["type"] == "edit":
            new_args = response["args"]["args"]
            return tool_obj.invoke(new_args)
        if response["type"] == "response":
            return response["args"]
        raise ValueError("Unsupported interrupt response type")
    return wrapped

Wrap original tool call in a review request.

Pass tool name and arguments for inspection.

Allow human to accept, edit, or respond.

Pause until decision is received.

Execute original tool call if accepted.

Execute tool with revised arguments.

Replace tool output with human response.

You can resume with on of:

{"type": "accept"}
{"type": "edit", "args": {"args": {"query": "weather in NY"}}}
{"type": "response", "args": "Skip the tool right now"}

The wrapper either executes the original tool call, executes with edited args, or returns a human message instead of calling the tool. With that, you inspect and approve tool calls before they hit the outside world.

Interrupts and side effects

Never put side-effecting code (API calls, DB writes) in the same node as interrupt(...). Always pause first, then execute effects only after explicit approval to avoid repeats when resuming.

There might be also the occasion, that you’d like to get parallel inputs with a single resume. In Example 2-19 you see how you can implement this.

Example 2-19. Parallel human interrupts

class DState(TypedDict, total=False):
    text_1: str
    text_2: str

def d_h1(state: DState):
    v = interrupt({"text_to_revise": state["text_1"]})
    return {"text_1": v}

def d_h2(state: DState):
    v = interrupt({"text_to_revise": state["text_2"]})
    return {"text_2": v}

def build_graph_D():
    g = StateGraph(DState)
    g.add_node("d_h1", d_h1)
    g.add_node("d_h2", d_h2)
    g.add_edge(START, "d_h1")
    g.add_edge(START, "d_h2")
    g.add_edge("d_h1", END)
    g.add_edge("d_h2", END)
    return g.compile(checkpointer=CHECKPOINTER)

Interrupt for first text input.

Update state with revised text_1.

Interrupt for second text input.

Update state with revised text_2.

Add both nodes to the graph.

Both nodes branch directly from START.

Compile with checkpointer so both interrupts can resume.

Here, both nodes start from START and each calls interrupt(...). You use Example 2-20 to build a dictionary that maps interrupt_id to its value, and then resume with Command(resume=that_map).

Example 2-20. Resume multiple interrupts

def wait_for_interrupt_and_prompt(app, cfg):
    state = app.get_state(cfg)
    ints = getattr(state, "interrupts", []) or []
    if not ints:
        print("No interrupts pending")
        return None

    if len(ints) > 1:
        print("\nMultiple interrupts pending:")
        for i, it in enumerate(ints, 1):
            print(f"[{i}] id={it.interrupt_id} value={jdump(it.value)}")
        print("Enter values per interrupt. Leave blank to echo original.")
        resume_map = {}
        for it in ints:
            val = input(f"Value for {it.interrupt_id}: ").strip()
            if val:
                try:
                    resume_map[it.interrupt_id] = json.loads(val)
                except Exception:
                    resume_map[it.interrupt_id] = val
            else:
                resume_map[it.interrupt_id] = it.value
        return Command(resume=resume_map)

Collect any pending interrupts from state.

Handle multiple interruptions at once.

Print each interrupt ID and its current value.

Ask the human for a new value per interrupt.

Try to parse as JSON for structured edits.

Keep the original value if no input is given.

Resume execution with a full map of updated values.

As you’ve seen, there isn’t a single “right” way to add human oversight. You can choose from different patterns depending on the sensitivity of the task and the amount of control you want to keep. I encourage you to experiment with these patterns in the provided notebook, as this makes it easier to see how human oversight can evolve from simple approval gates into richer collaborations between human and agents.

Advanced Agent Paradigms: Supervisor and Hierarchical Agents

If your tasks start getting more complex than the ones you’ve seen so far, or if you notice your agent struggling, it can help to break the task into smaller subtasks. Think of it like building a team: each person brings a different skill to the table, and together they solve the larger challenge.

In the same way, you can introduce smaller, independent agents that form a MAS. One common setup is the supervisor architecture, where every agent reports to a single supervisor that decides who should act next. A more advanced version is the hierarchical architecture, where you don’t just have one supervisor, but supervisors of supervisors. This generalization allows for more flexible control flows.

The diagram in Figure 2-4 shows how a single-agent setup compares to these two multi-agent patterns.

Figure 2-4. Comparison of single, supervisor, and hierarchical agent architectures.

The primary benefits of using a MAS are its modularity, specialization, and control. By separating functionality across agents, you make the overall system easier to develop, test, and maintain. Each agent can be designed as a domain expert, which boosts the system’s performance as a whole. And unlike simple function calling, a multi-agent setup also gives you explicit control over how agents communicate with one another.

Building a Hierarchical Research Team

To make the idea of supervisors and sub-agents concrete, let’s walk through an example where you build a hierarchical research team. I’ll use LangGraph’s prebuilt, reusable create_react_agent component to fast-track the implementation of a ReAct agent. Each worker agent then gets a clear role: some focus on web search, others scrape pages, query Exa for more structured search results, or even look up patents on Google. A supervisor agent sits on top, coordinating the specialists and deciding who should act next. The end goal is to assemble a market research team. Figure 2-5 shows the final application you’re building.

Figure 2-5. Final application with a hierarchical architecture.

But before you get there, you’ll start smaller by building your first supervisor setup: the research team. You see the overview of the team you create in Figure 2-6. Once that’s in place, the second step is to add a writing team. This group will take the output from the research team and save it to your file system as a simple text file.

Figure 2-6. Flow diagram for the research team.

I’ll leave out the details of how to create tools, since you already learned this in chapter one (Example 1-2).

The first thing you need is a helper (Example 2-21) to create your supervisor. This function acts as a router (see “Mapping FSM/HSM to Agent Frameworks”) that decides which worker should take the next step, or whether the process should finish.

Example 2-21. Helper function for supervisor

class State(MessagesState):
    next: str

def make_supervisor_node(llm: BaseChatModel, members: list[str]) -> str:
    options = ["FINISH"] + members
    system_prompt = (
        "You are a supervisor tasked with managing a conversation between the"
        f" following workers: {members}. Given the following user request,"
        " respond with the worker to act next. Each worker will perform a"
        " task and respond with their results and status. When finished,"
        " respond with FINISH."
    )

    class Router(TypedDict):
        """Worker to route to next. If no workers needed, route to FINISH."""

        next: Literal[*options]

    def supervisor_node(state: State) -> Command[Literal[*members, "__end__"]]:
        """An LLM-based router."""
        messages = [
            {"role": "system", "content": system_prompt},
        ] + state["messages"]
        response = llm.with_structured_output(Router).invoke(messages)
        goto = response["next"]
        if goto == "FINISH":
            goto = END

        return Command(goto=goto, update={"next": goto})

    return supervisor_node

Create the state to keep track of the next worker.

Define a helper that builds a supervisor node for any set of agents.

Provide a system prompt that tells the LLM how to act as the supervisor.

Use structured output to reliably decide which worker to call next.

Allow the workflow to terminate cleanly when the supervisor says FINISH.

Next, you need a helper to create your worker nodes, how to implement this is shown in Example 2-22. This makes the setup concise and reusable, so you don’t have to repeat the same boilerplate each time you add a new agent.

Example 2-22. Helper function for node creation

def make_react_worker_node(
    *,
    llm: ChatOpenAI,
    name: str,
    tools: list,
    prompt: str | None = None,
    goto: str = "supervisor",
):
    agent = create_react_agent(llm, tools=tools, prompt=prompt)

    def node(state: State) -> Command[Literal["supervisor"]]:
        result: Dict[str, Any] = agent.invoke(state)
        msgs: Sequence[BaseMessage] = result.get("messages", [])
        content = getattr(msgs[-1], "content", "") if msgs else ""
        return Command(
            update={"messages": [HumanMessage(content=content, name=name)]},
            goto=goto,
        )

    return node

Create a specialized ReAct agent with its tools and role prompt.

Invoke the agent with the current state.

Pass the agent’s latest output back into the state and return to the supervisor.

Once you have these helpers, you can move on to defining your agents. It’s helpful to give each agent a specific role. Since the goal here is to build a market research team, the roles should be very targeted. My role prompts are shown in Example 2-23.

Example 2-23. Assign roles to research agents

SEARCH_PROMPT = """Role: Web researcher. Use the search tool and return a
concise research note with sources. No follow-up questions."""

SCRAPER_PROMPT = """Role: Web scraper. Use the scraping tool to fetch details
from given URLs and summarize key findings. No follow-up questions."""

EXA_PROMPT = """Role: Research assistant. You can search for all recent info
on Exa Search. Your response should clearly articulate the key points you found."""

PATENT_PROMPT = """Role: Market researcher with 20 years of experience.
You are very knowledgeable in patent research and in finding up-to-date info
about patents using the Google Patents API."""

Now, let’s create the agents. Using the helper from Example 2-22, you can instantiate each worker and assign it the appropriate role and tool. The implementation is shown in Example 2-24.

Example 2-24. Create research agents

specs = [
    dict(
        name="search",
        tools=[tavily_tool],
        prompt=SEARCH_PROMPT,
    ),
    dict(
        name="web_scraper",
        tools=[scrape_webpages],
        prompt=SCRAPER_PROMPT,
    ),
    dict(
        name="exa_search",
        tools=[exa_search_tool],
        prompt=EXA_PROMPT,
    ),
    dict(
        name="patent_research",
        tools=[patent_search],
        prompt=PATENT_PROMPT,
    ),
]

nodes = {s["name"]: make_react_worker_node(llm=llm, **s) for s in specs}

search_node = nodes["search"]
web_scraper_node = nodes["web_scraper"]
exa_search_node = nodes["exa_search"]
patent_research_node = nodes["patent_research"]

Define each worker with its tools and role.

Use the helper to instantiate all workers without repeating boilerplate.

To complete the supervisor architecture, we need to set up a supervisor that can coordinate across all four research agents. This ensures the workflow doesn’t stall and that each agent contributes at the right time. See Example 2-25 for the implementation.

Example 2-25. Create research supervisor node

research_supervisor_node = make_supervisor_node(
    llm, ["search", "web_scraper", "exa_search", "patent_research"]
)

Create a supervisor that coordinates across all four research agents.

Finally, we need to build the actual graph by registering nodes and connecting the edges that define how control flows between them. This results in a fully functioning research team graph, as shown in Example 2-26.

Example 2-26. Build research team graph

research_builder = StateGraph(State)

research_builder.add_node("supervisor", research_supervisor_node)
research_builder.add_node("search", search_node)
research_builder.add_node("web_scraper", web_scraper_node)
research_builder.add_node("exa_search", exa_search_node)
research_builder.add_node("patent_research", patent_research_node)

research_builder.add_edge(START, "supervisor")
research_builder.add_edge("search", "supervisor")
research_builder.add_edge("web_scraper", "supervisor")
research_builder.add_edge("exa_search", "supervisor")
research_builder.add_edge("patent_research", "supervisor")

research_graph = research_builder.compile()

Register supervisor and worker nodes.

Connect edges so all workers route back to the supervisor.

Compile the graph into an executable workflow.

To test this setup, you can run the code from Example 2-27.

Example 2-27. Run research team with prompt

for s in research_graph.stream(
    {"messages": [("user", """What are AI agents? Are there any patents out there
                              about LLM agents?""")]},
    {"recursion_limit": 100},
):
    print(s)
    print("---")

Run the research graph on a query; the supervisor orchestrates which agents act.

This returns a short description AI agents and then uses the patent tool to check for recent patents. The steps look like this (I’ve left out the actual messages between agents for brevity):

{'supervisor': {'next': 'search'}}
{'supervisor': {'next': 'patent_research'}}
{'supervisor': {'next': '__end__'}}

You can set up your writing team in the same way. The notebook for this section includes the full working code, but Figure 2-7 gives you a quick overview of what the writing team looks like.

Figure 2-7. Flow-chart for writing team.

This modular approach makes it easy to compose increasingly capable MAS step by step: first by wiring up specialized workers, then by combining entire teams into larger graphs. In practice, this means you can start small—say with search, scraping, and patent lookup in a research team, and then plug that into a writing team to form a full end-to-end pipeline, or extend it even further with internal data and chart writing and plotting agents.

To move from individual teams to a proper hierarchy, you’ll need to build another graph that ties everything together. Think of this as your “super graph,” shown in Example 2-28.

Example 2-28. Build hierarchical architecture

super_builder = StateGraph(State)
super_builder.add_node("supervisor", teams_supervisor_node)
super_builder.add_node("research_team", call_research_team)
super_builder.add_node("writing_team", call_paper_writing_team)

super_builder.add_edge(START, "supervisor")
super_graph = super_builder.compile()

Once that’s in place, you can prompt your hierarchical MAS as shown in Example 2-29.

Example 2-29. Run hierarchical MAS

TARGET_FILE = "semiconductor_whitepaper.txt"

TASK_MSG = f"""
Write an 800-word research report white paper on semiconductor development.
Start with an executive summary of your findings.
Search for relevant recent patents and include links to them.
IMPORTANT: Provide links to all your sources.
Finally, save the full report to disk as a .txt file using the write_document tool.
Use file_name="{TARGET_FILE}".
"""

for step in super_graph.stream(
    {
        "messages": [
            ("user", TASK_MSG.strip())
        ],
    },
    {"recursion_limit": 150},
):
    print(step)
    print("---")

Running this setup will result in the following sequence of supervisor decisions (messages again omitted):

{'supervisor': {'next': 'research_team'}}
{'supervisor': {'next': 'writing_team'}}
{'supervisor': {'next': 'writing_team'}}
{'supervisor': {'next': '__end__'}}

At the end of the run, the report is successfully saved to semiconductor_whitepaper.txt. In practice, this workflow takes only about two minutes to gather the research, generate the text, and write everything to file.

Developing a Swarm of Agents

Another important paradigm is the swarm architecture. Here, agents handoff control to one another dynamically based on their specializations. Unlike a strict hierarchy, swarms emphasize peer-to-peer collaboration. The system keeps track of which agent was last active so that subsequent interactions continue seamlessly with the right one. This back-and-forth exchange allows work to bounce naturally between agents, such as when a researcher gathers sources and then hands them to a writer, who may in turn request more data before finishing the synthesis. Figure 2-8 illustrates this continuous flow.

Figure 2-8. Handoff flow between a swarm of agents.

LangGraph provides an open-source library to fast track the development of swarms of agents. By default, the agents in the swarm use handoff tools created with the prebuilt create_handoff_tool. You can also create your own custom handoff tools to better fit your workflow. For example, you might:

Change the tool name or description to make the handoff purpose clearer.

Add tool call arguments for the LLM to populate, such as a specific task description for the next agent.

Control what data is passed to the next agent. By default, create_handoff_tool passes the full message history (all prior agent messages) and a tool message confirming the handoff, but you may choose to send only a filtered subset of context.

Before wiring up a full swarm, you first need to define the handoff helpers as shown in Example 2-30. These tools act as signals that let one agent pass control to another. In the example below, the research assistant can handoff to the writer once enough sources are collected, and the writer can hand control back if more evidence is needed.

Example 2-30. Create handoff tools

to_writer = create_handoff_tool(
    agent_name="writer_assistant",
    description="""Transfer to the writing assistant to synthesize sources into
                an answer.""",
)
to_research = create_handoff_tool(
    agent_name="research_assistant",
    description="""Return to the research assistant to fetch or scrape more
                sources.""",
)

Defines a tool that lets the research assistant handoff to the writing assistant.

Defines the reverse tool so the writer can send control back to the research assistant.

Once you have the handoff helpers in place, you can move on to defining your agents. It’s helpful to keep the roles very distinct. In this setup you want one agent focused purely on research, and the other focused on writing. The handoff tools ensure the workflow can bounce back and forth when more sources are needed or when it’s time to synthesize. The roles are shown in Example 2-31.

Example 2-31. Assign roles to research and writing assistants

research_assistant = create_react_agent(
    model=llm,
    tools=[tavily_tool, scrape_webpages, to_writer],
    prompt=(
        """You are a research assistant. Search the web with Tavily.
        When you have 3 to 5 solid sources, scrape key pages for details,
        then hand off to the writer assistant."""
    ),
    name="research_assistant",
)

writer_assistant = create_react_agent(
    model=llm,
    tools=[to_research],
    prompt=(
        """You are a writing assistant. Read the provided Documents and messages.
        Synthesize a concise answer with citations by site name in brackets. If
        sources are thin or unclear, hand back to research with a short request."""
    ),
    name="writer_assistant",
)

Defines the research assistant, responsible for web search and scraping.

Grants the research assistant access to its tools, including the handoff to the writer.

Defines the writing assistant, responsible for synthesis and citation.

Allows the writer to hand control back to the research assistant if more evidence is needed.

Now that both roles are defined, we can wire them into a swarm. The swarm provides the coordination mechanism: it starts with the research assistant by default, but allows handoff to the writer and back again until the task is complete. The implementation is shown in Example 2-32.

Example 2-32. Build research–writer swarm

swarm = create_swarm(
    agents=[research_assistant, writer_assistant],
    default_active_agent="research_assistant",
).compile()

To test the system, you can run the following example prompt. Here, I requested a summary of fintech licensing in Switzerland. The swarm first searches for sources, then hands off to the writing assistant for synthesis, and if needed returns to research for more data. See Example 2-33 for the prompt.

Example 2-33. Run research–writer swarm

user_request = {
    "messages": [
        {
            "role": "user",
            "content": (
                """Find the current Swiss fintech licensing options for small
                 startups. Gather authoritative sources and produce a short
                 summary with three bullet points and references."""
            ),
        }
    ]
}

for chunk in swarm.stream(user_request):
    print(chunk)
    print()

Defines a user prompt asking for fintech licensing details in Switzerland.

Streams the results from the swarm, showing the handoff-driven workflow in action.

The run produces three clean bullet points with references, citing official Swiss regulator pages and trusted financial sources. However, if the writer finds the evidence too thin, it can trigger a return handoff to research. The full cycle takes just about 30 seconds to gather the information and generate the final structured output.

The supervisor, hierarchical, and swarm architectures from this section show how different coordination strategies affect MAS. Supervisors provide centralized control, hierarchies allow scalable organization into teams of teams, and swarms emphasize flexible peer-to-peer collaboration. Table 2-4 compares these patterns and highlights when to use each one.

Table 2-4. Pattern summary: Multi-agent coordination strategies

Architecture
When to Use
Benefits
Tradeoffs

Supervisor

Small teams of agents with clear task boundaries

Simple setup; strong control; avoids deadlocks

Central bottleneck; single point of failure

Hierarchical

Larger, multi-team workflows requiring scalability

Modular; scalable; mirrors organizational structures

Harder to debug; requires careful design

Swarm

Dynamic collaboration between peers (e.g., research ↔ writer)

Flexible; natural back-and-forth flow

Less predictable; requires robust handoff design

This shows how each coordination pattern supports different scales and styles of agent collaboration. By mixing and matching them, you can shape agent workflows that balance control, scalability, and adaptability within your overall MAS design.

Conclusion

The big takeaway from this chapter is that an AI agent’s intelligence comes also from its architecture, not just from the LLM it uses. In light of this, you explored how architectural patterns shape the intelligence and reliability of AI agents. By moving beyond single-step execution, you saw how reasoning loops like CoT, ToT, and ReAct embed structure into the agent’s thought process, making outputs more deliberate and transparent.

Equally important, you saw how human-in-the-loop can make your agents more secure. Interrupts, approval gates, and review steps demonstrated how guardrails make agents trustworthy in real-world settings where errors carry real consequences. The balance between autonomy and oversight is a defining theme of modern agent design.

You also learned how these reasoning methods extend into MAS. Supervisor and hierarchical architectures distribute tasks across specialized workers, while swarm setups enable flexible handoffs between peers. Together, these paradigms show that agent intelligence is not just about stronger models, but about how those models are orchestrated into systems that can plan, act, and reflect.

With these foundations in place, you are ready to move to the next level. In the following chapter, you will explore advanced paradigms that extend these architectures toward scalable execution, deeper planning strategies, and more sophisticated reasoning methods. This will show you how to design agents that can not only react and reflect, but also plan at scale and coordinate complex workflows across larger systems.

¹ Jason Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, https://arxiv.org/abs/2201.11903 (2023).

² Shunyu Yao et al. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models.”, https://arxiv.org/abs/2305.10601 (2023).

Chapter 3. Advanced Planning, Reasoning, and Scalable Execution in Agents

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 3rd chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at mcronin@oreilly.com.

In the last two chapters, you saw that AI agents aren’t magic, they’re engineered systems. But the techniques you’ll learn in this chapter may start to feel close. As Arthur C. Clarke once said:

Any sufficiently advanced technology is indistinguishable from magic. ¹

That sense of magic comes from what happens when agents stop reacting and instead start learning from their own experience or making smarter choices at test time. They are no longer non-player characters (NPCs), because they start to learn and adapt on their own when you apply the principles of reinforcement learning (RL) to LLMs. This is how Clarke’s quote finds new meaning in the world of AI agents.

But what may look uncanny at first, actually comes from a set of clear mechanisms. This chapter explains those mechanisms in depth: you’ll learn how RL builds a feedback loop between reasoning and outcomes, how tree-based search and adaptive planning lets agents simulate multiple futures before acting and decide if it’s better to explore more solutions or rather to refine a potential solution.

Each of these mechanisms contributes to the illusion of self-improvement, which is of course, the refining process by which agents become better over time. Agents gather feedback, reflect on their choices, and refine their behavior in ways that look alive. Rewards, feedback loops, and dynamic replanning connect with RL frameworks such as Agent Reinforcement Training (ART), Relative Universal LLM-Elicited Rewards (RULER) and reinforcement learning for language models (rLLM) to form a cycle of continuous refinement that spans across training and deployment.

In this chapter, you cross the threshold between design and autonomy. Your AI agents stop following scripts and start solving problems on their own. When foresight, reinforcement, and scalable execution work together this is the pivotal moment when you get the impression that your AI agents are no longer purely orchestrated. Here, you give your agents the awareness of choice and improvement via RL you build your system to learn, reason, and improve. In short, this chapter will help you design the emergence of agency for your agents.

Beyond Zero Sum Games: From Rewards to Reasoning

If you ever wondered where the learning in RL actually happens, in this section you’ll learn where the story begins. Everything starts with rewards, the signals that tell an agent whether it did well or not. But rewards alone don’t make an intelligent system. They simply create the feedback that drives it to improve.

At its core, RL is actually simple. Seriously, I mean it. Think about it for a moment, if you look closer, an agent takes an action, observes what happens, and uses that outcome to make a better choice next time. When this cycle repeats, patterns start to form. The agent begins to recognize what works and what doesn’t. That is the moment when the loop between perception and adjustment closes, and behavior starts to look intentional. You already do this in real life. You use RL every day without thinking about it, either learning from your own experiences or how you teach your kids or pets.

The only difference is that for LLM agents this loop is in a different space; it isn’t physical. It happens in language, thought, and reasoning. Every message, every tool call, and every piece of intermediate output becomes part of a trajectory. A trajectory is the full record of how an agent reached an answer, step by step, from the first prompt to the final result. Figure 3-1 captures the fundamentals of this learning loop in a single glance: the moment where action turns into feedback and feedback turns into improvement.

Figure 3-1. Agent learning loop: from action to feedback to improvement

When you collect multiple trajectories, you get rollouts. Each rollout is one complete story of reasoning. Some reach the goal, some fail, but all of them teach the agent something about how its own decisions lead to different outcomes. RL turns these experiences into structure, by turning raw sequences of text into measurable cause and effect. Table 3-1 gives you a conceptual compass of these concepts before we start looking into how you can apply them.

Table 3-1. RL Concepts

Concept
What it means
Why it matters for agents

Reward

A signal that tells the agent whether an outcome was good or bad.

Rewards close the loop between action and consequence. Without them, the agent has no sense of progress.

Trajectory

The full sequence of reasoning steps, actions, and tool calls that lead to an outcome.

A trajectory captures how the agent reached a decision, not just what it decided.

Rollout

A complete run of one or more trajectories for a task.

Rollouts record the agent’s experiences so they can be compared, scored, and used for learning.

Policy

The model’s internal decision rule that maps context to the next action.

RL updates the policy so the agent makes better choices over time.

Reward or Judge Model

A model that evaluates trajectories and assigns rewards based on quality or correctness.

Turns raw outcomes into learning signals that drive improvement.

Relative Evaluation

Comparing multiple rollouts to decide which performed best.

Removes the need for handcrafted rewards and allows self-improvement from comparison.

The agent doesn’t need a teacher to label every step. It only needs a way to compare its own trajectories and decide which ones lead to better reasoning. This is where reward models come in. Early systems used simple, hand-crafted rules, but that approach doesn’t work for LLM-based agents. Modern agents use relative evaluation to learn directly from comparison. They generate several rollouts for the same task and rank them to see which reasoning path performed best.

Once the agent starts to do that, it moves beyond a zero sum game. It no longer tries to win a single round. It learns how to improve its entire process. The goal shifts from getting the answer right to reasoning more effectively.

This is the first real magic trick of RL for agents: the model begins to learn from its own trajectories. It starts to use rollouts as memory and feedback as guidance. What was once trial and error becomes deliberate improvement in your system.

Better in Groups: Why Relative Ranking Matters

A single trajectory tells you what happened, while a group of trajectories shows you what could have happened. Comparing them is where real learning begins.

In traditional RL, the model learns from numeric rewards that describe how well each attempt performed. But for language-based agents, those numbers are hard to define. You can’t easily score creativity, coherence, or depth of thought with a single metric.

Relative ranking solves that problem. Instead of assigning absolute scores, you let the model compare several rollouts for the same task and decide which one worked best. The feedback doesn’t have to be perfect. It only needs to be consistent within the group. If one trajectory is slightly better than another, that difference already carries enough information to guide improvement.

Learning on a Leash: KL Divergence

When you teach an agent to learn from its own experience, there is always a risk that it learns too fast or in the wrong direction. In RL, this problem is not philosophical, it’s mathematical. If the new policy drifts too far from what the model already knows, it starts to forget useful behaviors.

The Kullback–Leibler divergence, or KL divergence, is the part of the equation that keeps that from happening. It measures how much the new probability distribution Q of the updated policy differs from the reference distribution P. In other words, it tells you how far the new “way of thinking” has moved from the old one.

If that distance becomes too large, the model begins to overfit to recent feedback and loses balance. By adding a KL penalty to the learning objective, you keep improvement under control. The agent can still explore and adapt, but not to the point of breaking its past understanding.

You can think of it as a leash on curiosity. The agent can wander, but it can’t run away too far. Each update is a negotiation between two instincts: the drive to improve and the need to remain stable.

This idea forms the basis of Group Relative Policy Optimization (GRPO) ². and Group Sequence Policy Optimization (GSPO) ³. Both methods let agents learn from relative feedback rather than fixed targets. GRPO focuses on the token level, while GSPO refines the process at the sequence level, helping agents reason more efficiently step by step.

From Token-Based Rewards to Sequence-Based Learning

With GRPO and GSPO, learning becomes social in a sense. Each trajectory teaches the agent about the others. Success and failure stop being isolated events and turn into references. The agent no longer asks “Was I right?” but “Was this reasoning better than before?”

This shift changes everything. It turns RL from a scoring game into a process of reflection. The agent doesn’t just correct mistakes, it learns how to think and act better next time.

GRPO

GRPO builds on a simple idea: a model learns better when it can compare its own answers. Instead of scoring each response with an absolute reward, GRPO collects several responses to the same query, ranks them, and learns from those relative rankings. Figure 3-2 shows a high-level overview of how this process works.

Figure 3-2. High-level GRPO mechanics: group-normalized rewards combined with a reference policy KL term.

The process begins by sampling prompts and generating multiple completions from the current policy. Each completion is then evaluated by a reward function or judge model, which assigns an individual score ri. These scores are normalized within each group, centering and scaling their values to create a set of relative comparisons that serve as the update signal for the policy.

Who’s Teaching What and How: PPO vs DPO vs GRPO vs GSPO

If you’re wondering which common RL methods actually require an extra reward or judge model, Table 3-2 makes this clear. PPO, as used in RLHF, combines a learned reward model with a critic for stability and variance reduction. DPO removes both, optimizing directly on pairwise preferences with an explicit KL to a reference. GRPO follows the PPO idea but replaces absolute rewards with relative rankings across groups, removing the critic while keeping ratio clipping and KL regularization. GSPO extends GRPO to sequence-level optimization, which improves stability on long or complex outputs.

Table 3-2. Which method is doing what?

Method
Reference policy
Reward or judge
Critic model
Optimization unit

PPO (RLHF)

Usually a frozen supervised model

Learned reward model Rϕ from preferences

Yes, value head or separate critic

Token-level with advantages

DPO

Frozen reference πref

None learned at train time, uses pairwise human prefs (chosen vs rejected)

No critic

Sequence-level, pairwise preference loss

GRPO

Often πref=πold or a frozen ref

Judge or reward function, usually LLM-as-judge or programmatic scoring, per sample

No learned critic

Token-level, but advantages are group-normalized across multiple candidates

GSPO

Often πold or a frozen ref

Judge or reward function, same spirit as GRPO

No critic

Sequence-level importance ratio, group-normalized

Each group of responses becomes a small ecosystem of experience. The model looks at its own work, compares outcomes, and updates its policy so that higher-ranked reasoning paths become more likely next time. This removes the need for a separate value model and stabilizes training through shared context.

Next, the KL divergence term is estimated between the current and reference policies. Before that, let’s define how the model assigns probabilities to each sequence. For a query q and a response o, the sequence likelihood is defined as:

πθ(o∣q)=∏t=1|o|πθ(ot∣q,o<t),

where |o| is the number of tokens in the response. This factorization reflects how an autoregressive model predicts each next token conditioned on all previous ones.

The KL divergence then measures how much the updated policy diverges from the reference one:

𝔻KL(πθ‖πref)=𝔼q,o~πθ(·∣q)[logπθ(o∣q)πref(o∣q)].

In practice, this KL term acts as a regularizer that penalizes the model for drifting too far from its reference, while still allowing meaningful improvement. You can think of it as a leash that keeps the policy stable during training. Now, the group-based rewards and the KL penalty are combined into the GRPO objective. For each query q, the model samples a group of responses {oi}i=1G from the old policy πθold. Each response contributes a token-level importance ratio wi,t(θ), which measures how much more (or less) likely the new policy is to generate the same token compared to the old one:

wi,t(θ)=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t).

The reward for each response is normalized within the group, giving the relative advantage Ai:

Ai=ri−mean(r1,r2,…,rG)std(r1,r2,…,rG).

With these elements, the GRPO objective becomes:

ℒGRPO(θ)=𝔼q,{oi}[1G∑i=1G1|oi|∑t=1|oi|min(wi,t(θ)Ai,clip(wi,t(θ),1−ε,1+ε)Ai)]−β𝔻KL(πθ‖πref).

This equation captures GRPO’s core intuition mathematically, Table 3-3 shows the conceptional overview of GRPO.

Table 3-3. GRPO Objective: what each part does

Term
What it means
Why it matters for agents

wi,t(θ)Ai

Encourages the model to increase the likelihood of higher-ranked responses.

Guides learning toward better reasoning paths within each group.

Clipping clip(wi,t(θ),1−ε,1+ε)

Prevents large updates when the new policy drifts too far from the old one.

Stabilizes learning by limiting off-policy jumps that could cause collapse.

−β𝔻KL(πθ|πref)

Penalizes the model for diverging too much from its reference policy.

Acts as a leash on exploration, keeping improvement balanced with stability.

This group-based normalization stabilizes learning by anchoring each update within its local context. As you learned in “Learning on a Leash: KL Divergence”, the KL divergence puts a leash on the policy, so that improvement doesn’t lead to instability via too much divergence from the reference.

In practice, GRPO operates at the token level. The clipping and KL penalty are applied incrementally as the model generates each token, allowing the training signal to remain both adaptive and efficient.

GRPO’s strength lies in its simplicity: no critic model, no handcrafted reward scaling, only relative comparison within each group. This design made GRPO one of the first RL methods to scale language models reliably, before GSPO refined it further.

GSPO

GSPO takes the principles of GRPO further by refining how the model interprets and optimizes its experience. While GRPO evaluates reasoning at the token level, GSPO moves one level higher and optimizes entire sequences as units. Figure 3-3 shows a high-level overview of the GSPO mechanics.

Figure 3-3. Unlike GRPO, which evaluates at the token level, GSPO computes rewards and KL divergence over complete sequences.

The motivation behind GSPO is straightforward: the reward belongs to the entire sequence, not to individual tokens. If the model receives feedback for the whole response, then the optimization should also act at that scale. By shifting the learning signal from tokens to sequences, GSPO aligns the unit of optimization with the unit of reward.

As before, the model starts from a query q and a set of responses {oi}i=1G sampled from the old policy πθold. But now, instead of token-level likelihood ratios, GSPO measures how much the new policy diverges from the old one across the entire response. This sequence-level importance ratio is defined as:

si(θ)=(πθ(oi∣q)πθold(oi∣q))1/|oi|=exp[1|oi|∑t=1|oi|logπθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)],

where |oi| denotes the number of tokens in the response. This length normalization keeps the ratio numerically stable, preventing longer outputs from producing disproportionately large updates.

The group-based advantage Ai remains the same as in GRPO and is computed as:

Ai=ri−mean(r1,r2,…,rG)std(r1,r2,…,rG).

With these definitions, the GSPO objective becomes:

𝒥GSPO(θ)=𝔼q,{oi}[1G∑i=1Gmin(si(θ)Ai,clip(si(θ),1−ε,1+ε)Ai)].

This objective follows the same logic as GRPO but applies the update at the sequence level instead of per token. By doing so, it eliminates the variance that can accumulate when token-level importance weights fluctuate independently within long responses. Implementations often include a sequence-level KL penalty analogously to GRPO, if omitted, early stopping and group normalization still stabilize training. Table 3-4 illustrates the core components of GSPO.

Table 3-4. GSPO Objective: what each part does

Term
What it means
Why it matters for agents

si(θ)Ai

Encourages the model to favor entire responses with higher relative rewards.

Aligns learning with full-sequence performance rather than token-by-token corrections.

Clipping clip(si(θ),1−ε,1+ε)

Restrains overly large updates when the new policy differs too much from the old one.

Reduces instability and prevents gradient explosions on long outputs.

Length normalization oi−1

Scales updates according to response length.

Keeps learning consistent across short and long completions.

This shift from token-level to sequence-level optimization has a profound effect. It removes the instability that can emerge when token-level corrections introduce noise, especially in long responses or very large models. By clipping and rewarding entire responses instead of individual tokens, GSPO makes learning smoother, more reliable, and better aligned with how rewards are assigned in practice.

It also simplifies reinforcement learning infrastructure. Because GSPO operates on sequence likelihoods, it can often reuse probabilities already computed during inference rather than recomputing token-level likelihoods during training. This makes it both computationally efficient and stable across large-scale systems. Table 3-5 compares both methods.

Table 3-5. GRPO vs. GSPO

Aspect
GRPO
GSPO

Learning signal

Learns from relative rankings of multiple responses to the same query.

Learns from sequence-level comparisons, weighting each full response by its overall likelihood.

Level of optimization

Token-level updates within each response.

Sequence-level updates across full responses.

Reward granularity

Same advantage shared across tokens, but importance ratios computed per token.

Advantage computed per sequence, importance ratios applied uniformly across all tokens.

Stability

Sensitive to noise accumulation over long outputs, may become unstable in very large or MoE models.

Significantly more stable, avoids token-level variance and prevents model collapse.

Clipping mechanism

Clips importance ratios per token to prevent large off-policy deviations.

Clips entire sequences, aligning reward and optimization scales.

Efficiency

Requires more frequent recomputation of token likelihoods during training.

Can reuse sequence-level likelihoods from inference, improving compute efficiency.

Infrastructure complexity

Needs value-model surrogates or routing replay for MoE stability.

Removes the need for extra stabilization strategies, simplifying large-scale RL training.

Practical effect

Enables reflection and learning from comparison but may struggle to scale.

Scales reliably to very large models and supports modern architectures like Qwen3 and MoE systems.

Together, GRPO and GSPO show how agents can learn not from explicit answers but from their own judgment. GRPO teaches the agent to compare outcomes and learn from relative success at the token level, while GSPO improves over GRPO by optimizing grouped policy sequences at the sequence level. This shifts evaluation from judging parts of an output to assessing the entire sequence under the policy. At this point, your agent has the essential components for self improvement. It can evaluate, compare, and refine. The next challenge is teaching it to apply these principles systematically, turning scattered experiences into structured trajectories that can be reused for continued growth.

Taking Off the Training Wheels: Teaching Agents How to Learn

By this point, your agent has seen a lot. Depending on your underlying language model, it may already have gathered experience, compared its own reasoning paths, and learned how to distinguish good decisions from poor ones. Yet general experience alone doesn’t make a system reliable for your specific tasks. What truly matters is how you can teach your agent a new skill or become better at an already learned one. That is what this section is about.

If you think of your agent as a learner, using RL to teach it a new skill is the moment the training wheels come off. Every wobble, correction, and forward push becomes part of its evolving policy. Yes, your agent might wobble, crash, and correct, but that’s how learning actually begins.

Through frameworks such as ART and RULER, your agent learns to refine itself through structured feedback loops. ART is an open-source framework designed to train agentic LLMs through structured rollouts centered on tool use and decision-making. It enhances an agent’s performance and reliability by learning from experience, building on GRPO. RULER complements this approach as a general-purpose reward function that uses an LLM-as-judge to rank agent trajectories by quality, providing consistent feedback without handcrafted metrics.

Together, ART and RULER help you guide your agents toward what makes a good output purely from the task description, without any expected outputs required. The result is not just an agent that performs tasks, but one that develops a deeper understanding of how to improve itself both in context and when collaborating with other agents.

Beyond Isolation: Why Absolute Scoring is Easier

Reward design used to be one of the hardest parts of RL. Custom reward functions often broke down as tasks changed or scaled. LLM-as-judge systems were meant to fix that, but early attempts struggled with inconsistent scores and noisy calibration. RULER solves this by shifting from absolute scoring to relative comparison. In addition, as you already know, GRPO and GSPO normalize scores within each group, so only the relative rankings matter, not the absolute values.

RULER generates several trajectories for the same task. Each trajectory records the agent’s reasoning, tool calls, and final outcome. Shared prefixes are removed, and only the unique parts are sent to an LLM-as-judge along with a simple rubric. The judge scores each trajectory between 0 and 1 based on goal completion and efficiency. Those scores are normalized within the group and used directly as rewards for a GRPO update. The judge can also explain its decisions, which helps you trace errors and cluster low-performing runs. Figure 3-4 shows how this process fits together.

Figure 3-4. RULER training loop: trajectories are ranked by an LLM-as-judge, normalized by GRPO, and used to update the policy iteratively.

Let me give you a concrete coding example to understand how simple RULER actually is. Imagine you want to evaluate short poems about cats. You’ll start by preparing the messages for the LLM, as in Example 3-1.

Example 3-1. Shared message setup

initial_messages = [
        {"role": "system", "content": """You are a poetic writer. Write short
                                      cat-themed poems that evoke emotion."""},
        {"role": "user", "content": """Write a poem about cats observing the
                                    sky."""}
    ]

The next step is to create a few sample trajectories that differ in quality. Example 3-2 defines a set of mocked trajectories that differ in quality, ranging from a well-written poem to an off-topic response. In a real RULER run, the agent would generate these outputs itself. Here, they’re predefined so you can clearly see how the scoring process works.

Example 3-2. Create trajectories with different quality levels

good_trajectory = art.Trajectory(
        messages_and_choices=[
            *initial_messages,
            Choice(
                finish_reason="stop",
                index=0,
                message=ChatCompletionMessage(
                    role="assistant",
                    content=(
                        "Upon the roof, their eyes like stars,\n"
                        "They watch the sky from silver jars.\n"
                        "Each whisker twitches, soft delight,\n"
                        "As moons reflect their borrowed light."
                    )
                )
            )
        ],
        reward=0.0
    )

    mediocre_trajectory = art.Trajectory(
        messages_and_choices=[
            *initial_messages,
            Choice(
                finish_reason="stop",
                index=0,
                message=ChatCompletionMessage(
                    role="assistant",
                    content=(
                        "Cats sit and look at stars at night,\n"
                        "They think the stars are shiny and bright."
                    )
                )
            )
        ],
        reward=0.0
    )

    off_topic_trajectory = art.Trajectory(
        messages_and_choices=[
            *initial_messages,
            Choice(
                finish_reason="stop",
                index=0,
                message=ChatCompletionMessage(
                    role="assistant",
                    content=(
                        "Dogs are great companions under the moon. "
                        "They bark at stars and wag their tails happily."
                    )
                )
            )
        ],
        reward=0.0
    )

After the trajectories are generated, Example 3-3 evaluates them in one pass. RULER scores and ranks the candidates so that the language model can act as a judge and select the strongest response.

Example 3-3. Score with RULER

group = art.TrajectoryGroup([good_trajectory, mediocre_trajectory,
                            off_topic_trajectory])
judged_group = await ruler_score_group(group, "openai/o3", debug=True)

With the group judged, Example 3-4 simply retrieves the rewards and prints a sorted leaderboard. This makes it easy to see how each poem performed under RULER’s evaluation.

Example 3-4. Show trajectory ranking

if judged_group:
        sorted_trajectories = sorted(judged_group.trajectories,
                                key=lambda t: t.reward, reverse=True)
        for rank, traj in enumerate(sorted_trajectories, 1):
            messages = traj.messages()
            print(f"Rank {rank}: Score {traj.reward:.3f}")
            print(f"  Response: {messages[-1]['content'][:80]}...\n")

A typical output looks like this:

[RULER] Pretty-printed LLM choice JSON:
{
    'scores': [
        {
            'trajectory_id': '1',
            'explanation': 'Poetic, cat-focused, night-sky theme fulfilled with
                            vivid imagery; brief and compliant.',
            'score': 0.9
        },
        {
            'trajectory_id': '2',
            'explanation': 'Meets topic but very simplistic and minimally
                            evocative; partial fulfillment.',
            'score': 0.6
        },
        {
            'trajectory_id': '3',
            'explanation': 'Ignores cat theme, focuses on dogs; fails the
                            main instruction.',
            'score': 0.05
        }
    ]
}

Here is why this works:

Comparing several candidates is easier for a language model than grading one in isolation. Differences become clear when options are viewed side by side.

GRPO only requires rewards that are consistent within a group. It does not need global calibration, which keeps learning stable and scalable.

In practice, this setup removes the need for labeled data or hand built rules. It also accelerates convergence because partial improvements receive proportional credit. The result is a reward signal that is self-correcting, interpretable, and efficient.

From this example you can clearly see how RULER helps grade textual or other open-ended tasks such as explanations, reasoning chains, summaries, essays, customer interactions, and scientific analysis. Quality in these settings depends on clarity, completeness, depth of reasoning, tone, and creativity — none of which can be reduced to a single formula. RULER uses a relative, model based comparison across multiple trajectories so the LLM-judge can rank subtle differences such as more coherent reasoning flow, fewer hallucinations, and clearer logic.

For deterministic problems such as the countdown tasks, where the objective is to reach a specific target using a fixed set of numbers and arithmetic operations, the reward can be defined programmatically. A rule based reward works well here because correctness is binary and can be verified automatically.

However, when you extend the system to include explanatory or reflective reasoning, where the agent must describe how it arrived at a valid expression or justify each operation, the evaluation criteria become more nuanced. In these cases, a hybrid reward can be introduced. This mechanism combines the precision of a deterministic correctness check with the qualitative judgment of RULER, so that only valid mathematical solutions are scored while their reasoning quality, structure, and clarity still influence the final reward.

This concept is not limited to numerical reasoning. The same hybrid reward can be applied to code agents, where functional correctness can be checked programmatically but style, readability, or documentation quality may require language model evaluation. It can also be extended to text generation tasks by mapping objective measures, such as factual accuracy or constraint satisfaction, to scalar correctness scores and combining them with RULER style assessments.

In essence, the hybrid reward offers an easy way to merge symbolic verification with language based evaluation, creating reward functions that balance precision and interpretability. The next section demonstrates how this can be implemented in practice using ART and GRPO to train an agent to solve the countdown tasks.

The ART of Learning from Experience

Now that you know how RULER works, you’re ready to take your agent to the next step and actually produce trajectories for a task using ART to help it improve. In the following example, you’ll teach your agent to solve the mathematical countdown tasks with a hybrid reward. The setup combines deterministic rewards for numerical correctness with RULER-based scoring for reasoning quality, allowing your agent to learn both precision and clarity.

Note

Note that I focus on the key code snippets here and omit helper functions or minor sections to highlight the core logic. The full runnable code is available in the book’s repository.

You begin by preparing the dataset (omitted) and a trainable model (Example 3-5) with a local backend (Example 3-6).

Example 3-5. Set up model and model configurations

model = art.TrainableModel(
    name="countdown-agent-001",
    project="countdown-agent",
    base_model="Qwen/Qwen2.5-7B-Instruct",
)

Example 3-6. Set up and register backend

backend = LocalBackend(in_process=True, path="./.art")
await model.register(backend)

Next you need to define the task interface: simple tools to read the current context, search for a valid expression, and return the final answer so ART can score the trajectory. I omitted this part here too, but in the notebook you’ll find this under the section: exact search tool with fractions.

Before you can run rollouts, you should add a programmatic judge for binary correctness. This gives you the hard reward that enforces exact solutions. Example 3-7 is an example of such a reward.

Example 3-7. Add deterministic judge for correctness

def judge_countdown_expression(
    target: int,
    allowed_nums: List[int],
    expr: str,
    enforce_integer_intermediates: bool = False,
) -> Tuple[float, Optional[float], Optional[str]]:

    expr = expr.strip()
    if not expr:
        return 0.0, None, "empty expression"
    if re.search(r"[^0-9\+\-\*\/\(\)\.\s]", expr):
        return 0.0, None, "invalid characters"

    used = numbers_used_in_expr(expr)

    def counts(xs):
        d = {}
        for v in xs:
            d[v] = d.get(v, 0) + 1
        return d

    allowed_counts = counts([int(x) for x in allowed_nums])
    used_counts    = counts(used)

    for v, c in used_counts.items():
        if v not in allowed_counts or c > allowed_counts[v]:
            return 0.0, None, f"illegal number usage: {v}"
    try:
        if enforce_integer_intermediates:
            val_frac = _eval_ast_fraction(ast.parse(expr, mode="eval"))
            if val_frac == Fraction(target, 1):
                return 1.0, float(val_frac), None
            return 0.0, float(val_frac), "wrong value"
        else:
            val = safe_eval_expr(expr)
            if abs(val - target) < 1e-9:
                return 1.0, val, None
            return 0.0, val, "wrong value"
    except ZeroDivisionError:
        return 0.0, None, "division by zero"
    except Exception as e:
        return 0.0, None, f"eval error: {e}"

Validate characters to be safe.

Check number usage.

In Example 3-8 you create scenarios from the dataset so each rollout has a clear prompt and identifiers.

Example 3-8. Build scenarios

class Scenario(BaseModel):
    id: str
    target: int
    nums: List[int]
    question: str

def mk_scenario(row, idx: int) -> Scenario:
    tgt  = int(row["target"])
    nums = [int(x) for x in row["nums"]]
    q = """Using only the numbers each at most once, write a valid
            arithmetic expression with +, -, *, or / that evaluates
            exactly to the target. Return only the expression string."""

    return Scenario(id=f"cd_{idx}", target=tgt, nums=nums, question=q)

train_scenarios = [mk_scenario(train_ds[i], i) for i in range(len(train_ds))]
test_scenarios  = [mk_scenario(test_ds[i], i)  for i in range(len(test_ds))]

With data, tools, and judge in place, you can define your rollout. Example 3-9 runs one task end to end, allows tool use, records the final answer, and writes rewards and metrics on the trajectory for ART.

Example 3-9. Create rollout to capture trajectories

class ProjectTrajectory(art.Trajectory):
    final_answer: Optional[FinalAnswer] = None

class CountdownScenario(BaseModel):
    step: int
    scenario: Scenario

@weave.op
async def rollout(model: art.Model, cd: CountdownScenario) -> ProjectTrajectory:
    scn = cd.scenario

    _nonlocal_final["value"] = None
    _current_scenario["target"] = scn.target
    _current_scenario["nums"]   = scn.nums
    _current_scenario["id"]     = scn.id

    nums_str = ",".join(map(str, scn.nums))
    traj = ProjectTrajectory(
        reward=0.0,
        messages_and_choices=[],
        metadata={"scenario_id": scn.id,
                 "target": float(scn.target), "nums": nums_str},
    )
    if traj.metrics is None:
        traj.metrics = {}

    system_prompt = dedent(f"""
        You are a math agent for Countdown tasks.

        Goal: produce one expression that evaluates exactly to {scn.target}
        using only the numbers {scn.nums} each at most once.
        Operators: +, -, *, /. Parentheses are allowed.

        Tools:
        - read_countdown_context()
        - countdown_search_tool(nums=[...], target=...,
          require_integer_intermediates=True)
        - return_final_answer_tool(answer=EXPR, reference_ids=[...])

        Process:
        1) Call read_countdown_context to confirm inputs.
        2) Call countdown_search_tool. If it returns a non empty expression,
           pass it verbatim to return_final_answer_tool.
        3) If it returns empty, reason and still obey number usage.
        The final tool must receive ONLY the expression string.
    """)

    tools = [read_countdown_context, countdown_search_tool, return_final_answer_tool]
    chat  = init_chat_model(model.name, temperature=0.6, top_p=0.9, max_tokens=256)
    agent = create_react_agent(chat, tools)

    try:
        config = {"configurable": {"thread_id": str(uuid.uuid4()),
                                   "seed": random.randint(0, 1_000_000)},
                                    "recursion_limit": MAX_TURNS}
        await agent.ainvoke(
            {"messages": [SystemMessage(content=system_prompt),
            HumanMessage(content=scn.question)]},
            config=config,
        )

        if _nonlocal_final["value"]:
            traj.final_answer = _nonlocal_final["value"]
            reward, val, reason = judge_countdown_expression(
                scn.target, scn.nums, traj.final_answer.answer,
                enforce_integer_intermediates=True
            )
            traj.reward = float(reward)
            traj.metrics["correct"] = float(reward)
            traj.metrics["value"] = float(val) if isinstance(val,
                                    (int, float)) else float("nan")
            traj.metadata["last_reason"] = str(reason) if reason else ""

            traj.metadata["rubric"] = (
                "Score higher if the expression (1) hits the target, "
                "(2) uses each provided number at most once, "
                "(3) uses fewer operations, "
                "(4) avoids redundant parentheses and trivial +/-0 or *1 tricks, "
                "(5) avoids division when +, -, * suffice."
            )
    return traj

Extend the trajectory with a typed slot for the agent’s final tool output.

Shared scratchpad objects that your tools read/write.

ReAct-style agent wired with your 3 tools.

Unique thread/seed per rollout → reproducibility and isolation.

Safety against infinite tool loops.

Persist the tool-returned answer on the trajectory.

Hard reward from deterministic judge.

Rubric used by RULER’s LLM-as-judge.

Example 3-10 shows you how to apply RULER within each group.

Example 3-10. Use RULER to score each group

ruler_model_id = "openai/gpt-4.1"
ruler_groups = []
for group in judged:
    try:
        rg = await ruler_score_group(group, ruler_model_id, debug=True)
    except Exception:
        rg = None
    ruler_groups.append(rg if rg is not None else group)

Ask the judge to score/rank within each group.

Fall back to the unjudged group if judging fails.

Example 3-11 combines both signals to shape learning beyond correctness. Only correct solutions receive credit, then RULER ranks them to refine the reward.

Example 3-11. Combining correctness with RULER using a hybrid reward

alpha, beta = 0.7, 0.3

def _combine(groups, alpha: float = 1.0, beta: float = 1.0):
    for g in groups:
        raw = []
        for t in g.trajectories:
            try:
                raw.append(float(t.reward))
            except Exception:
                raw.append(0.0)

        rmin = min(raw) if raw else 0.0
        rmax = max(raw) if raw else 0.0
        span = (rmax - rmin) if (rmax > rmin) else 1.0

        for t in g.trajectories:
            if t.metrics is None:
                t.metrics = {}
            if t.metadata is None:
                t.metadata = {}

            gate = float(t.metrics.get("correct", 0.0) or 0.0)
            gate = 1.0 if gate >= 0.5 else 0.0

            try:
                ruler_raw = float(t.reward)
            except Exception:
                ruler_raw = 0.0
            ruler_norm = (ruler_raw - rmin) / span
            ruler_norm = max(0.0, min(1.0,
                         ruler_norm)) if ruler_norm == ruler_norm else 0.0

            final_reward = gate * (alpha + beta * ruler_norm)
            final_reward = max(0.0, min(1.0, final_reward))

            t.metadata["ruler_score_raw"] = float(ruler_raw)
            t.metrics["ruler_norm"] = float(ruler_norm)
            t.metrics["final_reward"] = float(final_reward)
            t.reward = float(final_reward)
    return groups

hybrid_groups = _combine(ruler_groups, alpha=alpha, beta=beta)

Weight of correctness vs. quality; adjust for your own task.

Use judged rewards as the quality signal.

Robust group-wise normalization (avoids div-by-zero).

Binary gate from deterministic judge ensures precision first.

Clip and NaN-guard.

Simple, interpretable shaping formula.

Keep both raw and normalized scores.

Overwrite trajectory reward with the hybrid reward used for training.

Example 3-12 sets up and runs the training loop. ART iterates over scenarios, collects trajectory groups, applies the hybrid reward, and updates the policy.

Example 3-12. Run training loop

training_config = {
    "groups_per_step": 4,
    "num_epochs": 1,
    "rollouts_per_group": 2,
    "learning_rate": 1e-5,
    "max_steps": 3,
}

train_slice = train_scenarios[:64]

training_iterator = iterate_dataset(
    train_slice,
    groups_per_step=training_config["groups_per_step"],
    num_epochs=training_config["num_epochs"],
    initial_step=await model.get_step(),
)

for batch in training_iterator:
    print(f"Training step {batch.step}, epoch {batch.epoch}")
    groups = []
    for scn in batch.items:
        group = art.TrajectoryGroup(
            [wrap_rollout(model, rollout)(model,
            CountdownScenario(step=batch.step, scenario=scn))
              for _ in range(training_config["rollouts_per_group"]) ]
        )
        groups.append(group)

    finished = await art.gather_trajectory_groups(
        groups, pbar_desc="gather",
        max_exceptions=training_config["rollouts_per_group"] * len(batch.items),
    )

await model.train(
        hybrid_groups,
        config=art.TrainConfig(learning_rate=training_config["learning_rate"]),
        _config={"logprob_calculation_chunk_size": 8},
    )

How many scenarios per optimizer step.

How many rollouts per scenario (the comparison set).

Keep small while debugging to avoid runaway training.

Small slice for a fast first pass.

Resume-safe: read current global step.

Async gather lets slow rollouts finish without blocking the loop.

Apply GRPO update using the hybrid rewards.

To test the trained agent, you can run Example 3-13 on a few held out items.

Example 3-13. Test the trained agent

for scn in test_scenarios[:10]:
    res = await wrap_rollout(model, rollout)(model,
                CountdownScenario(step=0, scenario=scn))
    expr = res.final_answer.answer if res.final_answer else None
    reward, val, reason = judge_countdown_expression(scn.target, scn.nums,
                          expr or "", enforce_integer_intermediates=True)
    status = "CORRECT" if reward == 1.0 else f"WRONG ({reason})"
    print(f"nums={scn.nums} target={scn.target} -> {expr} => {status}")

Reuse the trained policy + identical tool loop for evaluation.

Pull the final tool answer if present.

Same hard judge as in training ensures apples-to-apples evaluation.

This results in the following print-out:

nums=[22, 89, 16] target=51 -> None => WRONG (empty expression)
nums=[85, 45, 75, 10] target=25 -> ((75-10)-(85-45)) => CORRECT
nums=[68, 96, 50, 3] target=75 -> ((50-3)-(68-96)) => CORRECT
nums=[49, 94, 73, 40] target=12 -> ((40-73)-(49-94)) => CORRECT
nums=[40, 79, 73] target=34 -> (73+(40-79)) => CORRECT
nums=[21, 1, 33, 2] target=40 -> (33+(21/(1+2))) => CORRECT
nums=[29, 1, 4, 46] target=66 -> ((4*(29-1))-46) => CORRECT
nums=[19, 97, 98] target=18 -> ((19+97)-98) => CORRECT
nums=[55, 76, 1] target=22 -> (1-(55-76)) => CORRECT
nums=[31, 4, 16, 36] target=20 -> None => WRONG (empty expression)

This clearly shows that the agent successfully learned how to solve the countdown tasks, even though I just used a small split of the dataset.

Test Time Compute: Balancing Size with Thought

There is a limit to how far you can go by scaling model size alone. Training ever-larger language models demands enormous resources and quickly becomes unsustainable. Yet, reasoning ability doesn’t come purely from size. Recent research shows that even smaller models can perform exceptionally well if they are given more time to think at test time.

Instead of adding more parameters, you can add more computation during inference. You can let the model reason longer, explore multiple ideas, and refine its answers before committing to a final one. This approach changes how you think about scale: not as more neurons, but as more deliberate reasoning. In short, so far you’ve taught your agent to learn, but now you teach it to think. This is where your learning systems turn into thinking systems.

This can mean sampling multiple reasoning paths, reflecting on intermediate steps, or revising answers based on feedback. The result is an agent that learns to spend compute where it matters. It can simulate alternative outcomes, evaluate them, and refine its reasoning dynamically. This idea of scaling thinking time rather than model size is one of the most promising paths toward more capable and efficient agents.

To support this shift, preference and reward models take on a new role. They no longer serve only during training but become part of the inference process itself. Instead of scoring a single final answer, they guide reasoning as it unfolds. Each step or branch in the reasoning process can be evaluated and compared, much like a human reviewing a line of argument or debugging code. This feedback loop allows smaller models to match or even outperform larger ones on complex reasoning tasks, especially when resources are limited or when models must run efficiently on local devices.

To scale test-time compute, you can use tree search algorithms which helps your agents allocate reasoning effort adaptively. Among these, Monte Carlo Tree Search (MCTS) is one of the most powerful. MCTS allows an LLM to explore a structured space of possible answers and gradually converge on the best one. It balances curiosity and focus, searching just widely enough to discover new ideas and deeply enough to refine them.

From Sampling to Search: Balancing Exploration and Exploitation

Before looking more closely at adaptive algorithms, it helps to start with two simpler strategies. There are two methods that set the perfect stage for you to understand the tradeoff between exploring more answers or going deeper into a specific answer: repeated sampling and sequential sampling, respectively. In repeated sampling, shown in Figure 3-5, the model draws multiple independent answers for the same prompt. This approach only explores breadth, producing variety but no refinement. In contrast, sequential refinement focuses on depth. It starts from one initial answer and repeatedly improves it. Both methods are useful, but each only covers half of what real reasoning requires.

Figure 3-5. Flat sampling strategies: repeated sampling versus sequential refinement

MCTS builds on this concept by building a search tree through four recurring steps. It begins with selection, where the algorithm traverses the tree from the root, choosing branches that balance exploration and exploitation according to a scoring rule such as the Upper Confidence Bound for Trees (UCT). This balance, known in RL as the exploration–exploitation tradeoff, represents the tension between gathering new information to discover potentially better solutions (exploration) and using the information already collected to maximize performance (exploitation). With UCT the value for expansion is selected by the next iteration. The UCT of a child state s is calculated as follows:

UCT(s)=V(s)+clnN(p)N(s)

Here, V(s) is the estimate of your node s, N(s) is the visit count of your node s, N(p) counts the visits of your parent node, and c stands for the exploration weight you can set. Example 3-14 shows how to implement this in Python.

Example 3-14. UCT implementation

def upper_confidence_bound(self, exploration_weight=1.0):
    if self.parent is None:
        raise ValueError("Cannot obtain UCT from root node")
    if self.visits == 0:
        return float("inf")
    parent_visits = max(1, self.parent.visits)
    average_reward = self.value / self.visits
    exploration_term = math.sqrt(math.log(parent_visits) / self.visits)
    return average_reward + exploration_weight * exploration_term

Once a promising branch is found, it moves to expansion, where new child nodes are created to represent new candidate solutions. From there, the model performs a simulation, also called a rollout, where it generates and evaluates a full solution from that node. Finally, backpropagation updates all parent nodes with the outcome, allowing the system to refine its understanding of which directions are most promising. Figure 3-6 illustrates this loop of planning, evaluation, and reflection.

Figure 3-6. Monte Carlo Tree Search applied to LLM reasoning

At test time, the model’s reasoning budget is spent running this loop: selecting, expanding, simulating, and backpropagating. The LLM generates candidate answers, evaluates them, and uses those evaluations to guide the next move. This process lets the model explore more possibilities than simple one-pass decoding, producing solutions that are both higher in quality and more explainable.

Adaptive Branching Monte Carlo Tree Search (AB-MCTS)

Adaptive Branching Monte Carlo Tree Search (AB-MCTS) ⁴ extends standard MCTS by dynamically deciding whether to explore new branches or to refine existing ones. This adaptive decision process represents the classic exploration–exploitation trade-off in RL.

This flexibility is essential for your agents, because even a single prompt can generate an infinite variety of responses. A fixed branching factor, as used in standard MCTS, limits this potential. AB-MCTS overcomes that by allowing the tree to grow as needed. Figure Figure 3-7 shows how it differs from traditional MCTS.

Figure 3-7. Comparison of MCTS with AB-MCTS

AB-MCTS introduces a GEN node under every tree node, when selected, this node signals that the model should generate a new candidate response, effectively creating a new branch. Whether to generate a new path or refine an existing one is guided by Thompson sampling, a Bayesian strategy that balances exploration and exploitation based on uncertainty.

There are two main variants. AB-MCTS-M (mixed model) uses a node-specific Bayesian model to estimate scores for both GEN and existing nodes. It draws on observed scores from subtrees to build posterior predictive distributions that guide search decisions. This allows the model to infer likely rewards even for paths that have not yet been explored, enabling principled decision-making with limited data. Figure 3-8 illustrates this mechanism.

Figure 3-8. Tree structure and posterior predictive distributions for AB-MCTS-M

The second variant, AB-MCTS-A (node aggregation), is simpler and closer to standard MCTS. It introduces a CONT node under each answer node to represent refinement. Each node can either generate new candidates (via GEN) or continue improving an existing one (via CONT). This setup limits the computational cost while keeping the search flexible. Depending on how rewards are distributed, AB-MCTS-A can use Gaussian priors or Beta priors to normalize scores between 0 and 1. Figure 3-9 shows how such a tree can evolve.

Figure 3-9. AB-MCTS-A tree structure with CONT and GEN nodes

Both AB-MCTS variants rely on Bayesian updating to guide the search. The LLM generates, evaluates, and refines responses iteratively, distributing its computational effort where it brings the greatest gain. This allows the model to reason adaptively: exploring when uncertain, exploiting when confident.

To illustrate, the next example uses the open source TreeQuest library to refine Python code for the Fibonacci sequence. It shows how tree search balances breadth and depth to improve an LLM agent’s answer step by step.

Example 3-15 creates the first candidate and scores it. It drafts a solution from scratch, calls the judge, and returns a State that MCTS can treat as a root child.

Example 3-15. Generate initial answer

def initial_generation() -> State:
    prompt = "Q: Write code in Python for the Fibonacci sequence. \nA:"
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
    )
    answer = response.choices[0].message.content.strip()
    score = evaluate_answer(answer)
    return State(llm_answer=answer, score=score)

Use a model that supports JSON style response formatting for judging later.

A modest temperature supports variation for future branches.

Self-evaluates the answer to seed the root children with scores.

Next, Example 3-16 performs a targeted improvement pass on an existing answer and re-scores it.

Example 3-16. Refine answer

def refine_answer(llm_answer: str, score: float) -> State:
    prompt = f"""The current answer is:\n\n{llm_answer}\n\nPlease improve this
            answer to be more informative, accurate, and clear."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
    )
    refined = response.choices[0].message.content.strip()
    score = evaluate_answer(refined)
    return State(llm_answer=refined, score=score)

Again, use a model that supports JSON style response

Always re-score the refined answer to update the search tree.

Example 3-17 asks an LLM judge to return a JSON object with a single score key. Uses the OpenAI client with structured parsing into ScoreResponse. On any error it falls back to a neutral score to keep the search momentum.

Example 3-17. LLM judge with structured parsing

def evaluate_answer(answer: str) -> float:
    prompt = (
        f"Evaluate the quality of this answer on a scale from 0 to 1.\n"
        f"Return a JSON object like {{\"score\": 0.92}}.\n\n"
        f"Answer:\n{answer}"
    )

    try:
        completion = client.chat.completions.parse(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
            response_format=ScoreResponse,
        )
        return completion.choices[0].message.parsed.score
    except Exception as e:
        print(f"[Evaluation error] {e}")
        return 0.5

Structured parse into ScoreResponse prevents brittle string parsing.

Neutral fallback keeps the search progressing when judging fails.

If there is no parent_state, Example 3-18 creates the initial candidate, or otherwise, refines the parent answer.

Example 3-18. Generation entry point for TreeQuest

def generate(parent_state: State | None) -> tuple[State, float]:
    if parent_state is None:
        return initial_generation(), initial_generation().score
    return refine_answer(parent_state.llm_answer,
                        parent_state.score), parent_state.score

First call produces the initial root child that AB-MCTS can expand.

Refinement corresponds to a CONT choice under AB-MCTS-A or a selected existing node under AB-MCTS-M. The second return value can be used by the library to compute deltas or edge utilities.

Example 3-19 implements and runs the AB-MCTS loop.

Example 3-19. TreeQuest loop

algo = tq.ABMCTSA()
search_tree = algo.init_tree()

for i in range(5):
    search_tree = algo.step(search_tree, {'LLM-Refine': generate})
    if (i + 1) % 5 == 0:
        best, _ = tq.top_k(search_tree, algo, k=1)[0]
        print(f"[Step {i+1}] Best so far: {best.llm_answer} (score={best.score:.2f})")

best_state, _ = tq.top_k(search_tree, algo, k=1)[0]
print(f"\n Final Best Answer: {best_state.llm_answer} (score={best_state.score:.2f})")

Run a small number of steps (five here). Increase gradually in real use.

Inspect best-so-far periodically.

Final selection.

Part of the print-out looks as follows for me:

[Step 5] Best so far: Certainly! Let's refine the explanation and the code
to make it more informative, accurate, and clear.

### Improved Code
Here's the revised version of the code with additional comments for clarity:

```python
def fibonacci(n):
    """
    Generate the first n terms of the Fibonacci sequence.

    Parameters:
    n (int): The number of terms to generate.

    Returns:
    list: A list containing the first n terms of the Fibonacci sequence.
    """
    # Initialize the sequence list and the first two Fibonacci numbers
    sequence = []
    a, b = 0, 1

    # Generate Fibonacci numbers up to n terms
    while len(sequence) < n:
        sequence.append(a)  # Append the current number to the sequence
        a, b = b, a + b # Update a + b to the next two numbers in the sequence
    return sequence

As you can see, I chose five improvement steps, and this is exactly what you see reflected here. Tree search shows what it means to scale reasoning instead of model size. By letting your agents explore, evaluate, and refine, you trade raw capacity for structured and guided thought. With algorithms like AB-MCTS, test-time compute reasoning becomes an active process, not a single pass.

Enabling General-Purpose Agentic Programs Through Post-Training and Search-Based Agents

As you’ve seen, RL gives your agents the ability to improve beyond static prompting, closing the loop between reasoning, action, and feedback. Frameworks like ART and RULER make this process practical: they let your agents learn from structured trajectories and relative feedback instead of fixed labels or handcrafted rewards. But as your agents grow more complex, combining multiple reasoning roles, branching workflows, or search-based strategies, training these systems becomes your next challenge. This is where rLLM enters the picture as a valuable tool in your agent-improvement toolbox. In this section, training becomes orchestration. You teach multiple LLM-based agents how to divide work, judge outcomes, perform conditional branching and parallel execution, and coordinate tree search as a team so they can improve together.

rLLM is an open-source framework for post-training language agents through RL. It allows you to design custom agents and environments, train them with RL algorithms such as GRPO, and deploy them for production-scale workloads. Conceptually, it builds upon the same principles you already learned with RULER and ART, but extends them to a higher level of abstraction. Instead of training a single policy, rLLM enables entire agentic workflows to learn collectively.

At its core, rLLM introduces a workflow engine and a trainer that together generalize the RL process. The workflow engine executes complex reasoning or multi-agent programs such as solver–judge systems and planner–executor loops. It also enables you to use tree-search-based agents with MCTS, while collecting trajectories from each component. The trainer component aggregates these trajectories, computes relative advantages, and applies updates with KL regularization to ensure stability in your system. rLLM helps you to transforms your individual agents into a trainable and collaborative system. Table 3-6 compares each of the three frameworks, to make their distinct yet complementary role within the RL stack for agentic systems clearer.

Table 3-6. Frameworks for RL-based agent training

Framework
Core focus
Learning mechanism
Scope

RULER

Relative evaluation via LLM-as-judge

Converts qualitative reasoning into normalized quantitative feedback

Reward modeling and judgment

ART

Reinforcement learning for tool-using agents

Structured rollouts and GRPO-based policy updates

Behavioral optimization and reliability

rLLM

Reinforcement learning for agentic workflows

Unified training across reasoning, judgment, and search processes

System-level learning and coordination

rLLM integrates and extends the mechanisms you’ve already seen in GRPO. It also works seamlessly with methods like MCTS. Each node in a search tree becomes a reasoning step that can be optimized through reinforcement learning. The phases of selection, expansion, simulation, and backpropagation (Figure 3-6) all generate trajectories with measurable outcomes. Within rLLM, these trajectories can be grouped, ranked, and refined so your agents gradually learn how to search more effectively instead of relying on fixed heuristics.

rLLM helps you move from theoretical reinforcement learning setups to practical reasoning systems. The same control flow you use for inference, whether it’s a planner and executor working together, a solver and judge comparing results, or a search tree exploring different solutions, can now serve as your training loop. This bridges the gap between design and deployment, because the learning process naturally adapts to the way your agents already reason.

rLLM allows you to train any agentic system, no matter how complex it is. You simply define a workflow by inheriting from the Workflow base class, implementing your logic within a run() method, and returning an Episode object that captures all trajectories and rewards. The framework then orchestrates execution, retry logic, trajectory collection, and PPO/GRPO optimization, so you can focus entirely on your agent’s reasoning structure rather than on the surrounding infrastructure to train it.

To illustrate this idea, the book’s repository includes a notebook demonstrating a solver–judge workflow, a setup where solver agent propose answers and a judge agent evaluates them to select the best one. Each solver’s reward depends on its solution’s correctness, while the judge’s reward reflects whether it chose the correct solution. By bundling all trajectories into a single Episode, both solver and judge can be trained jointly, allowing the system to evolve as one coordinated reasoning process, this process can also be applied to improve the coordination between MAS.

In the example notebook, this structure is applied to the 24-game, a classic mathematical reasoning challenge. The task provides four numbers, for example, 3, 3, 4, and 5, and asks the agent to combine them using arithmetic operations (+, −, ×, ÷) to reach exactly 24. The solver agents attempt different expressions, and the judge evaluates them. Over time, RL helps both components refine their reasoning strategies, illustrating how rLLM’s workflow design enables end-to-end training across reasoning, judgment, and evaluation.

This way of training marks a clear step forward. Rather than fine-tuning models in isolation and combining them afterward, you can teach the entire reasoning architecture to grow as a single system. Feedback travels through every part of it, connecting judgment, reasoning, and search into one evolving system. RULER and ART help your agents evaluate and improve their own behavior, while rLLM gives your entire system the ability to learn and reason as a unified, adaptive intelligence.

Conclusion

The core lesson of this chapter is that your agent’s capability comes from learning dynamics and inference strategy, not model size. Rewards connect actions to consequences, trajectories preserve reasoning, and rollouts turn experience into structure. With KL as the safety rope, GRPO and GSPO transform relative comparisons into stable policy updates, so agents improve their process, not just their answers.

You then learned how to teach agents efficiently. RULER provides consistent, rubric driven judgment without handcrafted metrics, and ART turns judged trajectories into updates that make tool use more reliable. Hybrid rewards combine hard programmatic checks with qualitative assessments of reasoning, producing a training loop that is interpretable, data efficient, and aligned with how your agents actually operate.

Reasoning does not end at training time. Test time compute changes the game: by allocating more thinking during inference, smaller models can improve by exploring and refining alternative solutions. MCTS organizes that thinking, and AB-MCTS adapts breadth and depth with Bayesian updates, so agents explore when uncertain and refine when confident. Search becomes a first class part of reasoning, not an afterthought.

Finally, you moved from single agents to system level learning. rLLM generalizes the pattern by treating workflows themselves as trainable programs. Planner and executor, solver and judge, and tree search can all produce trajectories, receive rewards, and improve together as a coordinated system. This closes the gap between design and deployment: the same control flow that runs your agents in production becomes the loop that makes them better.

Taken together, these components demystify the magic behind agency: relative evaluation for signal, KL for stability, ART for trajectory centric training, AB-MCTS for adaptive inference, and rLLM to make the whole workflow learn. In the next chapter, you will learn how to select an LLM and understand the tradeoffs that shape your agents’ performance.

¹ This quote is one of the three laws proposed by the British science fiction writer Arthur C. Clarke.

² Zhihong Shao et al. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.”, (2024).

³ Chujie Zheng et al. “Group Sequence Policy Optimization.”, (2025).

⁴ Yuichi Inoue et al. (2025). Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search. https://arxiv.org/abs/2503.04412

Chapter 4. Models Behind the Agents: Capabilities and Optimization

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at mcronin@oreilly.com.

By now you have already done the hard work to understand how to design AI agents and make them better. This is the right moment for me to ask you to step back, look behind the scenes, and think more deeply about the models that make them intelligent. You might ask yourself why I didn’t introduce the models at the beginning. There is a reason. Let me tell you.

For this chapter it’s crucial to understand what you can do with your AI agents, because it completes the narrative arc of how I want you to think about them. I want you to think about your agents as if you were hiring team members. To do that well, you need the bigger picture: how your team interacts, how you structure it, and how you coach it to improve and collaborate. All of this was covered in the previous chapters.

This is my own mental model for designing AI agents. I think of agents as dynamic, collaborative systems rather than isolated components. Every model size and model family has a distinct skill profile. If you build your multi-agent system with that in mind, you’ll not only get better results, you’ll also save money. Let’s think of it a different way. You have learned how to manage your team; now it’s time to learn how to hire team members. In this chapter you learn how you decide who to onboard to the team, what skills they bring, and why that matters for performance, improvement, and cost.

Just as in human teams, each member has different strengths, weaknesses, and ways of working, and their skills complement one another. Same holds true with AI Agents, when you combine them thoughtfully, they achieve much more together than any single model could on its own. Once you understand their strengths and how they can collaborate, you can assign the right tasks and unlock the full potential of your system.

However, it’s important to know that this chapter isn’t meant as an introduction to the transformer architectures or to parameter-efficient fine-tuning (PEFT) methods. I assume you’ve seen those before. What I want to do here is refresh that knowledge and line it up with the agentic mindset, so it becomes obvious why model choice, context strategy, and lightweight specialization (LoRA, adapters, quantized bases) matter in the context of orchestrating multiple agents.

To understand this better, you’ll examine different architectures under this light, compare open weight and closed API models. In addition, you’ll map context window tradeoffs to deployment strategies, and understand when specialization through fine-tuning or adapters pays off. By the end, you’ll know how to staff your team, assign the right tasks, and optimize for both performance and spend. Let’s meet the candidates, understand their strengths, and hire well.

The Big Picture: Choosing the Right Architecture

Modern transformer variants are designed to handle particular workloads and deployment needs. The right choice depends on your objective, how information flows through the system, and the constraints you face in serving or scaling your models. Each architecture (encoder-only, decoder-only, encoder-decoder, or even MoE) has a unique role to play. Selecting the right mix means aligning each model’s strengths with your goals, computational budget, and serving constraints to achieve the best balance between capability and efficiency. Table 4-1 gives you a high-level understanding of each architecture and capability.

Table 4-1. Transformer Architecture Types

Type
Primary capability
Typical use cases

Encoder-only

Representation learning. Produces contextual embeddings for input tokens.

Text classification. Semantic search. Named entity recognition (NER).

Decoder-only

Autoregressive generation. Learns to predict the next token given previous context.

Text generation. Code synthesis. Conversational agents and chat systems.

Encoder–decoder

Sequence-to-sequence mapping between input and output representations.

Translation. Summarization. Question answering over provided context.

Embedding models

Produce dense or sparse vector representations for efficient retrieval and similarity comparison.

Retrieval-augmented generation (RAG). Semantic similarity search. Clustering.

Mixture of Experts (MoE)

Sparse expert routing for scalable compute and specialization across tasks.

Efficient large-scale generation. Multitask learning. Adaptive model scaling.

Let’s take a deeper look into decoder-only, encoder-only and MoE models, since these are the backbones of modern AI systems.

Decoder-Only Models

Decoder-only models are what you usually think of when you hear “LLM”. They form the backbone of systems like GPT or Claude and follow a simple but powerful idea: generate one token at a time, using everything that came before as context.

You already know the structure: no encoder, no cross-attention, just a stack of transformer blocks predicting the next token again and again. This autoregressive (auto = self, regressive = based on prior values) setup means the model learns to look only backward in the sequence. Each new token depends on its own previous outputs, creating a feedback loop where every generation step builds on what came before. Figure 4-1 gives you a high-level overview of the decoder-only architecture.

Figure 4-1. Overview of a decoder only architecture, where y+L represents the final output transformation, meaning that the decoder’s hidden state y is passed through the learned output layer L to produce the logits for the next token.

The fact that these models generate outputs autoregressively makes them brilliant for text and code generation, reasoning, and dialogue. In short, decoder-only models are your generators, your coders, your talkers, and your planners. They are the ones that think step by step, always building on their past thoughts. However, understanding how they handle that memory, literally and conceptually, is the key to making them faster and more efficient. Let’s use math to see why. To enable this causal (autoregressive) attention, the self-attention mask enforces causality:

A=softmax(QK⊤d+Mcausal )V

This means every time the model generates a new token, it would have to recompute all previous attention states, because the model can attend only to tokens ≤t at step t. However, since the previously computed K1:t−1 and V1:t−1 never change, they can safely be reused for step t+1. That’s where key-value (KV) caching comes in.

KV caching: why it matters

Key-value caching is a clever optimization that stores previously computed keys and values from attention layers so the model can reuse them instead of recalculating them from scratch. KV caching speeds up inference dramatically, but at a cost. As the sequence grows, the cache grows too, layer by layer, until memory consumption starts to dominate. The art lies in balancing speed and memory, which becomes critical when deploying models at scale or across multiple agents. As a rule of thumb, memory grows with sequence length × layers × heads. Overall, you get a throughput win at the cost of memory footprint. It’s also important to understand that state-of-the-art (SOTA) models don’t use the original attention mechanism anymore. They are using usually Grouped-Query Attention (GQA), which instead of every query head having its own K and V projections, groups multiple query heads to share keys and values, reducing KV cache size dramatically. In Hugging Face you can easily access your model’s architecture by first loading the model with model = AutoModelForCausalLM.from_pretrained(model_name) and then calling model.config this will print the whole architectural setup, for Qwen2.5-7B-Instruct-1M this gives this information about its GQA setup:

"num_attention_heads": 28,
"num_key_value_heads": 4,

That Qwen 2.5 setup, 28 query heads sharing just 4 key-value heads, is the perfect modern example of GQA in practice. Table 4-2 details how this affects memory.

Table 4-2. Grouped Query Attention Configuration (Example: Qwen 2.5)

Component
Value
Meaning

Q-heads

28

Independent query projections to preserve expressiveness.

K/V-heads

4

Shared key and value projections to minimize cache size and memory bandwidth.

Grouping ratio

28 ÷ 4 = 7

Each K/V pair serves 7 query heads. ~7× reduction in KV cache memory and bandwidth during decoding

Efficient KV cache management is essential for scaling your decoder-only models, as it directly impacts both memory footprint and responsiveness. Proper management sets the basis whether your system scales gracefully or becomes a bottleneck under load.

Modernized Encoder-Decoder Architecture: Flash Attention T5

Encoder–decoder architectures can in some cases outperform pure decoder-only models, especially in multitask or zero-shot settings. Flash Attention T5 (FAT5) is a modernized encoder–decoder model that updates the original T5 ¹ design for today’s efficiency standards. By integrating Flash Attention into T5’s architecture, it removes the main attention bottleneck, achieves near linear memory behavior, and supports significantly longer context windows. The model preserves strong generation capabilities while enabling efficient encoder-only fine-tuning for classification tasks, reducing training time without losing accuracy.

One approach is to reuse KV caches for shared prefixes across beams during beam search decoding to avoid redundant attention computations. This can significantly cut memory usage and speed up inference, especially in large-beam setups. As the context grows, implement prefix trimming or a sliding window mechanism once the sequence approaches the model’s maximum context size. Older tokens that no longer contribute meaningfully to the output can be safely dropped, reducing memory use. In addition to memory optimization, this is also a practical mitigation for the haystack problem, where too much historical context obscures relevant information.

Softmax and the Haystack Problem

In transformers, attention weights are computed with Softmax. As the context grows, the denominator of Softmax, which sums over all token scores, becomes very large while each individual score stays roughly the same. This flattens the distribution, reduces contrast between important and irrelevant tokens, and makes it harder for the model to identify key information. This challenge is known as the haystack problem.

In deployment, you should continuously monitor cache hit ratio and memory pressure. Low hit ratios often reveal inefficient scheduling or unnecessary cache invalidations, while high memory pressure signals the need for paging or trimming strategies. Frameworks such as vLLM already support hierarchical paging and segment-level cache reuse to maintain stability under high concurrency. Table 4-3 shows an overview of the most important vLLM features for helping you to manage your KV cache efficiently.

Table 4-3. vLLM Cache Management Features

Mechanism
Description
Benefit

Paged Attention

Splits the KV cache into fixed-size memory blocks managed by a free_block_queue; the scheduler allocates and reclaims these blocks dynamically.

Enables efficient reuse, prevents fragmentation, and supports long-running sessions.

Prefix Caching

Hashes repeated token prefixes and reuses their cached K/V pairs for new requests with identical prefixes.

Eliminates redundant prefill computation for shared prompts or templates.

Chunked Prefill

Splits long prompt prefills into smaller chunks that are processed incrementally.

Prevents a single long prompt from monopolizing cache blocks and improves concurrency.

Continuous Batching

Schedules prefill and decode together based on real-time cache availability.

Increases throughput while keeping cache state consistent.

Hash-Based Verification

Compares stored token-block hashes with incoming ones before reusing a cache entry.

Ensures cache integrity after context changes, and avoids reusing stale or mismatched KV blocks when prefixes are trimmed or reordered.

Dynamic Cache Reclamation

Returns KV blocks to the pool immediately once a request completes or is preempted.

Keeps GPU memory utilization high and stable.

Disaggregated KV Transfer

Shares KV across prefill/decode workers (for example via a shared storage connector).

Supports distributed setups and keeps latency sensitive, decodes isolated from heavy prefills.

Another thing you would want to think about is how context engineering, the deliberate modification, insertion, or pruning of tokens in the model’s input context, directly affects KV cache integrity. Since the cache stores key and value projections of previously processed tokens, any modification that alters token order or identity may invalidate portions of it.

When new documents are injected, old prefixes are trimmed, or messages are reordered, cached entries no longer align with the model’s positional encodings. Reusing them leads to incoherent or hallucinatory outputs. Therefore, every structural context change must trigger selective cache invalidation. A layered policy, that is combining windowing, segmentation, and hash verification, helps you to preserve correctness while maintaining efficiency. Intelligent context design, such as preallocating static system prompts, enables partial cache persistence even in dynamic RAG or multi-turn settings. Modern inference frameworks address this through:

Sliding-window caching to retain only the most recent N tokens for bounded attention.

Chunked or segment-level caching to isolates system prompts, retrieved context, and user turns for partial invalidation.

Stale-key eviction which discards inactive caches after a timeout.

Context hashing to compare token checksums before cache reuse to ensure correctness.

I’m sure, after reading this section, it’s clear that deploying or making a decoder-only model faster isn’t just about retraining, quantization or how many GPUs you have. It’s about engineering the inference runtime and how you effectively manage its cache. And even though that might sound overwhelming at first, frameworks like vLLM simplify the process through paged memory, prefix reuse, and distributed KV sharing. In practice, you often only need to activate these mechanisms through the framework’s configuration options, which lets you manage KV caches efficiently and with minimal operational overhead. Some functionalities, like prefix caching, are even activated by default.

Encoder-Only Models

This section switches your perspective to the often underappreciated encoder-only models. Decoder-only models are all about generation, producing one token after another. Encoder-only models, in contrast, are about understanding. They don’t generate text, they analyze it. Their entire job is to look at the input from all directions and build a deep contextual understanding of what it means.

You’ve come across models like BERT ² and RoBERTa ³ before. These models read text bidirectionally, capturing relationships between words across the whole sequence. Instead of predicting the next token, they learn what’s missing by reconstructing masked parts of the input. This approach, known as masked language modeling, teaches them to understand language structure and nuance rather than produce text.

Encoder-only architectures remove the decoder stack entirely and rely on bidirectional attention, which means every token can attend to every other token in the sequence. Figure 4-2 illustrates an abstracted encoder-only architecture.

Figure 4-2. Abstracted encoder-only architecture.

Because there is no autoregressive loop, these models process entire inputs in parallel, making them faster, lighter, and cheaper to serve. They don’t need key value caching or token by token generation, which makes them ideal for classification, retrieval, and embedding workloads.

Even as large generative models dominate headlines, encoder-only transformers continue to play a crucial role in real world systems. In RAG pipelines, for example, they act as retrievers, the fast and sharp analysts that surface the most relevant context before a generative model takes over. In large scale deployments, they are often the workhorses that enable efficiency and precision at scale. And the good news is that, even though most of the older architectures are still working fine, there exist modern adaptions of the famous BERT model. Table 4-4 shows newer model architectures, and how they improved over the original BERT in terms of positional encoding, sequence length and FlashAttention support for faster training and inference.

Table 4-4. Comparison of Encoder Model Architectures

Component
BERT (base)
BERT (large)
NomicBERT (base)
ModernBERT (base)
ModernBERT (large)
NeoBERT (medium)

Layers

12

24

12

22

28

28

Hidden Size

768

1024

768

768

1024

768

Attention Heads

12

16

12

12

16

12

Parameters

120M

350M

137M

149M

395M

250M

Positional Encoding

Positional Embeddings

Positional Embeddings

RoPE

RoPE

RoPE

RoPE

Sequence Length

512

512

2048

1024 → 8192

1024 → 8192

1024 → 4096

FlashAttention Support

✗

✗

✓

✓

✓

✓

NeoBERT ⁴ is the latest modernization of the classic BERT architecture and performs extremly well on extended sequences. Figure 4-3 compares the model throughput (tokens per second) as a function of sequence length (higher is better).

Figure 4-3. Throughput comparison of the two most recent modernized BERT architectures, ModernBERT and NeoBERT. Image is taken from NeoBERT, Lola Le Breton et al.

So, even though encoder-only models once seemed overshadowed by the rise of decoder-only architectures, their evolution has quietly continued. Modern variants such as NomicBERT, ModernBERT ⁵, and NeoBERT integrate SOTA techniques like rotary positional encodings for extended context windows and FlashAttention kernels for faster training and inference.

Rotary Positional Embeddings (RoPE)

Traditional positional encodings, either fixed sinusoidal or learned, allow models to understand token order but fail to generalize well beyond their training context (for example, from 2k to 32k tokens). Rotary positional embeddings (RoPE) ⁶ address this limitation by encoding relative position through rotation rather than addition.

Instead of adding a positional vector to each token, RoPE rotates the query and key vectors in the attention mechanism by an angle that increases with token position. The dot product between two rotated vectors naturally reflects their relative distance, enabling the model to extrapolate to much longer sequences.

You can think of RoPE as arranging tokens around a circle: each word occupies a unique angle, and their angular difference represents relative distance. Nearby tokens align closely, while distant ones are separated by larger rotations. Figure 4-4 illustrates this.

Figure 4-4. Illustration of Rotary Position Embedding(RoPE). Image adapted from: Jianlin Su et al.

Extending RoPE-based models is straightforward. Modern libraries like vLLM or Transformers let you rescale RoPE frequencies to expand the context window at load time:

from vllm import LLM, SamplingParams

llm = LLM(
    model="answerdotai/ModernBERT-base",
    trust_remote_code=True,
    rope_scaling={"type": "linear", "factor": 4.0}
)

output = llm.generate("Summarize this 64k-token document...",
                      SamplingParams(max_tokens=512))
print(output[0].outputs[0].text)

Here, rope_scaling increases the model’s effective context length fourfold, from 32k to 128k tokens, allowing efficient long-sequence reasoning without retraining.

These innovations bring them closer to the efficiency and reasoning depth of their decoder-based counterparts while preserving the precision and bidirectionality that make them indispensable for retrieval, classification, and embedding tasks. As a result, encoder-only transformers remain central to modern AI systems, quietly powering the understanding layer beneath today’s most advanced agentic systems.

This is why encoder-only models are your analysts and investigators within your team of AI agents. They don’t speak: they listen, interpret and make sense of everything before passing it on. Their strength lies in understanding context deeply and efficiently, making them an essential part of any well-structured agent ecosystem.

Mixture of Experts Models

Now let’s look at one architecture that changes how you can think about scale. MoE models take inspiration from how teams work in the real world. Not everyone needs to do every task. Some people specialize, and the challenge is to know whom to ask for help. MoE models apply that same logic to transformers.

Instead of having every layer process every token, which a dense model does, MoE models divide the work among multiple specialized networks called experts. These experts replace the standard Feed-Forward Networks (FFNs) found in dense transformer blocks. A lightweight gating network acts as a router that decides which experts to activate for a given input. In dense transformers, every token flows through the same FFN, while in MoE models the router selects only a few specialized FFNs. Only a small number of experts are used at a time, while the rest remain idle. This means the model can hold billions of parameters but only use a small fraction of them per forward pass. Figure 4-5 illustrates this flow.

Figure 4-5. High-level overview of Mixture of Experts architecture.

The result is a model that scales capacity without scaling cost. A trillion-parameter MoE model can operate at a runtime cost similar to a much smaller dense model because it activates only the experts that matter. This concept is known as conditional computation: compute is spent only where it’s needed.

The gating mechanism itself can vary. Common approaches include top-k routing, where the router activates the k most relevant experts; noisy gating, which adds stochasticity to improve expert utilization; and expert-choice routing, where experts bid for tokens instead of being assigned by the router. While these differences matter mainly during training rather than inference, they strongly influence load balancing, stability, and overall model quality.

Again, if you’re thinking in team-member terms, MoE models are your specialists. They don’t try to be good at everything, but when you route the right problem to the right expert, the system becomes both powerful and efficient. This makes MoE one of the most important architectural advances in large-scale language modeling, allowing teams of models to collaborate just as human experts would, each contributing their unique strength at the right moment.

Reasoning Models

If decoder-only models are your generators, encoder-only models your analysts, and Mixture of Experts your specialists, reasoning models are your thinkers. They are the ones who pause before they act, break problems into steps, and reflect on their own conclusions.

Reasoning models are not defined by architecture alone but by behavioral optimization. They’re often decoder-based under the hood, but trained or prompted to use structured reasoning chains, planning traces, or tool-augmented reflection. Instead of predicting the next token directly, they learn to generate intermediate thoughts that guide better answers.

Some reasoning models, such as DeepSeek-R1, Qwen3, and OpenAI’s “reasoning” variants of GPT, extend this further with inference-time optimization: the model is encouraged to search its own reasoning space at runtime, sometimes generating multiple parallel solution paths before selecting the most consistent one. This shift marks the emergence of a new category, models that not only respond but reason dynamically.

Architecturally, they remain close to decoder-only transformers, but their real innovation lies in the inference loop rather than the network layers. They blend symbolic persistence (through reasoning tokens or hidden-state reuse) with probabilistic exploration (via multi-sample decoding or internal consistency checks). The result is slower generation, but deeper and more reliable decision-making.

In team-member terms, these are the colleagues who don’t rush to speak. They think, reflect, and sometimes even argue with themselves before offering an answer. They are slower, but they often save the whole team from avoidable mistakes. They are the reflective minds in an otherwise reactive system.

Thinking modes

Modern reasoning models support a control surface for how they reason. You can switch between a thinking mode and a fast response mode with deployment specific switches. With the Transformers library, you can enable or disable the internal reasoning trace directly in the chat template (Example 4-1).

Example 4-1. Switching between thinking and non-thinking mode with Transformers

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)

Enable thinking, which is the default mode.

With OpenAI compatible servers such as vLLM, you can pass mode switches through the client call, see Example 4-2.

Example 4-2. Switching between thinking and non-thinking mode with vLLM

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "user", "content":
        "What are reasoning models, in terms of Large Language Models?"},
    ],
    max_tokens=8192,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

{"enable_thinking": False} to disables thinking mode.

However, you can also “just tell” the model to not use thinking in the prompt. Even when thinking is disabled, many models keep an internal structure for the hidden trace, which preserves format consistency and allows you to toggle thinking without changing downstream code.

Use these switches to align latency and quality with the task. Enable thinking for planning, decomposition, tool selection, and verification. Disable thinking for short, low risk turns, or for steps where you already validated the plan.

Thinking budget

Reasoning tokens are not free. Many models expose a thinking budget that caps the number of tokens allocated to the internal trace. When the budget is exhausted mid-inference, the model can truncate its reasoning and continue the response gracefully. This can include a short handoff phrase that signals the budget boundary, as illustrated in Example 4-3.

Example 4-3. Set thinking budet

thinking_budget = 1024
max_new_tokens = 32768

# Rest of code is omitted for brevity

if 151645 not in output_ids:

    if 151668 not in output_ids:
        print("thinking budget is reached")
        early_stopping_text = "\n\nConsidering the limited time by the user,
        I have to give the solution based on the thinking directly now.\n</think>\n"
        early_stopping_ids = tokenizer([early_stopping_text], return_tensors="pt",
        return_attention_mask=False).input_ids.to(model.device)
        input_ids = torch.cat([generated_ids, early_stopping_ids], dim=-1)
    else:
        input_ids = generated_ids
    attention_mask = torch.ones_like(input_ids, dtype=torch.int64)

    generated_ids = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=input_length + max_new_tokens - input_ids.size(-1)
    )
    output_ids = generated_ids[0][input_length:].tolist()

Check if the thinking process has finished, 151668 is </think>, then prepare the second model input for the second generation

Second generation

It might yield a negative result if max_new_tokens is too small, since the early stopping text consists of 24 tokens.

Every Token Counts

Be mindful that each token counts, either if you’re running the model self-hosted or via a managed API. Each reasoning step, tool invocation, and intermediate generation contributes to the final bill.

Choose the budget based on acceptable latency and task complexity. A budget that is too small can limit multi-step improvement and reduce consistency. In practice, values at or above one thousand tokens tend to work well for non-trivial reasoning, and you can raise the budget for math, code generation, and multi tool plans where deeper exploration pays off.

Putting It All Together: Designing a Team of Experts

By now you’ve seen how different model architectures bring unique strengths to your agentic systems. In this section you’ll build a MAS based on this knowledge. Each agent in the following setup represents a distinct capability that contributes to the system’s overall reasoning, retrieval, and synthesis process. Table 4-5 shows the implementation with its roles and models.

Table 4-5. Agent team at a glance

Role
Architecture
Concrete model
Primary task
Tools it calls

Semantic research

Decoder only fast

Qwen3 32B

Neural search with highlights

External APIvia exa.search_and_contents

Patent research

Decoder only fast

Qwen3 32B

Past year patent hits

GoogleSerperAPIWrapper

Analyst

Hybrid controller plus encoder

Reasoning controller Qwen3 235B Thinking, Encoder ModernBERT base

Semantic filtering and extractive summary with layered reasoning then encoder scoring

semantic_filter_tool

Note taker

Decoder only fast

Qwen3 32B

Outline creation

create_outline, read_document

Writer

Decoder only fast

Qwen3 32B

Draft and edits

write_document, edit_document, read_document

Supervisor

Deliberate LLM

GPT 4.1

Routing and finish

agents as tools

Together, these specialized agents form a coordinated research and writing pipeline. The supervisor oversees the entire process, routing tasks between the agents and determining when the system has reached its final outcome. Figure 4-6 illustrates how these components connect, highlighting the flow of information and reasoning within the multi-agent system.

Figure 4-6. Illustration of the research team implemented in LangGraph.

The code in this section focuses on how each agent is defined. The complete code, including tools, graph wiring and routing, is intentionally omitted so the focus remains on the division of roles and the architectural choices behind them.

Example 4-4 defines the specific skill set for each agent in your team.

Example 4-4. Simple text generation

EXA_PROMPT = """Role: Research assistant. You can search for all recent info
on Exa Search. Your response should clearly articulate the key points you found."""

PATENT_PROMPT = """Role: Market researcher with 20 years of experience.
You are very knowledgeable in patent research and in finding up-to-date info
about patents using the Google Patents API."""

NOTE_PROMPT = """You can read documents and create outlines for the document
writer. Don't ask follow-up questions."""

WRITER_PROMPT = """You can read, write and edit documents based on note-taker's
outlines. Don't ask follow-up questions."""

ANALYST_PROMPT = """
You are the Analyst. You must call the `semantic_filter_tool` exactly once.
Inputs:
- query: the current task topic or question
- documents: a list of raw strings gathered by teammates

Rules:
- don't answer the user
- don't do free form writing
- Only return the tool result to the supervisor
""".strip()

For the open source models I used Nebius, but you can use any Open AI compatible cloud provider such as Novita or Together AI. Example 4-5 shows my models setup. You’ll see that I deactivate the thinking of Qwen3-32B, and that I assign a thinking budget for Qwen3-235B-A22B. This is usually the same schema for common cloud providers and is adapted from the Qwen documentation on deploying the model via vLLM.

Example 4-5. Setting up the models

fast_llm = ChatOpenAI(
    model="Qwen/Qwen3-32B-fast",
    temperature=0,
    api_key=NEBIUS_API_KEY,
    base_url="https://api.studio.nebius.ai/v1/",
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

thinker_llm = ChatOpenAI(
    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
    temperature=0,
    api_key=NEBIUS_API_KEY,
    base_url="https://api.studio.nebius.ai/v1/",
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True},
        "thinking": {"type": "token", "budget": 1536}
    },
)
supervisor_llm = ChatOpenAI(
    model="gpt-4.1",
    temperature=0
)

Example 4-6 shows how to implement a helper function to create your agents faster.

Example 4-6. Create react worker node

def make_react_worker_node(
    *,
    llm: ChatOpenAI,
    name: str,
    tools: list,
    prompt: str | None = None,
    goto: str = "supervisor",
):
    agent = create_react_agent(llm, tools=tools, prompt=prompt)

    def node(state: State) -> Command[Literal["supervisor"]]:
        result: Dict[str, Any] = agent.invoke(state)
        msgs: Sequence[BaseMessage] = result.get("messages", [])
        content = getattr(msgs[-1], "content", "") if msgs else ""
        return Command(
            update={"messages": [HumanMessage(content=content, name=name)]},
            goto=goto,
        )

    return node

With this helper function, you can create your individual agents as show in Example 4-7.

Example 4-7. Create agents

specs = [
    dict(
        name="search", tools=[tavily_search, tavily_extract],
        prompt=SEARCH_PROMPT, llm=fast_llm
        ),
    dict(
        name="exa_search", tools=[exa_search_tool],
        prompt=EXA_PROMPT, llm=fast_llm
        ),
    dict(
        name="patent_research", tools=[patent_search],
        prompt=PATENT_PROMPT, llm=fast_llm
        ),
    dict(
        name="analyst", tools=[semantic_filter_tool],
        prompt=ANALYST_PROMPT,llm=thinker_llm
        ),
    dict(
        name="note_taker", tools=[create_outline, read_document],
        prompt=NOTE_PROMPT, llm=fast_llm
        ),
    dict(
        name="doc_writer", tools=[write_document, edit_document, read_document],
        prompt=WRITER_PROMPT, llm=fast_llm
        ),
]

nodes = {
    s["name"]: make_react_worker_node(llm=s["llm"], name=s["name"],
    tools=s["tools"], prompt=s["prompt"])
    for s in specs
}

When you run the code and inspect the output of the analyst agent you’ll see something like this:

Analyst (ModernBERT) processing 2 docs...
'analyst': {'messages': [HumanMessage(content='\n\n[\n {\n ""text": "### Latest
Developments in AI Agents: A Focus on Coding Agents\\n\\nAI agents, particularly
those focused on coding, are rapidly evolving and transforming the landscape of
software development.",\n "source": "Source Doc [1]",\n "score": 0.956\n  },
# Rest of output omitted

You see in the second last line the score of 0.956, which is the relevance ranking score ModernBERT assigned to this particular content.

This setup gives you what you want: a single, deliberate analyst that reasons carefully over evidence, cross-checked by a modern encoder model, while everyone else remains fast and cheap. This is an example how to design self-auditing systems, by checking for relevance and potential hallucination in addition to prompting the model to provide the links to its resources.

Open vs. Closed Models: Why the Choice Matters

There is a long-standing debate on whether one should use open or closed source models. But the open-vs-closed choice is not just a technical decision, it defines your company’s strategic trajectory in building products with AI agents. When you look more closely, this choice affects everything from cost structure and compliance to innovation velocity, vendor dependence, and long-term data sovereignty. This section aims to help you to think about this from an analytical standpoint, by considering the following factors.

Speed of innovation and deployment

Contributions from the global research community drive new architectures and optimization techniques, while ecosystems such as Hugging Face, vLLM, or Ollama make these innovations accessible. In contrast, closed-source models progress at the pace of internal R&D cycles and scheduled releases. But it’s not just this speed of innovation. You should also think about your own speed of innovation. That is your deployment speed adds nuance to this discussion. Closed models can be used instantly through APIs. Most open models can be accessed via managed cloud deployments, using providers like Together.ai, allowing rapid experimentation before teams eventually self-host for cost control and customization.

Accessibility

Open-source models are freely accessible for experimentation, inspection, and benchmarking. They often provide visibility into training data, architectures, and weights, which can be critical for auditing and compliance. Closed-source APIs restrict visibility and control, with limits on rate, cost tiers, and regional access. Accessibility also includes operational ease. Closed models remove infrastructure complexity and provide direct access, whereas open models might require infrastructure management.

Model optimization and customization

Open models still offer the broadest form of control. While you can fine-tune or customize a closed source model like GPT-4.1 via DPO or RL, this model exists still only inside the vendor’s environment. You don’t control the model weights, deployment, or runtime. In other words, customization is possible, but you are still vendor locked in. Open-source models let you carry your improvements with you; closed systems let you borrow flexibility, not own it.

Performance

Closed-source models often lead benchmark leaderboards due to proprietary compute budgets and training data advantages. However, the performance gap has narrowed significantly. Modern open architectures such as Qwen-3 or Kimi K2 come close in performance to proprietary models. And with optimization techniques as you’ve read about in the previous chapter (“The ART of Learning from Experience”, “Enabling General-Purpose Agentic Programs Through Post-Training and Search-Based Agents”), you can even train your models to improve their tool usage or collaborate better as a team.

Cost

Closed models can cost up to $150 for input and $600 (o1-pro) for output per million tokens, while open models fall around $0.3 for input and about $3 for output per million tokens when used via low-cost cloud providers even for models over 200B parameters. In addition, when you host them yourself, trade-off becomes operational. Open-source deployments require GPUs, scaling infrastructure, and ongoing maintenance. In the short term, managed APIs can appear cheaper because they remove the need for specialized staff, but in the long run, self-hosted open models usually yield a lower total cost of ownership once scaled efficiently.

Data security and compliance

When self-hosted or deployed on private cloud infrastructure, open models ensure full data ownership and compliance control, important for regulated industries like finance or healthcare. Organizations can implement granular access control, audit mechanisms, and on-prem encryption layers. With managed models, either via closed or open models, you rely on vendor-managed security and compliance certifications (SOC-2, ISO-27001, etc.). This can simplify procurement but at the cost of reduced visibility and dependence on vendor diligence.

Given all of this, a hybrid strategy often makes the most sense. When you start to build your agentic system using managed closed APIs or managed open-source providers lets you move fast and focus on building. It keeps things simple while workflows and responsibilities are still forming. However, over time, as your team grows and your use cases begin to require more control or customization, it becomes more practical to bring open-source deployments in-house. With a larger team and clearer ownership of responsibilities, moving infrastructure and model management under your own roof often becomes the more effective path.

Optimizing Open Source Models

When you decide to go the open-source route, there are several ways to optimize both performance and cost. Traditionally, fine-tuning a large model meant updating all of its parameters as a continuation of the original training process. For models with billions of parameters, this becomes expensive, since each trainable parameter adds roughly a fourfold memory overhead during optimization, and every checkpoint can occupy tens of gigabytes of storage.

Modern pipelines therefore rely on PEFT, a family of techniques that adapts large models by updating only a small subset of weights. The most common forms are LoRA and adapter modules, which specialize a base model for new tasks without retraining it from scratch, and quantization, which reduces memory and inference cost by lowering weight precision.

The following subsections walk you through these methods, serving as a practical guide to choosing the right optimization strategy when building and deploying agentic systems.

LoRA and Adapters

Low-Rank Adaptation ⁷ (LoRA) and adapter modules are among the most popular methods to make fine-tuning more efficient. LoRA works by injecting low-rank matrices into existing weight layers, learning small corrective updates without touching the original model weights. Even though only a fraction of the parameters are updated, LoRA achieves performance comparable to full fine-tuning. During inference, the adapter weights are merged with the base model parameters so the system operates as a single unified model.

Adapters follow a similar principle by inserting small trainable layers between transformer blocks. This makes it possible to achieve near full fine-tuning performance at a fraction of the compute and memory cost. Instead of retraining the entire model, you insert additional lightweight modules that learn task-specific representations while keeping the base model frozen. This isolation helps prevent catastrophic forgetting, a phenomenon where a model loses previously learned capabilities when fine-tuned on new data. By preserving the original weights and only training a small number of additional parameters, adapters retain prior knowledge while learning new tasks. Both methods make it easy to switch between task-specific configurations by loading or unloading adapter weights as needed. This modularity is especially useful in production environments where a single model backbone must support multiple domains or languages.

A recent innovation that make these ideas easy to implement is LoRA Exchange (LoRAX), which redefines how fine-tuned models are served in production. Instead of dedicating separate GPU resources to each LoRA adapter, LoRAX enables dynamic loading and batching of multiple fine-tuned models on a shared GPU. Through mechanisms such as dynamic adapter loading, tiered weight caching, and continuous multi-adapter batching LoRAX helps you to pack over a hundred specialized LoRA models into a single deployment.

Dynamic adapter loading allows adapter weights to be fetched just-in-time at runtime, keeping latency minimal while eliminating the need to preload all fine-tuned variants. Tiered weight caching manages GPU, CPU, and disk memory to avoid out-of-memory errors, while continuous multi-adapter batching ensures requests for different adapters can be processed concurrently and fairly. Together, these strategies make it possible to serve many specialized adapters on a single GPU with one base model, helping you to lower your inference costs. For instance, a team could fine-tune different domain-specific models for customer support, legal analysis, or code generation, and deploy them all under one shared GPU pool, something that could otherwise cost tens of thousands of dollars in cloud resources.

Quantization

Quantization reduces model size and speeds up inference by representing weights with fewer bits, such as 8-bit or even 4-bit precision. This compresses the model’s memory footprint, lowers GPU requirements, and enables faster inference. Frameworks such as bitsandbytes and unsloth simplify this process, offering robust quantization routines and optimized training kernels. In addtion, many recent models released by developers such as Meta, Google, or Alibaba Cloud already include quantized variants by default, which makes deployment easier and more cost-efficient.

Usually, a well-optimized open-source pipeline often combines these techniques: quantized weights for efficient inference, LoRA or adapters for task specialization, and PEFT for adaptation. Together, they enable you to achieve high-quality results on smaller budgets and with shorter iteration cycles.

Conclusion

The central lesson of this chapter is that your agentic system’s capability comes from the fit between task, model, and inference strategy. Models are team members with distinct strengths and skills. Decoder-only models are your generators and planners. Encoder-only models are your analysts and investigators. Mixture of Experts models are your specialists that add capacity without paying full compute on every token. Reasoning models are your deliberate thinkers, the ones that pause, reflect, and refine before acting. When you staff your system with this mindset, you get higher quality at lower cost.

You saw that architectural choice is as much an inference problem as a modeling problem. KV caching turns sequential generation into a throughput win by reusing past keys and values, but it increases memory pressure. With reasoning models, you also need to decide when to enable thinking modes and how to set thinking budgets, so you spend compute where it matters and keep fast paths where it does not.

You composed these pieces into a working organization of agents. A deliberate controller that reasoned, an encoder that verified relevance, fast decoders for drafts and edits, all orchestrated by a supervisor that routed and decided when to finish the task.

You learned that open and closed model choices are not purely technical. They set your trajectory for cost, compliance, and data sovereignty. A hybrid path is often the pragmatic route. Start with managed APIs for quick iteration, then move to open-source self-hosting as your team and workflows mature.

Parameter-efficient fine-tuning lets you adapt models without touching every weight. LoRA and adapters provide modular specializations you can swap in and out per task or domain. Quantization shrinks memory and accelerates inference. Together with frameworks like bitsandbytes and unsloth, these techniques turn advanced optimization into a routine part of your build process.

Taken together, these ideas close the gap between architecture, inference, and operations. Choose the right roles, allocate thinking where it pays off, optimize the parts you own, and adopt a deployment strategy that matches your stage.

The next chapter shifts the focus from model architecture to system integration and execution. You will learn how to connect your agents to the outside world through structured tool calls, secure execution environments, and well-defined interface contracts.

¹ Colin Raffel et al. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.”, (2019).

² Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.”, (2019).

³ Yinhan Liu et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach .”, (2019).

⁴ Lola Le Breton et al. “NeoBERT: A Next-Generation BERT.”, (2025).

⁵ Benjamin Warner et al. “Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference.”, (2024).

⁶ Jianlin Su et al. “RoFormer: Enhanced Transformer with Rotary Position Embedding”, https://arxiv.org/abs/2104.09864 (2021).

⁷ Edward J. Hu et al. “LoRA: Low-Rank Adaptation of Large Language Models.”, (2021).

About the Author

Nicole Koenigstein is the Founder, CEO, and Co-Chief AI Officer of a deep-tech company specializing in large language model (LLM) systems and complex multiagent architectures. She is a globally recognized thought leader in LLMs and agentic architectures, with a particular focus on their transformative applications in quantitative finance. Nicole has served as an external expert for the European Commission on LLM research funding and for the International Organization of Securities Commissions (IOSCO), advising on GenAI adoption in regulated industries. She also serves as an advisory board member for QuantMinds International and QuantStrats, two of the world’s largest and most influential fintech, AI and quant strategy conferences. Over the past 2.5 years, Nicole has delivered more than 50 invited talks, keynotes, and technical workshops globally—including at Oxford University, Cornell University, Bank of America, J.P. Morgan, and the International Faculty of Finance—as well as at international events such as European Women in Technology, ACM Tech Talk, and ODSC Europe.

Nicole is a faculty member at the Artificial Intelligence in Finance Institute in New York, where she teaches coding sessions on Python, machine learning, deep learning, large language models and agentic AI. She is the author of Math for Machine Learning and Transformers in Action (Manning Publications), and her forthcoming title Transformers: The Definitive Guide will be published by O’Reilly Media.