Generative AI on Microsoft Azure

From Large Language Models to Advanced Multi-Agent Systems

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

Adrián González Sánchez, Jaime De Mora, and Jorge Garcia Ximenez

Generative AI on Microsoft Azure

by Adrián González Sánchez, Jaime De Mora, and Jorge Garcia Ximenez

Copyright © 2026 Adrián González Sánchez, Jaime De Mora, and Jorge Garcia Ximenez. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

Acquisitions Editor: Megan Laddusaw

Development Editor: Sara Hunter

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Kate Dullea

May 2026: First Edition

Revision History for the Early Release

2025-04-17: First Release

2025-07-18: Second Release

2025-08-22: Third Release

2025-10-27: Fourth Release

2025-12-08: Fifth Release

See http://oreilly.com/catalog/errata.csp?isbn=9798341623286 for release details.

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Generative AI on Microsoft Azure, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.

The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

979-8-341-62328-6

Brief Table of Contents (Not Yet Final)

Preface (unavailable)

Chapter 1: Technical Fundamentals of Generative AI Models and Applications (available)

Chapter 2: Advanced AI Development in Azure (available)

Chapter 3: Model Selection and Prompt Engineering Best Practices (available)

Chapter 4: Retrieval Augmented Generation (RAG) (available)

Chapter 5: Fine-Tuning Generative AI Models in Azure (available)

Chapter 6: Agentic Systems (available)

Chapter 7: GenAIOps and LLMOps in Azure (available)

Chapter 8: Generative AI Governance Framework (available)

Chapter 9: Case Studies and Expert Interviews (unavailable)

Chapter 1. Technical Fundamentals of Generative AI Models and Applications

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the first chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

We can affirm that generative AI is already a big industry in itself. It can dominate supply chain and geopolitics, directly impact trading stock prices, and define the immediate destiny of companies and executives around the world. This chapter intends to be the entry door to this revolution, by introducing not only Microsoft Azure as a differentiator for your generative AI projects, but also complex topics in terms of advanced adoption patterns, model training and customization phases, etc.

From a purely technological perspective, since the arrival of the first GPT models, Microsoft Azure has played a significant role in its evolution, providing the infrastructure and tools necessary for large-scale AI model development and deployment. Because of the role of Microsoft technologies as the enterprise-grade de facto choice, most of the big adopters have leverage tools such as Azure AI Foundry, Azure OpenAI Service, and more recently Microsoft Fabric. Don’t get me wrong, we don’t intend to sell anything here, but the market talked and there is an omnipresence of Azure for all generative AI, enterprise-level things.

One of the key differentiators of Microsoft Azure since 2022 was the launch of the Azure OpenAI Service. This collaboration between Microsoft and OpenAI has provided enterprises with access to models GPT and other models at scale, allowing them to integrate AI-driven solutions into their workflows. Beyond offering access to pre-trained models, Microsoft Azure also has supported product evolutions in fine-tuning and custom AI deployments, enabling businesses to tailor AI solutions to their specific needs. That was a first mover advantage that had wide impact at industry adoption level, but also to continue adding functionalities beyond the core AI models part.

But the rise of generative AI models has driven demand for high-performance computing, scalable cloud solutions, and efficient data management. Azure has responded by expanding its AI capabilities, offering more powerful Graphic Processing Units (GPU) and Provisioned Throughput Units (PTU) instances to support intensive AI workloads. The integration of AI accelerators and optimized pipelines has made it possible to train and fine-tune large language models (LLMs) and other generative AI applications with greater efficiency.

Since then, Microsoft Azure has continued to refine its suite of AI development tools. Azure AI became the Azure AI Foundry, and introduced new automation, safety, web app deployment, API management, and monitoring features, improving the testing and management of AI models in production. It also evolved the model catalog to expand and add new models from other providers such as Mistral, Meta, DeepSeek, or Cohere, among others. Some of them are available via Model-as-a-Service (MaaS) serverless APIs, and others as deployable options via cloud native infrastructure such as virtual machines and containers. Additionally, the Azure AI Foundry included new enhancements to the original Azure Cognitive Services and expanded AI capabilities in areas like natural language understanding, video and image recognition, vertical applications for healthcare, etc. Everything is on the same platform.

As generative AI becomes more widespread, concerns around security, bias, and ethical AI usage have grown. Azure has introduced updated governance frameworks and AI monitoring tools to promote transparency and fairness in AI applications. This includes features such as AI safety to detect unfair usage, LLMOps to track performance metrics, and detailed model cards to provide as much transparency as possible. Features like these help organizations deploy AI responsibly, ensuring alignment with global regulations and standards.

Beyond 2026, Azure’s role in generative AI will likely continue to evolve, with ongoing investments in AI chip development, efficiency improvements, and hybrid AI-cloud integrations. Research collaborations and advancements in AI infrastructure will further shape the landscape, allowing developers and businesses to explore new frontiers in AI-driven creativity and automation.

Without exaggerating the value of the Microsoft Azure suite, this position as first mover and reference for enterprise-grade implementations, along with the accelerated evolution illustrates broader trends in AI democratization, scalability, and ethical considerations. As AI continues to evolve, Microsoft Azure remains a pivotal platform for innovation for a lot of companies, bridging the gap between cutting-edge research and real-world applications.

Evolution of the Adoption Patterns

But this accelerated pace of innovation requires an increasing level of AI knowledge, at both model and architecture levels. Compared to the initial moment of the appearance of OpenAI’s ChatGPT, adopter companies have evolved their generative AI developments with more complex building blocks, and so do the upskilling process of their professionals.

This is due to the adoption pattern from Figure 1-1 below, that have evolved in an iterative manner to keep adding new pieces to the puzzle of their end-to-end architectures:

Figure 1-1. Evolution of the Generative AI Adoption Patterns

Let’s dive into each phase of the process in greater detail:

Deploying initial “ChatGPT” instances → Most of the companies started replicating Generative Pre-trained Transformer (GPT) enabled bot-kind of applications, with private instances deployed on their own Microsoft Azure environment, often based on the Azure OpenAI Service capabilities, or equivalent models on Azure. In this case, there was still no connection to remote systems such as databases or Customer Relationship Management (CRM) systems. These private instances didn’t include live internet searches either, or linked references to the original sources. Only the baseline GPT model and an integration with a front-end web application.

Adding new knowledge bases → Baseline GPT models include a limited knowledge scope which represents a “snapshot” of the data sources used for training purposes, such as public websites, open datasets, books, and documents. That snapshot of the information extracted from the sources goes only until a certain time period of time, depending on the model version release and their data refresh (e.g. data up to April 2025 for version 1, January 2026 for version 2). For that reason, and given that most of the AI adopter companies needed to add their own private data (e.g. documents, images) for knowledge management and productivity purposes, companies started to integrate and customize with Large Language Model (LLM) developments in different ways. You can read about the available customization techniques in chapter 3, including Retrieval Augmented Generation (RAG), fine tuning, and others.

Securing model and data → As a natural evolution of the model experimentation, and the connection with the data sources, new needs arrive in terms of security, safety, and privacy. For example, it became necessary to create private links between the models, search engines, and data sources (including vector databases). Other actions include: protecting the prompt interfaces from potential attacks like jailbreaking attempts, filtering negative topics as part of the model inputs and outputs, and securing the perimeter of the cloud environment.

Exploring code-based frameworks and libraries → Besides connecting to data sources, adopters in this phase started to include other remote systems such as local environments, external CRM or ERP (Enterprise Resource Planning) applications, and even other remote LLM for communications between agents. Chapter 3 will include agentic AI applications, including the famous Multi-Agent Systems (MAS), and chapter 4 will finish with a selection of frameworks and libraries.

Monitoring performance → As part of the new LLM operations (GenAIOps or LLMOps) concept, adopter companies started to validate model performance and behaviour in both manual and automated ways. Initially manually with expert reviews to evaluate answers from the LLM-enabled systems, and then with automated metrics. This last one is a work-in-progress domain with continuous product announcements and new startups into the field. Chapter 5 will include metrics, tools, and techniques that measure the quality of generative AI implementations, so you can establish an advanced approach to GenAIOps.

Deploying at scale with AI landing zones → Besides the initial building blocks (models, data, security, frameworks, monitoring), companies soon required new ways to reach high scalability at both performance and cost level for their LLM deployments. You will explore in chapter 4 options like API managers to combine different models in several geographies, optimized AI infrastructure, or reserved instances for provisioned capacity at fixed cost and high service availability thanks to higher API usage limits. All these elements together are part of what adopters companies consider as “AI landing zones”, and their way to move their initial experiments to production.

Converging with other cloud environments → Following the natural evolution of technology and infrastructure adoption at corporate level, organizations expanded their AI footprint along their multiple private and public clouds. Sometimes with on-prem data sources connected to cloud-enabled models by leveraging secured, private connections, or with different source cloud data lakes feeding the LLM deployments and vector databases in Microsoft Azure.

Combining models and multimodal capabilities → Companies and developers are combining different models from the same or different providers, large and small (i.e. Small Language Models or SLM), and new multi-modal capabilities that handle all sorts of data (image, text, video) in one single model. In chapter 3, you’ll learn about the unified model catalogs from the Azure AI Foundry, or even development platforms such as Github with the Github Models feature.

Incorporating data and AI governance considerations → These topics are not the last step, but continuous and iterative ones. The increasing level of AI adoption, along with the new regulations and standards (e.g. EU AI Act, ISO 42001) brought the AI discussion not only to the technical and business departments, but also to the legal and compliance teams that wanted to understand the potential considerations of using generative AI models. Chapter 6 proposes an advanced framework that combines all Microsoft and third-party tools available for an end-to-end generative AI governance, including organizational and technical levels.

Because of that increased level of advanced AI adoption, the scope of what generative AI represents in terms of capabilities and use cases is wider and way more complex than a few years ago. The other variable to consider is the overall evolution of the Generative AI models.

As you can see in Figure 1-2, Generative AI models started with text and code-based applications. They developed to include vision (for both video and images) and even real time audio. This evolution refers to the road to multimodality that most of the top model providers are following and that means supporting these modalities in a native way. From a capabilities point of view, there has been an increase of the window context (the maximum amount of tokens that a model can consume each time), new functionalities like function calling, added reasoning abilities, and direct interaction with tools via computer use agents.

Figure 1-2. Advancements in Generative AI - Modalities and Capabilities

In reality, Generative AI capabilities have evolved in an unstoppable manner during the last decade. From the initial research on Generative Adversarial Networks (GAN) during the modern 10s, passing by the first “completion” GPT-3 scenarios, the initial release of OpenAI’s ChatGPT in 2021, and the most recent advancements in image and video generation.

The pace of adoption from the research sandboxes to the inclusion of these capabilities as new features for user facing applications have increased the level of general awareness, and the ability to create new and better use cases. However, there is still space for innovation and there are several capabilities rising up. Here is a selection of state-of-the-art generative AI applications among along multiple industries:

Traditional LLM-enabled scenarios

Often related to classic search engines and chatbot applications, these scenarios have evolved during the last years to incorporate new capabilities that improve and expand the user experience. For example:

Increased context window for document intelligence, which means the ability to directly inject the whole document content as part of the context (i.e. system message or meta-prompt), rather than connecting to a remote data lake that contains that or other document.

Memory-based scenarios for adaptive personalization, by including specific user information when that user is logged in. For example, adapting the answers from the LLM-engine by leveraging role, location, age, or other features.

Multimodal interactions that include multiple formats at the same time, such as images, videos, text, and voice. For example, cross modality translation that converts the video scene into a text output based on the audio script and the video scenes. Generative AI models are progressively incorporating native multimodal characteristics.

Agentic AI applications

You will read more about AI agents and multi agents in chapter 3, but these systems are the star of the new wave of advanced adoption patterns with generative AI. Compared to regular applications, the agentic behavior includes autonomous ways to achieve specific tasks with no supervision, trial-error mechanisms with self-improvements, more complex chains of thought, and connections with a variety of external tools in a dynamic way. It also comprehends a variety of multi-agentic patterns that includes two or more agents communicating in different ways to perform tasks together.

Role-based activities

The continuously increased maturity of generative AI technologies is enabling a new subset of role-based applications that increase the productivity and advancements for a series of industries and professionals. These applications leverage LLM and other models, agentic patterns, and specific user interfaces (UI) that facilitate activities such as:

Administrative work - Beyond the classic chat-based scenarios, generative AI models are used to process and analyze entire documents, including complex ones that combine text, images, and tables. Multimodal models can map the content to specific output formats and fields that can be then exported to other remote systems such as databases or ERP (Enterprise Resource Planning) tools.

Creative tasks - Content generation has come a very long way from the initial text and image generation models. Among others, there are several incipient areas that bring new possibilities to creative professionals: Long-form creation for text-based content that keeps the entire context all the way (e.g. generative a book with relevant content that connects all chapters), advanced music creation for lyrics and chords, image editing and remixing, voice replication and licensing, and the generation of dynamic and interactive 3D environments.

Code productivity - Software development was one of the first GPT-enabled cases, with the original OpenAI Codex model that initially powered tools such as Github Copilot. Since then, generative AI models have progressively increased their capabilities to analyze and improve software code, regardless of the programming language.

Generative cybersecurity - As an evolution of the classic cybersecurity activities, generative AI brings both new challenges and opportunities. Challenges, because generative AI models require additional safety and security measures (you will read more about this in chapter 6). And opportunities because generative AI technologies are also a powerful ally to spot threats and malefic patterns.

Scientific research and discovery - One of the key areas of development for generative AI technologies. Examples like protein or DNA predictions contribute to analyze and anticipate potential diseases, and to explore scientific data in ways never seen before.

Advanced user interfaces - Another work-in-progress area of development that will continue evolving during the next few years. This includes adaptive UI that change their shape and content depending on the user profile and context, and visual avatar-kind of systems that connect generative AI models, knowledge bases, and speech services. These avatars enable new applications such as career archetypes (e.g. for coaching and upskilling) and education systems with historical figures or tourism guides.

These are just a few examples of a highly evolving set of advanced applications powered by generative AI technologies. Now, let’s explore the different types of generative AI models, depending on the different levels of code access, with proprietary models offering the least flexibility and open-source models the most.

Types of Generative AI Models

There are a lot of discussions on the value of proprietary vs open source models. Also a great misunderstanding of what open sourcing means for generative AI models. You will now learn the key definitions, but the main point is that you can leverage all these three families of models by leveraging Azure AI Foundry’s model catalog, in addition to other model deployment options:

Proprietary (closed) models → In proprietary models, the model, weights, and often the code are fully controlled and owned by an organization or company. Users can typically interact with the model through APIs, but they cannot download, inspect, or modify the model’s internals. The model’s weights are hidden, and access is often restricted by commercial licenses, meaning users must pay to use it. This limits transparency, and any modifications or improvements to the model are tightly controlled by the company. This includes models such as OpenAI’s GPT series, Google’s Gemini, or Anthropic’s Claude among others.

Open weights models → These models refer to models where the weights (parameters) are publicly available, meaning users can download and run the model locally, but they may not have full access to the code or training data used to create it. Users can fine-tune or experiment with the weights, but there may be restrictions on redistribution or commercialization, depending on the license. While the weights allow for some flexibility in usage and adaptation, the model may still be subject to the creator’s license terms that limit how it can be deployed or modified. This category includes well known models such as DeepSeek, Meta’s Llama series, Mistral, or Falcon.

Open source models → Open source software (OSS) models offer full access to both the code and weights of the model, allowing anyone to inspect, modify, distribute, or collaborate on the project freely. These models are typically released under permissive open-source licenses such as MIT, Apache 2.0, or GPL, which encourage community contributions and transparency in AI development. Open-source models foster collaboration and innovation, making them ideal for research, development, and deployment without commercial restrictions. Users can tune, train, and redistribute the model as they see fit. Some of the most relevant examples include Microsoft Phi series, BLOOM, or Stable Diffusion.

The choice really depends on both technical considerations and model preference. The discussion on proprietary and open source models is almost philosophical (and you will understand the difference between them, along with the notion of “open weights” as an intermediate option, later in this chapter), but the choice impacts everything at company, architecture, and compliance levels.

Let’s now explore the key technical concepts that will enable you to understand the full model lifecycle, from the initial pre-training to advanced optimization phases, as well as the variety of techniques related to generative AI models, and the main architectures and use cases.

Technical Concepts behind Generative AI

Industry professionals often regard the “Attention Is All You Need” paper¹ as the starting point of the generative AI era. Published during the late 2010s - a pivotal period for the maturation and widespread adoption of deep learning and neural network-based techniques - this paper introduced the notion of Transformers that serve as the foundation for modern generative AI advancements. Concretely, Transformers is the foundational architecture behind most of the modern language models such Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT²), or T5.

The key advantage of Transformers is the high scalability for large data contexts and relationships, and the notion of self-attention mechanism that allows language models to focus only on relevant parts of the user input, compared to traditional language models that follow the regular sequence. For example, complex sentences where the verb and the subject are more important than the articles or any other word. That impacts not only the performance you can perceive from modern LLM-enabled systems, but also the computational efficiency based on parallelization techniques.

However, the main difference of GPT models (e.g., OpenAI’s, among others) when compared to the initial architecture presented in this paper is the fact that they use decoder-only architectures, rather than the encoder-decoder pattern presented by the authors. In general terms, encore-decoder patterns can handle more types of language tasks, while decoder-only focuses on next-word completion problems. However, for models such as GPT-4 or similar, this means a simplified pattern that relies more on the model training based on massive datasets that cover a very wide variety of tasks, and a prompt design that transforms a variety of tasks such as translation or summarization into next-work problems.

Precisely, that training process relies on several stages related to the generative AI model lifecycle, which includes the required steps from the initial model conception and parametrization, to the final customization and consumption. The specifics depend on the specific model version and provider (e.g., it is not the same training OpenAI’s GPT than Meta’s Llama models), but Figure 1-3 shows a general overview of some techniques and steps for each stage.

Figure 1-3. Generative AI techniques, from training to inference

Pre-training

The foundational phase where the model learns from large-scale, often unstructured, datasets to develop a fundamental understanding of language. This phase is crucial because it enables the model to grasp grammar, semantics, syntax, and contextual relationships before any fine-tuning is applied. Depending on the model and the specific training approach, the pre-training phase may include techniques such as:

Self-Supervised Learning (SSL) → A type of unsupervised learning where a model generates its own labels from the data rather than relying on human annotations. It allows language and multimodal models to learn representations autonomously from text, images, or others. Unlike supervised learning, where labeled data is required, SSL leverages inherent structures in the data. The model is trained to predict missing or occluded parts of its input (e.g., predicting masked words in a sentence).

From a Microsoft Azure perspective, this SSL process relies on Azure’s AI Infrastructure with enough computing power via Graphic Processing Units (GPU). The compute requirements depend mostly on the amount of information, training iterations, intended model size, etc.

Vectorization → It converts raw text into numerical representations (known as embeddings) that a model can process. Compared to regular one-hot encoding (which treats words as discrete, independent symbols), the goal of embeddings is to encode the semantics and relationships between words efficiently, so words, sentences, or documents are transformed into multi-dimensional vectors, where similar words are closer in vector space. These models are used for text similarity, semantic search, and retrieval augmented generation (RAG) use cases. There are different embeddings techniques, depending on their vectorization scope:

Word embeddings → Word2Vec, GloVe, and FastText generate static word representations. These embeddings are dense vector representations of words that encode both semantic meaning and relationships in a continuous, high-dimensional space. They capture contextual similarities, allowing models to understand relationships such as synonymy, analogy, and context awareness.

Sentence embeddings → Models like SBERT and Universal Sentence Encoder provide vectorized sentence representations, which are numerical embeddings that capture the semantic meaning of entire sentences rather than individual words. Unlike traditional word embeddings, which provide static representations for words, sentence embeddings take context into account and provide a more holistic understanding of the input text.

Contextual embeddings → These embeddings dynamically change based on surrounding words. These embeddings are generated using deep neural networks, particularly transformers-kind of algorithms, and can handle highly changing contexts, polysemy (words with different meanings), and higher dimensionality to include more features to represent information in vector spaces.

Compared to one-hot encoding, these varieties of embeddings have different advantages and considerations, as you can see in Table 1-1:

Table 1-1. Types of embedding techniques vs others

One-Hot Encoding
Word Embeddings
(Word2Vec, GloVe, FastText)
Sentence Embeddings
(SBERT, USE, InferSent)
Contextual Embeddings
(BERT, GPT, T5, etc.)

Context Awareness
❌ No
❌ No
✅ Some (captures sentence meaning)
✅✅ Fully Context-Aware

Sentence Representation
❌ No
❌ No
✅ Yes (fixed-length vector)
✅ Yes (words and sentences both get dynamic vectors)

Polysemy Handling
❌ No, same vector per word
❌ No, same vector per word
✅ Some (sentence meaning affects output)
✅✅ Excellent, dynamically changes meaning per sentence

Dimensionality
Very High
Medium
Medium-high
High

Semantic Meaning
❌ No (words are independent)
✅ Yes (words in similar contexts are closer)
✅ Yes (sentences with similar meaning are closer)
✅✅ Yes, meaning is adjusted based on surrounding words

Word Similarity
❌ No (all words are equidistant)
✅ Yes (words with similar usage are close)
✅ Indirectly (sentence similarity more relevant)
✅ Yes (word and sentence similarity both captured)

Sentence Similarity
❌ No
❌ No
✅ Yes
✅ Yes

Computational Cost
Very Low (simple lookup)
Low (pre-trained vectors are fast)
Medium (requires encoding full sentence)
High (transformer-based, requires GPUs/TPUs)

Best For
Simple text classification, rule-based NLP
Word similarity, keyword-based search
Document similarity, semantic retrieval
Chatbots, translation, summarization, question-answering

Example
Same vector always [0,0,0,1,0,...]
Same vector always [0.7,0.2,0.1,...]
Captures overall sentence meaning, but not per-word context
Different vectors based on context (“bank” in finance ≠ “bank” in nature)

Microsoft Azure supports multiple embeddings generation models from different providers, including Azure OpenAI, Cohere, or Meta. All of them available via Azure AI Foundry’s Model Catalog, along with all relevant HuggingFace embeddings models via “Deploy with Azure” option.

Expanding Multi-Modal → Traditional LLMs process only text, but multimodal models integrate multiple types of data, such as images, video, and audio. From a pre-training perspective, the model can learn joint representations across different modalities, allowing it to first understand and then generate diverse types of content. The main advantage is to do so in a native way, instead of combining multiple models during the integration and inference tasks. However, it requires a large-scale dataset covering multiple modalities, and aligning different data types effectively is computationally expensive.

Data augmentation → Data augmentation is a data-centric way to enhance the model performance by artificially expanding the training dataset through various transformations. This may include techniques such as:

Synthetic data → This approach involves creating artificial text datasets to enhance training, fine-tuning, and alignment. This approach helps overcome data scarcity, reduce biases, and generate domain-specific content efficiently. Same as regular datasets, it requires techniques such as data quality control and bias reinforcement. The Microsoft Phi series of small language models (SLM) are pioneering synthetic data generation for model training purposes, and you can leverage Azure AI Foundry for your own synthetic data generation purposes.

Other data augmentation techniques → For example, back-translation to translate a sentence into another language and back to create diverse paraphrases, word replacement with synonyms to maintain meaning while altering structure, or noise injection to add minor perturbations to text to improve robustness against typos.

Data generation and augmentation help reduce overfitting and make models more generalizable, and they are especially useful for minority languages where limited labeled data exists.

Distributed training → Training large-scale LLMs requires distributed computing across multiple GPUs to handle massive datasets efficiently. These GPUs are easily accessible via Microsoft Azure’s AI infrastructure, but but they rely on distributed training building blocks such as NVIDIA Megatron and Microsoft’s DeepSpeed on Azure, and FSDP (Fully Sharded Data Parallel) for large-scale training with Apache Spark capabilities via Microsoft Fabric or/and Azure Databricks. Distributed training can be implemented using different parallelization strategies, including data, model, or pipeline-level mechanisms:

Data parallelism → This means that the entire model is replicated across multiple GPUs, and each GPU processes a different mini-batch of data. During training, each replica computes gradients independently, which are then averaged and synchronized across GPUs before updating the model parameters. This approach is effective when the model fits within a single GPU’s memory but requires large-scale data processing, so it is more suitable for highly efficient SLMs.

Model parallelism → This requires splitting different parts of the model across GPUs to distribute computation. The model itself is split across multiple GPUs, with each GPU handles the different parts. This approach is useful when the model is too large to fit into a single GPU, as it is the case for most of the LLMs. Thad said, this introduces communication overhead.

Pipeline parallelism → In this case, training is done by splitting into sequential stages across multiple processors. The model is divided into different stages instead of parts, each assigned to a different GPU, forming a sequential pipeline. The batches are processed in stages, passing intermediate information across GPUs. This method is commonly used in transformer-based models where training large-scale networks requires optimizing both memory and computational efficiency.

Mixture of Experts (MoE) → In MoE, the model is based on multiple “expert” subnetworks, where each expert is specialized in learning different aspects of the data. During the pre-training, the model dynamically selects a subset of these experts for each input, allowing the model to leverage different experts’ specialized knowledge for different types of tasks or input features. This approach is designed to make the model more efficient by activating only a subset of the available experts at any given time, which reduces computational costs and memory usage compared to using all experts simultaneously.

Pre-training techniques are not widely known by most of the generative AI adopters, but this section contains a good selection of them, as well as the initial selection of Microsoft and Azure-related blocks that can support your model pre-training. Also, keep in mind that you can pre-train a model once, or in a recurrent based with continuous pre-training techniques.

Post-training

Once pre-training is complete, models require additional refinement through post-training techniques to improve efficiency, alignment, and performance, as well as adaptation to specific kinds of tasks. Among others:

Fine-tuning → Fine-tuning is the key part of the post-training stage, as it modifies pre-trained models for specific tasks by updating their weights using task-specific datasets. Actually, you need to know that model weights are the numerical parameters that a model learns during training, and they determine how the model processes inputs to make predictions. During the fine-tuning process, the goal is to take the pre-trained model and further train it on a smaller, task-specific dataset to adapt it for a particular application. This is great because fine-tuning adjusts model weights while retaining general knowledge, making it more efficient and flexible in terms of knowledge. There are different ways to perform fine-tuning:

Regular fine-tuning → In this case, all model parameters are updated, requiring extensive computational resources but leading to high task performance.

PEFT fine-tuning → Based on Parameter-Efficient Fine-Tuning (PEFT) techniques, technologies like Low-Rank Adaptation (LoRA) LoRA provide an optimized approach means that instead of modifying all model weights, injects small trainable matrices into transformer layers, significantly reducing memory usage. As a matter of fact, LoRA is a de facto standard created by Microsoft in 2021. This also includes QLoRA based on the quantization techniques we will explain in this section.

Fine-tuning includes the notion of model instruction. When a LLM is instructed, the model is specifically trained to understand and respond to conversational prompts, where the focus is on aligning its behavior with user instructions. This means the model is designed to follow user commands or queries more accurately, ensuring that its responses match the expected tone, format, or context of the conversation. The model can handle more complex instructions, provide precise answers, and adapt its responses based on the specific guidance given by the user.

From a chat capability perspective, a non-instructed LLM could still engage in conversation and generate responses, but it might not always interpret specific user requests properly, which could lead to the generation of more general or less targeted answers, or require the user to provide more context. However, most of the non-instructed LLMs are made available for posterior fine-tuning, and not to be directly used.

Reinforcement Learning from Human Feedback (RLHF) → RLHF is a method used to fine-tune machine learning models, particularly large language models (LLMs), by integrating human feedback into the training process. After the initial pre-training process, the model is deployed in a real-world environment where it interacts with humans. Human evaluators act as subject matter experts (SMEs) for specific domains of knowledge and industries, and provide feedback on the model’s outputs, rating them based on factors like accuracy, relevance, and ethical considerations. This feedback is used to create a reward model, which assigns numerical values to the outputs based on their quality. The model is then optimized using reinforcement learning (RL) techniques, for it to learn to adjust its responses and maximize these rewards over time. This process allows the model to align its behavior with human preferences, improving its performance and reducing undesirable outputs.

Pruning → Pruning is a post training technique used to reduce the size and complexity of a pre-trained model by removing less important or redundant parts, such as weights, neurons, or entire layers. The goal of pruning is to make the model more efficient, improving memory usage, computational speed, and reducing inference time while striving to maintain performance. This process is especially useful for deploying large models in resource-constrained environments, like mobile devices or edge computing, where hardware limitations make running full-scale models challenging.

Step 1 → The process of pruning starts with identifying which parts of the model are unnecessary or contribute minimally to its output. This can be done by examining the magnitude of weights, where small weights are considered less important, or using more advanced methods such as sparsity-based pruning, where certain connections between neurons are set to zero or removed entirely. In addition to pruning individual weights, entire neurons or even layers of the model can be pruned if they don’t significantly affect the model’s performance. For example, some parts of a transformer model, like attention heads, can be pruned without severely impacting accuracy.

Step 2 → After pruning, the model typically undergoes fine-tuning to recover any lost performance due to the removal of weights or neurons. This new fine-tuning process allows the model to adapt to its reduced structure while retaining most of its learning and generalization abilities. In this case, post pruning fine-tuning is essential because pruning can sometimes introduce a loss of accuracy, and the fine-tuning helps to restore the model’s effectiveness by re-optimizing the remaining parameters.

Steps 3+ → Pruning is often performed incrementally, meaning parts of the model are pruned step-by-step, with performance being evaluated and re-optimized (via iterative fine-tuning) after each step. This helps mitigate the risk of overly degrading the model’s capabilities. If too many important components are removed at once, the model could lose its ability to generalize, leading to a significant drop in performance, especially in complex tasks requiring deeper reasoning.

In general, LLM pruning optimizes LLMs by eliminating unnecessary components, which makes them more manageable for real-world applications and reduces the compute requirements for deployment.

Distillation → This technique trains a smaller model (e.g. SLM) to replicate a larger model’s (LLM) behavior, allowing efficient deployment for specific topics and tasks. Distillation helps more efficient “student” models from larger “teacher” models without sacrificing the performance. The goal of distillation is to reduce the size and computational requirements of a model while maintaining a similar level of accuracy, making it easier to deploy in resource-constrained environments with limited processing power. Two main challenges: Performance, when distillation does not work for highly specialized teacher models, and balance, concretely between model size compression and performance.

Quantization → This technique can serve both training and inference purposes, but quantization-aware training (QAT) is a technique that helps reduce both memory and compute requirements of generative AI models by representing the model’s weights and activations with lower precision, preparing the model during the training phase for efficient inference. In general terms, this process involves mapping model parameters (such as weights) to lower-bit representations, to make models more efficient for deployment on hardware with limited resources, like mobile devices, edge devices, and embedded systems, without significantly sacrificing performance. After the initial quantization, the model can be fine-tuned using the quantized parameters to compensate for any loss in accuracy caused by the reduced precision.

AI Red Teaming (AI Safety) → We will explore safety and security techniques on Azure later in Chapter 5, but depending on the model, all these techniques are applied in different moments of the model lifecycle, including post-training to ensure the model behave properly, and continuous red teaming activities to challenge and evaluate the model in different test scenarios.

Summarizing, the overall training process includes multiple steps and they are the base for obtaining something between good or superlative generative AI models. In Figure 1-4, you can see the overall pre and post-training process, that serves also as a absolute reference to the magnificent AI Engineering book, from the Chip Huyen, O’Reilly author and even more amazing AI professional:

Figure 1-4. Generative AI Model Training Workflow (source: Chip Huyen, AI Engineering book)

Inference

It refers to the process of using a generative AI model to generate outputs or predictions based on new, unseen input data. During inference, the model takes an input prompt and processes it to produce a response, leveraging the patterns and knowledge it has learned during its pre and post training phases. This process is distinct from end-to-end training, as the model does not update its parameters during inference, but instead simply applies the learned knowledge to make predictions.

Customization

After the initial training stages, the customization phase adapts the model for real-world applications. You will explore more details during Chapter 3, but some of the techniques for customized inference are:

Chunking → This technique means breaking long documents or datasets into smaller segments to improve information retrieval with smaller, more manageable segments, or “chunks,” before processing inference queries. Since LLMs have a limited context window (they can only process a certain number of tokens at a time) chunking is a way to handling long documents, improving retrieval efficiency in retrieval augmented generation (RAG) systems, and enhancing performance in tasks like summarization, question answering, and document search by complementing models with specific knowledge bases.

Hybrid search → This approach combines keyword-based (lexical) search and semantic, embeddings-based search to improve information retrieval. Keyword-based methods rely on exact word matches, making them precise but limited when queries use synonyms or paraphrasing, while semantic search uses vector embeddings, and retrieves results based on meaning rather than exact word matches, improving recall but sometimes returning less precise results. By combining both methods, hybrid search ensures more accurate and comprehensive retrieval, making it ideal for assistants, search engines, and document retrieval in retrieval augmented generation (RAG) systems.

Reranking → After retrieving an initial set of results, reranking applies a re-kanker model to reorder the results based on deep contextual relevance. Unlike traditional ranking methods that rely on precomputed scores, these re-rankers dynamically compare the query with each document, leading to more accurate rankings, especially for complex queries. While reranking significantly enhances precision in retrieval systems, it comes with higher computational costs, so it is typically used in a two-stage pipeline where hybrid search selects candidate documents first, and reranking optimizes their order.

Optimization

During the inference phase, there are various optimizations that help improve response quality, efficiency, and speed for real-word applications. They are all complementary, and the list includes:

Semantic caching → An industry term that refers to the practice of storing and reusing the results of previous generative AI model queries based on their underlying meaning or semantic content. Instead of recomputing the entire output for every request, the system caches results for queries with similar meanings or contexts, allowing for faster responses by retrieving cached outputs for similar inputs. This is particularly useful in scenarios where identical or similar prompts are frequently requested, reducing the computational load and improving efficiency. Semantic caching relies on understanding the content and context of the inputs, enabling intelligent reuse of past results without requiring full model inference for every new query. From an Azure perspective, building blocks like Azure API Management enable semantic caching at scale for multiple models and deployments.

Memory handling → Efficient memory handling in large language models (LLMs) combines several innovative approaches to optimize memory usage and improve model performance. These techniques dynamically allocate memory resources, manage large key-value caches, and reduce fragmentation to ensure that memory is used effectively during inference. By separating memory encoding and retrieval processes, some methods enable the model to store and access long-term contextual information, enhancing its ability to handle tasks involving long-form content. Additionally, models are equipped with the ability to explicitly store and retrieve knowledge in structured formats, improving their capacity for reasoning and handling complex, knowledge-intensive tasks. Temporal memory management further ensures that the model can adapt and update its knowledge over time, leading to better performance in tasks that require long-term understanding and context. Together, these strategies improve memory efficiency, scalability, and responsiveness across a wide range of applications.

Batch parallelism → With this technique, you can process multiple requests simultaneously for scalability. It involves processing multiple input prompts simultaneously in a single batch rather than sequentially. This approach allows generative AI models to generate multiple pieces of content (e.g., text or images) at once, leveraging the parallel processing power of hardware accelerators like GPUs. By processing inputs in parallel, batch parallelism reduces inference time, improves hardware utilization, and increases throughput, making it ideal for high-demand, large-scale applications. However, it may introduce some latency for individual requests and require more memory, especially for complex tasks or large batches.

Prompt optimization → You will learn more about all prompt engineering things in Chapter 3, but the idea of optimizing prompts imply carefully designing them to improve model responses, and compressing them to speed up LLM inference (e.g. with Microsoft LLMLingua that leverages small and specialized language models to identify and to remove non-essential tokens in prompts). It can also focus on analyzing and improving the provided prompt based on advanced techniques that retrofeed the original prompt request to obtain better results.

1-bit quantization - Same as the quantization concept you have seen during the post-training stage, there one specific type of quantization enables efficient and lossless inference of 1-bit quantized LLMs on CPUs instead of GPUs, significantly reducing memory usage and computational requirements. The key industry development for this technique is Microsoft BitNet, released in 2024 by the Microsoft Research team, and it is an authentic revolution for hyper-optimized inference in local devices.

Top-K sampling → It is a technique that injects randomness into the inference process of LLMs while ensuring that the model only selects tokens from the top K most probable ones. The idea is to control text generation randomness to obtain coherent responses. When generating text, an LLM predicts the next word (or token) based on its current context. Instead of always picking the token with the highest probability, top K sampling limits the pool of candidate tokens to the top K most likely options. From this restricted set, the model randomly selects a token, introducing variability into the generated text. This randomness helps the model avoid deterministic outputs while allowing diverse and creative responses.

Beam search optimization → Compared to top K sampling, beam search optimization is a more systematic, deterministic approach, aimed at optimizing the overall sequence of tokens rather than individual token choices. During inference, LLMs use this technique to explore multiple potential sequences of words at each step and select the sequence that maximizes the overall probability. Instead of selecting just the most likely next word, beam search keeps track of the top N sequences at each generation step, expanding each sequence with the most probable tokens. This allows the model to explore multiple “paths” in parallel, leading to more coherent and contextually accurate outputs, especially in complex or long-form generation tasks.

Container-level optimization → Tools like Microsoft’s KAITO (Kubernetes AI Toolchain Operator) offers significant advantages for Large Language Model (LLM) inference by automating GPU node provisioning, optimizing resource utilization, and reducing operational costs through the use of lower-end GPUs for distributed workloads. It supports popular open-source LLMs, simplifying their deployment and management within Azure Kubernetes Service (AKS) environments. Additionally, KAITO provides fine-grained control over model configurations, data security, and fine-tuning parameters, allowing for tailored deployments while maintaining privacy and security. These features collectively streamline LLM inference, making it more efficient, cost-effective, and flexible for various AI applications.

As you can see, this section represents the different complex training cycles and techniques, along with the considerations during the inference phase. A highly evolving area with new techniques and libraries, that starts to get some degree of standardization at industry level.

Now, before we finish the first chapter, let’s explore the connection with the main Microsoft Azure building blocks so you can get the 360 view before entering into other details in next chapters.

Mapping to Microsoft Azure

This book focuses mostly on Microsoft Azure-related topics that will help you implement new generative AI solutions. Because of the nature of the Microsoft ecosystem, Azure also converges with other platforms such as Microsoft Copilot (via Copilot Studio) and even Github Models, which leverages part of the capabilities of the Azure AI stack. Chapter 2 will be the place for deep dive, but here are the main considerations you need to keep in mind for now:

Regardless of how you decide to implement your generative AI solutions, you can see in Figure 1-5 different architectural building blocks you need to keep in mind while working with Microsoft Azure.

Figure 1-5. Mapping to Microsoft Azure services

Azure AI Foundry → This is the key building block for any AI or generative AI implementation with Microsoft Azure. As you will see in Chapter 2, the AI Foundry is the evolution of the former Azure AI and Azure ML Studios, and it combines an agnostic model catalog for LLMs, visual playgrounds for testing, endpoint management spaces, additional LLMOps and AI safety features (that you will see in Chapters 4 and 5).

Orchestration with Semantic Kernel → Chapter 3 will cover all customization patterns, including agentic AI with SDKs such as Semantic Kernel and Microsoft Autogen. This along with other elements like Azure AI Search that facilitate RAG scenarios between the models and data sources.

Data with Microsoft Fabric and vector stores → Also in Chapter 2, you will explore the different data storage options, including the onelake concept in Microsoft Fabric, and vector databases for embedding storage like Azure CosmosDB.

Governance with Microsoft Purview → There will be the entire Chapter 5 to discuss governance topics, but one of the typical building blocks in Azure architectures is Microsoft Purview, as a solution for data and AI governance, compliance, and security.

Development with Azure API Management and others → The development part of your Azure architectures may contain blocks such as the Azure API Management that serves as generative AI gateway, Azure App Service for webapps, cloud native technologies like Azure Kubernetes Service (AKS), and others.

Underlying AI Infrastructure → The AI infra part contains not only the physical CPUs and GPUs that support all training and inference activities, but also the ensemble of techniques you have learnt about in this chapter. This compute capacity is the key to implement generative AI at scale.

Integration with Microsoft Copilot → Last but not least, the integration between Microsoft Azure and Microsoft Copilot is relevant as Azure brings extended capabilities that combine with the regular Microsoft 365 universe of Copilot, 365 data, and the Microsoft Graph API for Office, Sharepoint, Teams, OneDrive, and others. This integration happens via Microsoft Copilot Studio, which is an evolution of the former Power Virtual Agents.

Finally, here are some key aspects that you need to keep in mind to know how to leverage Microsoft Azure for generative AI projects include:

Types of model and APIs → Of course, besides the mix of providers, models, and versions, there are different types of regular completion, chat, reasoning, image, and other models. This is highly evolving as model providers release new capabilities. Same for the type of APIs, with additional capabilities such as Assistants and Responses APIs, or features like Function Calling or Structured Outputs.

Levels of service → Besides considerations such as the geographic region in which you want your models to be deployed, there are also options to choose how you want to access the model capabilities. For example, you may want to simplify everything and access serverless APIs via model-as-a-service (MaaS) options in Azure AI Foundry. Or maybe you prefer to use managed compute (also known as model-as-a-platform or MaaP) to choose your own virtual machines and deploy your models.

Purchasing options → Last but not least, you will need to know how to purchase and consume AI capacity on Azure. Basically, you can choose between regular pay-as-you-go (PAYG) options, or reserved instances (PTU or provisioned throughput units) to obtain a stable capacity with reduced latencies and a monthly flat fee. You also have intermediate options such as the Batch API to obtain better prices and services, for asynchronous tasks that don’t require immediate answers.

This is just part of the concepts you will explore in the next chapters, but a good starting point for you to understand the principles of implementing Generative AI on Microsoft Azure.

Conclusion

Summarizing, the generative AI era has entered a new stage of widespread adoption, but also of standardization in terms of model lifecycle, training techniques, impact on applications and user interfaces.

Since the first years of generative AI innovations, platforms like Microsoft Azure have evolved at an accelerated pace to incorporate product features beyond the generative AI models, which are now easily available for testing and API consumption via visual model catalogs. The next chapter is your deep dive for all generative AI resources on Microsoft Azure.

¹ [Paper] Attention is All You Need, Google - https://arxiv.org/abs/1706.03762

² [Paper] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Google - https://arxiv.org/abs/1810.04805

Chapter 2. Advanced AI Development in Azure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the second chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

For a time, model size and performance were seen as the primary obstacles to deploying generative AI solutions at scale. Yet over the past two years it has become clear that the real challenge lies elsewhere: in assembling the surrounding infrastructure needed to move from experimentation to production. Orchestration, security, data pipelines, model integration, evaluation and observability are where most organi zations struggle. While Microsoft has been investing heavily in building a comprehensive, end-to-end AI stack to address these gaps, this chapter will also explore the broader ecosystem of tools and frameworks that complement Azure’s capabilities.

Our goal is to give you a practical, in-depth understanding of how to build advanced generative AI applications by combining Azure’s native strengths with the flexibility of GitHub Copilot, Azure Databricks, Semantic Kernel and other frameworks that support multi-agent orchestration and operationalization.

The focus of this chapter is on practical architecture patterns and enterprise-grade techniques for AI professionals who already understand the fundamentals of large language models (LLMs) and are ready to build production-ready applications. We’ll walk through the critical components required to move from experimentation to deployment on Azure, including model hosting, vector stores, orchestration frameworks, low-code integration tools and web front-ends. Along the way, we’ll explore real-world design choices, highlight trade-offs (such as cost vs. performance) and provide actionable guidance for assembling scalable, secure and maintainable GenAI solutions. To reinforce learning, we include illustrative scenarios and a quiz at the end of the chapter to test your understanding.

Overview of the Advanced AI Ecosystem on Azure

Microsoft Azure provides a comprehensive ecosystem for building generative AI solutions, spanning from fully managed AI services to flexible infrastructure for custom models, multi-agent frameworks or Responsible AI tools. A central theme of Azure’s approach is offering full choice and integration: developers can choose frontier models like o3 via Azure OpenAI Service or open-source models like Llama 4 via Azure AI infrastructure and integrate them within one single application. Azure’s tooling lets you discover and evaluate foundation, open, task and industry models for your app and integrate them into your code with a unified API. In practice, this means you can mix and match models from Microsoft, OpenAI, Hugging Face, Meta, and others as needed.

Azure’s AI ecosystem goes beyond the models, though. It’s an end-to-end stack. For example, the recently introduced Azure AI Foundry (formerly Azure AI Studio) serves as a one-stop hub for building AI applications and agents. It provides a unified web portal, SDK and APIs to streamline the AI development lifecycle, from securing data access to deploying and monitoring models. Additionally, Azure integrates with popular developer tools: you can work with Azure AI directly from Visual Studio Code, GitHub and also Microsoft’s Copilot tools. This means the AI ecosystem extends into the workflows developers already use, such as coding with GitHub Copilot or building low-code solutions with the new Copilot Studio (which we’ll cover by the end of the chapter). Another pillar of Azure’s AI ecosystem is support for enterprise requirements like security, compliance and scalability. Azure’s platform emphasizes Trustworthy AI with built-in content filtering and safety controls and provides enterprise-grade governance for AI apps.

There’s also seamless integration with Azure’s data services—for example, Azure AI Search for indexing, storing, and querying enterprise knowledge, or Microsoft Fabric for unified data engineering, lakehouse and analytics workloads across OneLake. These services pair tightly with Azure’s infrastructure, from cloud-based GPU VMs to hybrid deployment options like Azure Stack HCI. In short, Azure’s advanced AI ecosystem spans “every layer of the Copilot stack, from silicon to software,” as Satya Nadella has put it, offering everything from cutting-edge AI hardware, to orchestration frameworks, to end-user applications like Microsoft 365 Copilot.

Finally, Azure’s ecosystem acknowledges that real-world AI solutions are often multi-cloud and multi-tool. Microsoft actively partners with other leaders in the space: Snowflake integrates with Azure OpenAI and Copilot, NVIDIA provides optimized LLM microservices and inference toolkits and GitHub (a Microsoft company) delivers powerful AI developer tools that now include models from Cohere and others. For example, a financial services company might use GitHub Copilot for code development with models from OpenAI and Cohere, train a fraud detection model using NVIDIA’s NIM containers on Azure Machine Learning and store customer interaction data in Snowflake connected to Azure Synapse for analytics. The final app is deployed on Azure Kubernetes Service, with CI/CD powered by GitHub Actions. We will explore how these partnerships enhance Azure’s capabilities and how you can leverage them to accelerate solution development.

The “Others” in the title refers to these external but deeply-integrated tools that complement Azure’s native offerings. An advanced practitioner should be aware of how to leverage Azure’s strengths together with third-party AI tools to build the best solution possible.

In the rest of this chapter, we’ll deep-dive into specific components and patterns: Azure AI Foundry, the GitHub Copilot ecosystem, related data and AI platforms, deployment strategies (MaaS vs. MaaP), using a diversity of models (open-source and proprietary), data architectures for LLM applications, orchestration frameworks for complex workflows and front-end tools to deliver generative AI to users.

Deep Dive on AI Foundry: Models, Workflows and Safety

The cornerstone of Microsoft’s advanced AI stack is Azure AI Foundry, an integrated platform for building, customizing and managing AI applications and agents at scale. AI Foundry can be thought of as the “mission control” for generative AI on Azure, bringing together model discovery, deployment options, safety tools and lifecycle management under one roof.

Key Capabilities of AI Foundry

Let’s break down AI Foundry’s key capabilities:

Model catalog: Every AI application begins with model selection and AI Foundry offers more than 1800 leading models across a wide range of tasks and modalities. The model catalog houses frontier and open-source models from various providers (Microsoft, OpenAI, Hugging Face, Meta, Mistral, and others)​. These include large language models for text generation, but also models for vision, speech, translation and domain-specific tasks, like GPT-4 Turbo with Vision and Whisper (audio)​. A developer can browse models by category (e.g. chat completion, text embedding, image generation) and by provider. Most importantly, AI Foundry lets you compare models on standard benchmarks or your own test data before choosing one.

For example, if you’re building a multilingual customer support assistant, you might compare OpenAI’s GPT-4o -a highly efficient multimodal model- with Mixtral-8x22B, a recent sparse Mixture-of-Experts model designed for fast, low-latency inference at scale. Using Azure AI Foundry’s built-in evaluation SDK, you could test both on sample user queries from your support logs and measure task adherence, intent resolution, latency and cost-per-token to make an informed deployment decision.

Interactive playgrounds: AI Foundry includes interactive playgrounds that allow developers to quickly prototype and experiment with models and AI services across text, vision and speech. These playgrounds, available for both pre-trained foundation models and services like speech-to-text and translation, offer a rich, GUI-based environment for testing prompts, system instructions, temperature settings and tool use. A recent update introduced tighter integration with the Azure AI Agent Service and Semantic Kernel, enabling developers to prototype agent behaviors and tool-calling flows directly within the playground. Foundry now also supports playgrounds tailored for multi-agent scenarios, making it easier to simulate task routing and reasoning chains before deployment. This allows for validating model behavior, testing prompt engineering strategies and iterating quickly across model choices and parameters.

Customization and Fine-Tuning: While the model catalog provides powerful pre-trained models, you will often want to further tailor a model to your data or specific use case. Foundry supports model fine-tuning for certain models (for instance, fine-tuning GPT-4o on your dataset, or fine-tuning an open-source model like Llama 4 Maverick for your domain).

AI Agent Service: Foundry also provides an “agent toolchain” for building complex AI agents by composing models, knowledge sources​ and actions. We’ll touch more on agents later (especially in Copilot Studio and orchestration frameworks), but note here that Foundry allows creation of prompt flows –multi-step prompt workflows– and chaining model calls with tool usage, which is key for agent development.

Safety and Monitoring: With great power (of generative models) comes great responsibility, and Azure AI Foundry builds in Trustworthy AI measures from the start. Beyond basic content filtering and safety system prompts, Foundry now includes AI-assisted risk and safety evaluations to assess outputs across categories such as hate speech, sexual content, violence and jailbreak attempts. These evaluations not only label risks (e.g., Low, Medium, High) but also provide grounded justifications for each case, helping teams make more informed decisions before deployment.

A particularly impactful recent addition is Prompt Shield, designed to detect and block prompt injection attacks (including indirect ones) and groundedness detection, which flags hallucinations and helps ensure model responses are based on trustworthy sources. You can configure safety system messages (structured instructions that guide model behavior) to align outputs with enterprise policies.

Foundry also supports red teaming workflows: for instance, you can test models against predefined “unsafe” prompts and automatically track violations. You define your evaluation criteria and Foundry runs systematic tests to verify that the model or agent behaves as intended.

Once a model or agent is deployed, Foundry integrates deeply with Azure Monitor and Application Insights for real-time observability. Dashboards allow teams to track latency, token consumption, safety trigger frequency and usage patterns. New metrics (like groundedness score, filtered content rate and tool usage audit logs) help detect drift, abuse and performance issues early, enabling fast iteration. This makes responsible AI not just a development concern but a full lifecycle practice.

Management and governance: Azure AI Foundry integrates tightly with Azure’s resource management stack and now includes advanced project-level governance and collaboration capabilities. Each Foundry hub, still backed by an Azure Machine Learning workspace, can host multiple isolated projects, each with its own set of deployments, resources and role-based access controls (RBAC). Organizations can assign scoped permissions (e.g., reader, contributor, project Admin) to users or groups, ensuring only authorized individuals can deploy, modify or access models and data. The Foundry Management Center provides a centralized dashboard to track usage, quotas and policy compliance across all projects and hubs. Audit logs record key actions such as deployments, evaluations, and endpoint usage, providing transparency and accountability for enterprise governance.

Admins can enforce model-level restrictions (e.g., disallowing certain open models for compliance) and configure private networking with VNet injection to ensure workloads remain within secure, enterprise-controlled environments. Foundry also provisions and integrates essential Azure-native services automatically—including Azure Key Vault, Azure Storage, Azure Container Registry, and Azure AI Search—so teams can begin building production-ready AI solutions with minimal setup overhead.

All these capabilities turn Azure AI Foundry into a one-stop development environment for AI, where you not only pick models but also develop how they interact and respond.

Example: Deploying and Testing a Model in Azure AI Foundry.

To make all these capabilities and workflows concrete, let’s walk through a simplified scenario of deploying and testing a model via Azure AI Foundry.

Select a Model from the Catalog → Imagine you need a model for text summarization. In the Azure AI Foundry portal, you navigate to the Model Catalog and filter by the task “Summarization.” You will see options such as Azure OpenAI GPT-4 Turbo, known for its high accuracy in summarization tasks, and Meta’s Llama 4 Scout-17B-16E-Instruct, a recent addition designed for summarization, personalization, and reasoning tasks. The catalog shows benchmark metrics and allows side-by-side comparisons. You decide to use GPT-4-Turbo for its superior accuracy on summary tasks (noting that it’s a more expensive model​, but you require high quality).

Choose Deployment Mode (MaaS vs MaaP) → Depending on the model selected, you have the choice to deploy it as Model-as-a-Service (MaaS) or Model-as-a-Platform (MaaP). For Azure OpenAI models like GPT-4o, the only way is MaaS, in which the model is hosted in Azure’s multi-tenant pool and you get an endpoint to call it. This is essentially a click-through in Foundry: you create a deployment, which reserves capacity for you (possibly using PTUs if you need guaranteed throughput, which we discuss later). Alternatively, if this were an open-source model that you wanted to self-host, Foundry can deploy it to your own infrastructure (MaaP). For example, deploying a Mistral 7B model might spin up a GPU VM in your subscription and run the model there.

Configure Deployment Settings → In the deployment form, you’ll define a unique name for your endpoint (e.g., summarizer-gpt4), select the model version (e.g., GPT-4 Turbo with 128k or 32k context window), and specify your scaling requirements. If you anticipate high demand or need consistent performance, you can reserve Provisioned Throughput Units (PTUs) — for example, 10 PTUs — to ensure dedicated capacity and minimize latency. For Model-as-a-Service (MaaS) deployments, Foundry takes care of the underlying infrastructure and GPU allocation automatically.

If you’re deploying via Model-as-a-Platform (MaaP), you’ll instead select a specific compute target in your own Azure subscription. This could be an Azure ML cluster, or a VM SKU like Standard_ND96amsr_A100_v4 (8xA100) or Standard_NC24ads_A100_v4 (1xA100) depending on your performance and cost needs. Foundry simplifies this process with dropdown selectors for compute options, abstracting the complexity while still giving full control over infrastructure and scaling.

Deploy and Test in Playground → Once your deployment settings are finalized, click Deploy and Azure AI Foundry will handle the orchestration. For MaaS deployments like GPT-4 Turbo, Foundry automatically provisions the model behind a secured Azure OpenAI endpoint and exposes it through the unified Foundry interface. When the deployment status turns to “Deployed,” you can immediately validate it using Foundry’s interactive playgrounds.

Open the Chat Playground, select your summarizer-gpt4 endpoint and enter a sample input. For instance, a paragraph of text with the system prompt: “Summarize the content above in one sentence.” The model’s response appears in real time, confirming that your deployment is live and functioning. The playground lets you fine-tune parameters like temperature, top-p, and max tokens on the fly, making it ideal for iterating on prompt design and output quality before you integrate the model into your application code.

Integrate via SDK or API: Satisfied with the tests, you now integrate this model into your application. Azure AI Foundry provides a unified SDK (in Python, and other languages in preview) to call any deployed model or agent. In your Python code, you might use the Foundry SDK like:

from azure.ai.foundry import AIClient
from azure.identity import DefaultAzureCredential

# Authenticate using Azure identity (managed identity, CLI login, or environment vars)
credential = DefaultAzureCredential()
client = AIClient(credential=credential)

# Generate text using your deployed model
response = client.generate_text(
    deployment_name="summarizer-gpt4",
    prompt="Summarize the following paragraph in one sentence: [...]"
)

print(response.content)

This single client can invoke OpenAI models, Azure AI services, or open models alike – Foundry handles routing to the correct backend. Under the hood, it’s calling Azure OpenAI’s API for GPT4-Turbo. The key point is that after deployment, using the model is uniform and secured (it respects the Azure RBAC permissions of your Foundry project, etc.). You could also use standard REST calls with the endpoint URL and API key provided by Foundry if you prefer.

Monitoring and Iteration: As the app goes live, Foundry’s dashboard shows usage metrics. Let’s say you see the latency is higher than expected at peak times – you might scale up from 10 PTUs to 20 PTUs for the GPT4-Turbo deployment (this can often be done on the fly, ensuring you have more throughput capacity)​​. Or, if cost is a concern, you might try fine-tuning a smaller model in Foundry (maybe GPT-4o mini or an open model) to see if it can achieve acceptable performance with less cost. Throughout, Foundry’s integrated tools for evaluation can be used to periodically run quality tests on the summarization (e.g., ensure it’s not missing key info or check that it doesn’t produce prohibited content). If issues are found, you could update your prompts or add more few-shot examples via the Foundry interface, and redeploy improved versions of your agent.

This example illustrates how Azure AI Foundry simplifies advanced AI workflows: you find the right model, deploy it through a managed process (either fully-managed or on your infrastructure), test it in a sandbox and then integrate it in your application via an API with with enterprise-grade reliability. Foundry, in combination with Azure’s other services, abstracts much of the heavy lifting while still giving flexibility in model choice and deployment strategy.

Callout: Cost vs. Performance Trade-off in Model Deployment

One of the recurring themes in building AI applications is balancing quality/performance and cost. Larger models like GPT-4o often yield the best performance on complex tasks, but they are also more cost-intensive to run. Azure AI Foundry enables an experimentation approach to this trade-off. For instance, you can deploy a big model (GPT-4o) and a smaller model (say, Mistral 7B) and compare their outputs side by side on your use case. You might find that the smaller model, perhaps with some fine-tuning, gives “good enough” results at a fraction of the cost.

In some cases, free or compact models, like Microsoft’s Phi-4, have exhibited excellent performance on common tasks, narrowing the gap with larger proprietary models in recent benchmarks. These smaller models are especially valuable when latency, cost or deployment footprint are critical. However, using them may require additional development effort, such as more extensive prompt engineering or retrieval augmentation, to achieve comparable results. In terms of deployment, Provisioned Throughput Units (PTUs) also factor into cost: reserving capacity via PTUs can reduce the cost per call for high-volume usage, while pay-as-you-go might be more economical for spiky or low-volume traffic.

Always analyze your use case: if maximum quality is mission-critical (e.g. legal document summarization with zero tolerance for errors), a top-tier model like GPT-4o may be justified. But if cost efficiency or scalability is paramount (e.g. a chatbot serving millions with some tolerance for minor inaccuracies), consider smaller, optimized models like Phi-4 or open-source alternatives. Azure’s tooling encourages experimentation, allowing developers to compare multiple models under real-world conditions and make informed trade-offs between performance, cost and complexity.

Model Diversity and Multi-Provider Strategies

One hallmark of an advanced AI solution is the use of the right model for the task. From an internal research run by Microsoft in 2024, we learned that 97% of GenAI applications use 3 or more model providers in production. Microsoft Azure’s AI strategy actively supports a multi-model, multi-provider approach. In this section we’ll discuss why model diversity matters and how to implement a multi-provider strategy.

Why Model Diversity? Not all LLMs are equal. They differ in size, training data, skills, strengths/weaknesses, context length, latency and licensing. A specific application might benefit from using multiple models in different roles. For example:

A large, powerful model (like GPT-4o) might handle complex reasoning or fluent paragraph generation.

A smaller, faster model (like Phi-4) might be used for quick tasks like classification or keyword extraction.

A specialized model (like a code model, e.g., Codestral or a biomedical text model) might outperform a general model on certain domains.

An open-source model (Llama 4 Maverick) can be deployed on-prem or in custom ways for data locality, whereas a closed API might not.

Furthermore, relying on a single provider could be risky (provider downtime, price changes, or model changes). Enterprises may prefer to have flexibility and negotiation leverage by maintaining integration with multiple AI providers.

Azure AI Model Catalog – Many Models, One Platform:

As we saw, Azure AI Foundry’s model catalog already includes models from Microsoft, OpenAI, Meta, Hugging Face, Mistral, Databricks, Stability AI and more. You can think of this as Azure acknowledging that no single model is best for everything and that the value lies in the applications built on top of them.

In practice, you can use the Foundry interface to filter models by provider, by task, industry and even by fine-tuning availability. For instance, you might filter to only see “Meta” models if you are interested in Llama variants, or only “OpenAI” models if you want their latest. Tasks could be “chat” vs “embeddings” vs “image generation” vs “translation”, etc. This catalog makes it easier to adopt a multi-model strategy because it provides users with model leaderboards by different categories (quality, cost, throughput) and allows the users to put the different model side by side for comparison.

As an advanced user, you might want to run internal benchmarks and Foundry allows you to evaluate models on either sample or your own datasets. If you have a set of Q&A pairs from your specific domain, you could quickly test GPT-4o, Llama-4, and Mistral-Large on those and see how they rank (perhaps GPT-4o gets 90% accuracy, Mistral-Large 86%, etc.). As noted earlier, model performance is a moving target since the LLM performance picture changes on an almost daily basis. So building with model diversity in mind “future-proofs” your solution to some extent. If a new model comes out that’s better or cheaper, you want to be able to swap it in.

Model Providers:

AI Foundry provides models from a wide range of model builders. While OpenAI is a key provider, Foundry also supports alternative and complementary models from various industry players:

Hugging Face Hub: More than 1500 open models from Llama, Falcon, Bloom, etc. can be accessed via APIs or downloaded. With Azure’s one-click deployment integration, these models can be added to your workflows just like proprietary ones, enabling rapid experimentation and customization.

Microsoft: In addition to its partnership with OpenAI, Microsoft is also a model provider. The Phi family of compact, high-performance small language models is optimized for client-side and edge scenarios. Microsoft has also released TAMGEN, a healthcare-specific model trained on anonymized medical transcripts in partnership with the NHS. TAMGEN is designed to support safe, accurate medical summarization and reasoning. In finance, Microsoft has released models for extracting and interpreting financial reporting data, such as earnings statements and SEC filings. These domain-specific models can be called directly within AI Foundry workflows and Copilot-based deployments.

Cohere: Users can call from AI Foundry Cohere’s command, embed and classify models for various use cases, including RAG, semantic search and enterprise document understanding.

NVIDIA: Their foundation and domain-specific models, such as Nemotron and BioNeMo, are increasingly supported within Azure’s AI and HPC infrastructure. While not all of them are yet fully integrated natively into AI Foundry, you can call NVIDIA models using endpoints or containers orchestrated via Azure Kubernetes Services (AKS), Azure ML or AI Foundry connectors.

Stability AI: their latest models, including Stable Diffusion 3.5 Large (SD3.5 Large), Stable Image Ultra and Stable Image Core are now natively available in the Azure AI Foundry model catalog. This provides everyone access to high-quality, professional-grade image generation for creative and industrial applications, from marketing content to design prototyping.

Custom models: Any fine-tuned or proprietary model you create effectively becomes a provider within your architecture. AI Foundry supports custom model registration, container-based deployment and runtime scaling, so you can integrate your own model checkpoints just like any other hosted model.

Deployment Patterns on Azure: MaaS, MaaP, PaaS, IaaS, and Scaling with PTUs

Once you have chosen the model/s that better fit your use case, it’s time to choose the right deployment pattern. This will impact cost, performance, scalability and the amount of work you’ll need to manage the infrastructure in the long run. Let’s first decode the acronyms and then examine how they apply with Azure’s offerings:

MaaS (Model-as-a-Service): In this mode, the model is offered as a fully managed service you just make API calls to. You don’t have to worry about the underlying infrastructure it runs on.

All the Azure OpenAI models are a prime example of MaaS: you get access to o3, GPT-4o-audio, etc., via an endpoint and you pay for usage as you go. Azure handles in the background the model hosting on its GPUs. All Azure OpenAI models are MaaS​ and recently Azure even started offering some open-source models (like Mistral or Llama) in a MaaS style. The advantage of MaaS is simplicity and often immediate scalability, since you can start small and let Azure scale the service behind the scenes when needed. The downside can be less control (you might not be able to run custom code with the model or ensure it’s always on a certain hardware spec) and potentially higher cost per instance if you have constant high usage. In this last scenario is where provisioned throughput can help, more on it in the next section.

MaaP (Model-as-a-Platform): In this mode, you’re essentially treating the model like an application you host on the cloud, deploying it on infrastructure of your choice, only dedicated to you. Azure AI Foundry’s support for deploying open models to your own subscription is an example of MaaP​. When you use MaaP, you deploy the model into any VM that are running in your subscription​. For instance, you might deploy a Llama-4-Scout-17B model on a Standard_ND96isr_H100_v5 (with eight H100 GPU). Here, the model runs totally isolated for you, no other customers share that VM and you have more control. You could potentially SSH into the VM or add custom libraries, etc.

MaaP on Azure often leverages Azure Machine Learning endpoints or Azure Kubernetes Service under the hood. One interesting option that Azure offers if you have no GPU quota is to provide you with a shared GPU for MaaP deployments (like a time-limited allocation, 24-168 hours) to try out a model. This allows you to experiment with practically all models despite not having the right GPU quota.

In general, use MaaP when you need a model that Azure doesn’t offer as a service or when you need custom modifications to the model runtime. The trade-off is you are responsible for that instance: if it goes down you might need to redeploy and you have to manage scaling, though Azure can help with automation.

PaaS (Platform-as-a-Service): PaaS is a broad category in Azure, typically referring to managed services for running applications (like Azure App Service, Azure Functions, Azure Container Apps, etc.). In the context of generative AI, PaaS might not directly host a large model (since those need GPUs), but you might use PaaS to host your application logic or API that calls the model. For example, you could have an Azure Functions app that calls an Azure OpenAI (MaaS) endpoint and processes the responses. Or use Azure App Service to host a Flask web app that users can leverage to interact with the model. There’s also Azure Machine Learning endpoints which are kind of PaaS for ML: you give it a model (could be a custom one) and it provides a hosted endpoint with autoscaling, etc. That blurs the line between MaaP and PaaS.

Think of it this way: if you want to deploy code with your model, you’re moving into PaaS territory (or even IaaS). For instance, deploying a custom container that has both an LLM and some pre/post-processing code to Azure Container Instances (ACI) or AKS (Azure’s Kubernetes Service) is using PaaS capabilities of Azure (ACI/AKS) to host your model. PaaS generally means you don’t manage VMs directly, but you manage an environment (an app service, a function, a container orchestrator).

IaaS (Infrastructure-as-a-Service): This is the lowest level, you manage the virtual machines or hardware and install everything yourself. On Azure, this would be deploying a VM with GPUs (like a NCas_T4 or NDm A100, etc.), then manually installing frameworks and running your model. Or even setting up a VM scale set or a Kubernetes cluster by yourself. IaaS gives maximum control: you choose drivers, libraries, maybe use custom hardware configurations (like multiple GPUs per node, special networking). The primary cost driver is on operations: you have to handle updates, scaling, load balancing, etc.

Most enterprise solutions try to minimize pure IaaS for AI because it’s heavy to maintain, unless they have very custom needs or are integrating with on-premises. A scenario for IaaS could be when running a specialized research workload that Azure services don’t support (e.g., training a model with custom CUDA kernels). Another case is edge deployment: if you want to run an AI model in an Azure Stack Edge box (physical appliance) or on customer’s on-prem servers with Azure Arc, you might end up treating it like IaaS (though Azure is extending PaaS patterns to Arc-enabled Kubernetes as well).

In practice, Azure often blurs the traditional boundaries between infrastructure and platform services. For example, Azure Machine Learning and Azure Batch AI provision and manage VMs behind the scenes, delivering the flexibility of IaaS with the simplicity and automation of a PaaS experience. Similarly, while Azure OpenAI runs on Azure Kubernetes Service (AKS) under the hood, it’s exposed to users as a fully managed SaaS/MaaS offering, abstracting away all the underlying orchestration and infrastructure complexity.

Building Agents in Azure

Building AI-powered agents has become much easier thanks to emerging frameworks that handle the heavy lifting of orchestration, planning, memory and tool integration. Rather than writing complex prompt sequences and logic from scratch, developers can leverage different frameworks (from no-code to full-code) to create agents that reason, act and even collaborate with other agents.

Inside Azure, there are different options for creating an agentic solution, starting for the Azure Agent Service in AI Foundry to Copilot Studio or to Microsoft’s Semantic Kernel. In chapter 5 we’ll pay special attention to Semantic Kernel’s agent tooling, which reached General Availability in April 2025, and illustrate practical examples of what these agent frameworks can do.

Microsoft Copilot and Copilot Studio

One of the most significant shifts in how Microsoft is delivering AI to end-users is through the concept of Copilots, AI assistants that are woven into the fabric of products like Office 365, Dynamics 365, GitHub and even Windows. In this section, we’ll explore Microsoft 365 Copilot (often just called “Microsoft Copilot”) and the new Copilot Studio that allows organizations to customize, build and even commercialize their own copilots. We will cover the high-level architecture of these copilots and technical integration patterns – i.e., how you can extend Copilot with your data and how Copilot Studio enables building custom AI agents integrated with Microsoft’s ecosystem.

Microsoft 365 Copilot: The Era of Embedded AI Assistants

Microsoft 365 Copilot is an umbrella term for AI features integrated into the Office suite and other Microsoft 365 services. If you’ve used or it seen the demos, Copilot can draft an email for you in Outlook, summarize a Teams meeting, help you create a PowerPoint deck from a Word document outline, etc. It’s essentially GPT-4 (and beyond) working behind the scenes on your behalf, with access to your work context (your emails, files, calendar, Teams chats, etc., all governed by the Microsoft Graph API with security) to provide very contextual and actionable assistance.

Copilot Studio: Build Your Own Copilot

While Microsoft 365 Copilot brings AI into Microsoft’s first-party applications, Copilot Studio is a platform that enables developers and power-users to create custom AI assistants (copilots) from scratch or by customizing the base Copilot. Launched in preview in late 2023, Copilot Studio provides a low-code environment to design conversational AI agents that can connect to various data and services.

Think of Copilot Studio as a fusion of Power Virtual Agents (chatbot builder), Logic Apps/Power Automate (workflow) and the new generative AI capabilities. It allows you to define an agent’s knowledge, conversational behavior and integrate it with “1200+ data connectors” to take actions.

Key components of Copilot Studio:

Topics and Prompt Flow: You can define managed topics or conversational flows. For example, a topic might be “Order Status Inquiry” for a customer support copilot. Within that, you can design a dialog (the bot asks for order number, then responds with status). Copilot Studio likely includes a dialogue management system (similar to Bot Framework’s adaptive dialogs or PVA topics) but augmented with generative AI, meaning the bot can handle unexpected turns by relying on the LLM for open-ended queries, yet stick to structured flows for critical process steps​.

Generative Actions (Plugins): This is one of the most exciting features – Copilot Studio allows you to integrate plugins/actions that the Copilot can execute, using that library of 1200+ connectors (which cover things like databases, CRM systems, SAP, ServiceNow, etc.). For instance, you might give your copilot access to a “Create Ticket in ServiceNow” action or “Query SAP for inventory” action. When a user asks something that requires those, the copilot can invoke the action (the system uses the connector behind the scenes). This is analogous to enabling a tool in LangChain or a function in OpenAI, but done in a low-code fashion: you select from pre-built connectors and perhaps provide parameters.

Orchestration with LLM + Dialog Manager: Under the hood, Copilot Studio’s agent uses a dialog manager for managed topics​. That means you can explicitly orchestrate certain conversations (ensuring compliance or business rules in those flows) and for everything else, the LLM can handle it more freely. It’s the best of both worlds: you maintain control where needed and let generative AI improvise elsewhere. The orchestrator decides when to use a certain knowledge source or tool. For example, if the user asks a question that matches a “managed topic” (like a known process), the copilot follows that script; if it’s an off-script query but still within the bot’s capability (like a general question answerable from a knowledge base), it lets the LLM handle it with knowledge retrieval.

Knowledge integration: You can feed organization-specific knowledge into Copilot Studio. This might be via connecting SharePoint sites, files, or uploading documents that the copilot should use. There’s mention of using Azure OpenAI on your data, bringing your own custom language models and knowledge bases​. For example, you could connect an internal FAQ document as a data source. The copilot will then use that content to answer relevant questions (likely via an indexed search). This is similar to RAG but managed within the Studio. You don’t have to write code to do the retrieval, you configure it.

Multi-channel deployment: Once built, a copilot can be published to multiple channels easily. This means you can have it available in a Teams chat, on your website via an embed, in a mobile app, or even through a telephony/IVR if integrated with voice. The promise is “build once, deploy anywhere.” Under the hood, this is leveraging Azure Bot Service channels (Teams, Telegram, Direct Line, etc.) because Copilot Studio is likely an evolution of the Bot Framework/Power Virtual Agents infrastructure with gen AI added. The mention of being able to escalate to human agents in Dynamics 365 or others also shows it plays nicely with existing support workflows.

Analytics and Governance: Copilot Studio provides a dashboard for analytics to monitor usage, success rates, feedback, etc.. This is critical, as with any LLM-based application you need to know if your copilot is actually helping users and where it might be failing. It can show metrics like resolution rate, deflection rate (how often it solved an issue without human help), and which questions it couldn’t answer. On governance, there’s an admin center for Copilot where you can manage who has access to build or use copilots, ensure data compliance (DLP policies) are applied to what the copilot can see, etc.

In summary, Copilot Studio is Microsoft’s answer to “how do I build my own ChatGPT-like assistant for my org, integrated with my org’s data and processes, without having to assemble all the pieces manually?” It leverages the heavy machinery (LLMs, connectors, bot runtime) behind the scenes, but surfaces a manageable interface for makers to create powerful agents.

Integration patterns with Copilot Studio:

Customizing Microsoft 365 Copilot: Copilot Studio allows organizations to extend the Microsoft 365 Copilot itself by adding custom topics or actions. For instance, if you want the Copilot in Teams to be able to handle an HR policy question that’s very specific, you could add that as a custom topic. Essentially, you’re injecting your custom Q&A or workflow into the Copilot that employees already have. This is huge, since it means the Copilot your users access isn’t just an out-of-the-box MS experience, but one augmented with your organization’s brain.

Building standalone copilots: You can also create entirely new copilots (e.g., a Customer Support Copilot for your website or a Copilot for suppliers to check order statuses​). In this case, you might not be embedding into Office, but rather using Copilot Studio to host a bot that interacts via a custom channel (website, app). This pattern is akin to building a chatbot with Bot Framework, but now it’s a gen AI-powered chatbot that can do more than just follow predefined dialogs – it can handle unstructured questions with on-the-fly answers.

Using Azure AI services in Copilot Studio (Pro-Code integration): Microsoft acknowledges that low-code will not cover every need, so they allow bringing in Azure services and custom models into your Copilot Studio project. One pattern is using Azure OpenAI Service with your own fine-tuned model or Azure Machine Learning endpoints for some custom tasks and hooking that in as an action. For instance, if you have a custom model that classifies financial documents, your copilot could use that model via an Azure ML endpoint when needed. This hybrid of low-code and pro-code means you’re not limited if you have the expertise, you can extend Copilot Studio with custom logic.

Tool usage and approval flows: Another integration aspect is how the copilot interacts with systems that require approval or logging. For example, if Copilot is to execute an action like “reset a user’s password” via a connector, you might want an approval from IT for security. Copilot Studio might provide a way to inject such approval steps or at least log them. A pattern could be using Power Automate flows triggered by the copilot’s action where you can put in an approval step. This way, the copilot hands off to a human or a workflow for sensitive actions and resumes after approval – ensuring humans stay in the loop appropriately.

Deep reasoning and future: The Copilot Studio page mentions “deep reasoning and agent flows” making it easier to build agents. This suggests ongoing improvements to how the LLM reasoner works with the dialog manager. Microsoft might incorporate chain-of-thought prompting under the hood to let the copilot break down complex tasks. As a builder, you might not see that, but you might notice your copilot can handle multi-part questions better over time. The “How Copilot Studio will evolve” section suggests upcoming features like more data sources supported, multi-modality (maybe image understanding or generating responses with images) and broader availability.

Integration with Azure AI Foundry: It’s worth noting that Azure AI Foundry also connects with Copilot Studio. For instance, Foundry’s Agent Service could be the underlying engine that Copilot Studio uses for orchestrating skills (since Agent Service is powered by SK and Copilot Studio may be a higher-level interface). Also, the ability to deploy open source models (via Llama 4 on Foundry) could mean your custom copilot might not always use OpenAI’s model. In future, maybe you point it at an internal Llama 4 model deployed on Foundry. Microsoft’s strategy seems to give flexibility: some organizations will be fine using OpenAI’s hosted models; others might demand an isolated model for compliance, either way, Copilot Studio could accommodate.

Putting it together: Copilot Integration Scenarios

Enterprise Knowledge Copilot: A company uses Copilot Studio to build a copilot that employees can ask any question about the company (policies, org chart, how-tos). They load up the copilot with internal SharePoint and wiki content as its knowledge base, maybe some FAQs as managed Q&A pairs, and publish it to Teams. Now an employee can chat with this copilot in a Teams channel – “How do I enroll in the 401k plan?” – the copilot retrieves the relevant policy doc and answers. If the question is complex or sensitive (like “What is my remaining PTO?” which might need personal data), the copilot could be configured to use a connector to query an HR system securely, or escalate to an HR rep if it’s something it can’t handle.

Copilot for Customer Support: The support team builds a copilot that customers use on the website (via a chat widget). This copilot can answer product questions (using the product manual and troubleshooting guide as knowledge sources) and can even create a support ticket or check order status via connectors to the backend systems. The team defines certain managed topics like “returns and refunds” with clear flows (to ensure the copilot gives the right policy info and initiates a return process if needed). They also ensure that at any point the user can say “I want a human” and the copilot will seamlessly transfer the chat transcript to a live agent system. Copilot Studio’s multi-channel publishing and escalation feature enables this.

Copilot inside Microsoft 365 with extensions: The IT department wants the new Microsoft 365 Copilot to also handle internal IT help. Instead of building separate, they use Copilot Studio to add a plugin so that when a user in Teams asks the built-in Copilot “My laptop is running slow, what should I do?”, Copilot now not only knows about Word and Teams, but also checks a custom knowledge base of IT support answers. Perhaps it even can create a ticket or schedule a tech appointment through an integration. To the user, it’s the same Copilot they use to draft emails, but now it can also assist with IT issues because it was extended via Copilot Studio.

These scenarios show how Copilot and Copilot Studio together aim to bring generative AI directly into both the software people use daily and the custom workflows of each organization. The integration patterns revolve around federating data access securely to the LLM, defining where structured process is needed vs free-form generation and ensuring human oversight where appropriate. Microsoft is essentially providing a blueprint (with Copilot Studio) so that organizations do not have to start from scratch wiring up LLMs, knowledge sources and dialog logic, a lot comes out-of-the-box, ready to be configured.

Vector Databases

Foundation models excel at producing and understanding text, images, speech or even videos, but they don’t have long-term memory of your proprietary industry-specific data. This is where vector databases come into play. A vector database (or vector index) is designed to store high-dimensional embeddings – numerical representations of data (text, images, etc.) – and perform similarity search in highly efficient ways.

In essence, vector databases enable semantic search in Gen AI applications: you encode documents or facts as vectors and likewise encode user queries, then find the closest matches to a query vector to retrieve relevant context for the LLM. This forms the backbone of Retrieval-Augmented Generation (RAG) systems, where the LLM is augmented with relevant retrieved knowledge before answering.

In this section, we overview key vector database solutions available on Azure and beyond: Azure AI Search (formerly Azure Cognitive Search) with its new vector search capability, other Azure databases with integrated vector functions (like CosmosDB, PostgreSQL or SQL Server), and popular third-party or open-source options like Pinecone, Milvus, and Weaviate. We’ll discuss how to choose among them and typical use cases.

Azure AI Search

Azure Cognitive Search has been for long a cloud service for full-text search over indexed documents. Recently rebranded as Azure AI Search, it has evolved into a powerful vector database and retrieval service suitable for generative AI scenarios and the go-to option for many users developing solutions with Azure first-party services. In November 2023, Microsoft announced the general availability of vector search in Azure AI Search, which meant that users can now store embeddings in their search index and perform vector similarity queries with low latency, fully managed by Azure.

Key features: Azure AI Search’s vector capability provides a comprehensive vector database solution to store, index, query, filter and retrieve your AI data in a secure, enterprise-grade environment. You can index not only text, but also embeddings of images or audio, etc., and then query by embedding. Importantly, Azure Search allows you to combine vector search with filtering on metadata. For example, you might restrict results to a certain category or date range using traditional filters, while the similarity search finds the nearest vectors among those. The GA release also introduced multi-vector queries (searching with more than one vector at once, useful for multi-modal scenarios), exhaustive k-NN (exact nearest-neighbor search when needed) and an improved semantic ranker​. The semantic ranker (previously “semantic search”) reorders results using large language models to ensure the most relevant passages are highlighted and it provides highlight snippets for transparency.

Azure AI Search shines in production-ready settings because it integrates well with Azure’s security (Azure AD, role-based access) and scales easily. You can have hybrid search – combine keyword and vector search – which is useful if you need to handle precise keyword matches and broader semantic matches together. And with SDKs in Python, JavaScript, .NET, and Java, it’s accessible to most developers. There are even integrations with popular libraries: Microsoft provides samples of using Azure Search with LangChain, LlamaIndex, and Semantic Kernel, so whichever framework you use for your AI agent, it likely plugs into Azure Search as a retrieval plugin.

Use cases: Azure AI Search is ideal when you have enterprise documents or knowledge bases that you want to augment your LLM with, things like intranet pages, manuals, wikis, customer reports, etc. You can ingest this content, have Azure Search compute embeddings (it supports built-in embedding generation or you can push in your own embeddings from a model of your choice) and then at query time, do a similarity search on user questions. Because it’s an Azure service, you get SLA guarantees and don’t worry about managing servers or indexes manually. It’s also great if your data needs to stay within your Azure environment for compliance.

Azure Cosmos DB with Vector Search

Azure Cosmos DB is a globally distributed NoSQL database service known for offering multiple APIs (SQL, MongoDB, Cassandra, etc.) and low-latency access. It has recently added native vector search capabilities to its feature set, allowing you to use Cosmos DB as a vector database in addition to a document store. In preview as of late 2023, the Cosmos DB team introduced vector indexing for the Core (SQL) API using a technology called DiskANN.

Key features: Cosmos DB’s vector search is built on DiskANN, an algorithm optimized for high-dimensional approximate nearest neighbor search that smartly uses disk and memory to scale to very large vector sets. A big advantage of this approach is global scalability: you could store billions of embeddings, far more than memory alone could hold, because DiskANN keeps a compressed portion in memory and the rest on a fast SSD, allowing search to still be efficient. The system is designed to handle typical vector operations at Cosmos DB’s global scale and with its familiar throughput provisioning (Request Units or RUs).

Cosmos DB offers different types of vector indexes:

A Flat index for exact brute-force search (100% accurate, best for small collections or when absolute precision is required).

A Quantized Flat index, which compresses vectors to reduce space (useful for moderately sized collections, giving a balance of accuracy and efficiency).

The DiskANN index for large-scale, approximate search with high performance and lower cost per search at scale.

One distinguishing feature is that Cosmos DB’s vector store supports updates and inserts in real-time without needing expensive re-indexing. Many standalone vector databases using HNSW have costly reindex operations for dynamic data; Cosmos’s approach is built to handle operational workloads with continual writes , aligning with Cosmos DB’s strength as an operational database. Plus, you get Cosmos DB’s other benefits: multi-region replication, auto-scaling throughput, and a 99.999% availability SLA.

Use cases: Cosmos DB with vector search is compelling if you want to add semantic search to your application’s operational data. For example, imagine a support ticket system: you could store tickets and their embeddings in Cosmos. When a new issue comes in, you vector-search for similar past tickets to suggest solutions. Since Cosmos can also store the full JSON of the ticket, you don’t need two separate databases. It’s also useful if you need geo-distribution – e.g., vector search in multiple regions close to users. If you already use Cosmos DB for your app data, enabling vector indexing on certain fields can simplify your architecture (no separate vector DB to sync). Keep in mind it’s relatively new, so ensure the preview (or GA in the future) meets your performance needs. But the promise is that you can have a one-stop database for both standard queries and vector queries, which is very attractive for microservice architectures on Azure.

Callout: Vector Capabilities in Azure PostgreSQL and SQL Server

Microsoft has also recently introduced vector search extensions in both Azure Database for PostgreSQL and Azure SQL Database, expanding vector-native functionality beyond the more widely used vector DBs.

PostgreSQL: With support for the pgvector extension, Azure Database for PostgreSQL now allows you to store and index embeddings natively. You can perform cosine, L2 or inner product similarity searches on vector columns. This is especially useful when your application already relies on relational logic but requires RAG or similarity search capabilities. For example, if you store user reviews in PostgreSQL, you can generate and store embeddings alongside metadata and query both with SQL.

Azure SQL: In 2024, Microsoft added support for vector data types and indexes in Azure SQL Database (in preview). This lets you create vector indexes on columns, use vector_distance functions in SQL queries and combine vector similarity with standard relational filtering. This is particularly useful for enterprise developers who want to bring GenAI capabilities into legacy applications without migrating to new infrastructure.

These additions make it possible to build end-to-end AI applications using traditional relational databases, with full control over schema, joins and transactional logic, while benefiting from vector-native search for tasks like document retrieval, recommendations or matching workflows.

Selection Criteria and Best Practices

With several good options in the market, you might want to consider these criteria when selecting a vector database:

Scale and Performance Needs: How many vectors and what throughput? If you’re dealing with tens of millions of vectors or less, Azure AI Search or Pinecone can handle it. For extreme scales (billions+), you might consider self-hosting Milvus on powerful VMs with disk indexes or using Cosmos DB’s DiskANN for a truly global scale. Latency requirements also matter – for sub-50ms response, an in-memory index (Pinecone, Weaviate, Milvus with HNSW) might be needed. If 100ms+ is fine, and especially if you need exact results, brute-force on a smaller set or DiskANN on large sets could be okay.

Integration and Ecosystem: If you are all-in on Azure and want seamless integration with Azure OpenAI and other services, Azure AI Search is very attractive (with Azure OpenAI, you could even use the built-in CogSearch plugin support in OpenAI’s function calling). Cosmos DB makes sense if you’re already using it. If your architecture is polyglot or multi-cloud, a neutral solution like Pinecone or open source (Milvus/Weaviate) might fit better. Weaviate’s GraphQL might appeal if you have front-end apps that want to directly query the vector DB.

Managed vs Self-Hosted: Managed services (Azure Search, Pinecone, Weaviate Cloud, Zilliz Cloud) remove ops burden, which is key if you lack a team to babysit indexes. Self-hosting (Milvus, open Weaviate) gives more control and potentially lower cost at large scale, but you assume the complexity of operations. Azure Cosmos sits somewhere in between: it’s managed by Azure, but you still manage the DB provisioning/tuning as part of your app (though not the server itself).

Features & Query Capabilities: Do you need complex boolean filters or aggregations with your vector search? Azure Search, Weaviate, and others support filters. If you need real-time updates (streams of data coming in), consider how each handles upserts: Cosmos is designed for that, Pinecone handles updates well, some others might require periodic reindex steps (check HNSW-based solutions for how deletes are handled, etc.). If you want an all-in-one that also generates embeddings, Weaviate with modules or Azure Search with Cognitive Skills might be useful. If you need custom scoring or re-ranking beyond pure vector similarity (like blending in business metrics), see which allow customizing the search pipeline.

Cost: This can vary widely. Azure AI Search charges mainly by the number of indexes, document count, and query volume. It can be very cost-effective for moderate sizes, but if you have huge data, the cost might go up (though still, its DiskANN may be cheaper than keeping everything in RAM). Pinecone is usage-based (pods hours and vector count). Self-hosting cost is just Azure VM or Kubernetes cost + your management overhead. Sometimes a combination works: e.g., use Azure Search for moderate vector search needs, but if you later scale, you could migrate to a self-hosted Milvus on Azure.

Typical use cases for vector DBs in Gen AI solutions:

Enterprise Q&A (RAG): As mentioned, storing company knowledge (policies, wikis, SharePoint docs) in a vector DB and using it to ground a Copilot or Agent’s answers. Azure AI Search is practically built for this scenario, but Pinecone is also quite perfomant here. The integration with LLMs (via frameworks like LangChain or SK) is straightforward: a user asks something, you vector search for top N passages, prepend them to the LLM prompt and the LLM can cite or use that info in its answer.

Personalization & Recommendations: Beyond text, vector embeddings of user behavior or preferences (from products they viewed, etc.) can be stored to recommend similar items. For example, Azure Cosmos DB’s vector search could allow a retail site to find “customers with similar purchase patterns” and make recommendations. Or Pinecone might be used by a media app to suggest songs with similar audio embeddings.

Anomaly Detection: In security or IT operations, you can embed system events or traces and use a vector DB to find outliers (things that are not similar to anything seen before). While often such uses might require custom logic, a vector store helps quickly query “is this event’s embedding far from all known normal events?”.

Image and Multi-modal Search: Vector DBs aren’t just for text. If you use Azure AI Vision to get embeddings for images, you can index them to allow image similarity search (find duplicate or near-duplicate images or search by image content). Milvus is known to be used in visual search engines, for instance. Weaviate has modules for specific types like image2vec. Azure AI Search can index image embeddings too (you’d have to supply the vector, since Azure Search itself doesn’t generate image embeddings at the moment of writing).

In summary, vector databases are a crucial component in generative AI architectures when you need external knowledge or memory. Azure provides first-party solutions (AI Search, Cosmos DB) that integrate with its ecosystem, while excellent third-party and open alternatives exist for various needs. Often, the decision comes down to convenience vs control: managed services for ease, open-source for flexibility. Whichever you choose, the pattern remains: store embeddings of your data, use similarity search to fetch relevant info and feed that into the generative model. This pattern significantly enhances the model’s usefulness and accuracy, making responses more grounded and allowing real-time data to be incorporated.

Additional data platforms

Snowflake Cortex on Azure

Snowflake Cortex is Snowflake’s managed generative AI service, bringing LLM functions directly into Snowflake’s environment, allowing users to run AI workloads without exporting data externally. Core features include LLM SQL Functions like COMPLETE(), EMBED(), and text-to-SQL conversions. This enables use cases like customer feedback sentiment analysis, RAG pipelines and automated reporting — all inside SQL or Python.

At GA (May 2024), Snowflake Cortex offered access to models from Mistral, Meta’s Llama 3, Google and its proprietary Arctic family. Arctic Embed, in particular, is a high-performance embedding model optimized for retrieval tasks within Snowflake.

On Azure, Cortex is particularly powerful: Snowflake instances co-located in Azure regions can integrate seamlessly with Azure services (e.g., Azure Data Factory orchestration, Logic Apps, or enrichment pipelines). Cortex’s serverless fine-tuning capabilities allow users to customize Llama models directly on Snowflake compute. AI safety is addressed through Llama Guard, helping enterprises maintain compliance when running AI workloads internally.

For Azure architects, Snowflake Cortex becomes a complementary AI runtime: ideal for embedding-heavy RAG systems, document classification and internal AI copilots, while Azure OpenAI or Hugging Face services can handle broader multimodal or conversational workloads.

Azure Databricks and DBRX Models

Azure Databricks has evolved from a data platform into a full generative AI ecosystem. With native support for distributed LLM fine-tuning (using MosaicML libraries after acquisition) and Hugging Face Transformers, Databricks allows enterprises to train, serve and scale LLMs at industrial scale. Databricks ML Runtime for GenAI simplifies this further with preconfigured environments for LLM training and inference.

Databricks also launched serverless Vector Search, enabling vector storage and retrieval directly from Delta Lake. Combined with Model Serving, organizations can deploy models like Dolly (instruction-tuned open LLMs) or MosaicML’s MPT models efficiently.

DBRX models likely reflect Databricks’ push to standardize optimized GenAI runtimes, building on both internal models (like Dolly) and fine-tuned versions of open models (e.g., Llama or MPT). Through DeepSpeed, HuggingFace Accelerate and TensorRT optimizations, Databricks enables training and hosting of multi-billion parameter models across Azure GPU clusters (ND A100, H100, or Grace-Blackwell VMs).

Typical workflows include full RAG applications over Lakehouse datasets, custom model fine-tuning, internal LLM evaluation and endpoint deployment, all orchestrated with MLflow and Azure-native data integrations.

NVIDIA on Azure

NVIDIA’s partnership with Azure has accelerated the deployment of powerful GenAI solutions. NVIDIA NIM microservices are now available in Azure AI Foundry, allowing organizations to deploy fully optimized inference microservices, based on Triton Server, TensorRT-LLM, and HuggingFace-compatible APIs, with minimal setup. These pre-tuned microservices support models like Llama-3.3-70B and Mixtral 8x7B, with native integration into Azure’s agent frameworks (e.g., Semantic Kernel, Azure AI Agent Service).

TensorRT-LLM enhancements enable 2x-4x faster inference speeds through transformer kernel optimizations, quantization and multi-GPU scaling, making them critical for cost-effective production deployment of large models.

At the infrastructure level, the ND GB200 v6 VMs bring NVIDIA’s Grace-Blackwell superchips to Azure. With a 2.5x performance improvement over previous generations and innovations like unified CPU/GPU memory via NVLink, these instances are ideal for training or deploying frontier-scale AI models on-demand.

Together, NIM, TensorRT-LLM, and Blackwell VMs ensure Azure remains one of the fastest and most flexible clouds for high-performance generative AI workloads.

Web Application Frameworks for AI

Once you’ve built an AI model or agent and armed it with data via vector search, you need a way for end-users to interact with it. This is where web application frameworks come in – tools and services that help you create user-facing applications (websites, dashboards, chat UIs) to showcase your generative AI capabilities. In this section, we’ll look at several approaches to deploying AI applications on Azure: using Azure App Service for scalable web hosting and leveraging quick-build AI app frameworks like Streamlit, Gradio, and Chainlit. We’ll walk through a mini end-to-end example of an AI web app, including pseudo-code and a sample deployment configuration (YAML), to illustrate how all the pieces can be glued together.

Azure App Service for AI Web Apps

Azure App Service is a fully managed platform for hosting web applications, APIs and backends. It supports various runtime stacks (Node, Python, .NET, Java, etc.) and abstracts away the underlying servers, allowing developers to focus on the app code. For AI applications, Azure App Service can be an excellent hosting option when you have a web frontend or API that needs to scale to many users.

Why use App Service? If your AI solution is, say, a custom chatbot that users access through a web page or Teams, you can build a web frontend (HTML/JavaScript or a framework like React) and run the backend on App Service. It handles deployment (through CI/CD or container images), auto-scaling and integrates with Azure networking (VNet, private endpoints) and monitoring. Essentially, it gives you a production-grade web host with minimal DevOps effort. You push your code and Azure runs it reliably.

For example, suppose you created an AI-powered FAQ bot with a REST API. You could host that API on App Service (or Azure Functions for a serverless approach). The service could be configured with the necessary environment variables for keys (Azure OpenAI key, etc.). App Service also supports WebSockets, which is useful if you want real-time updates or streaming responses from the AI (like token streaming for a smoother chatbot experience).

One advantage for AI apps is you might initially prototype something in a simpler way (like using Streamlit or Gradio, discussed next) and then later integrate or deploy via App Service for a more robust solution. Azure App Service can even host Streamlit apps via container deployment: you can containerize a Streamlit app and run it as a web app on Linux in App Service.

When using App Service, consider using Azure Monitor and Application Insights to track your AI app’s performance. That way you can monitor response times of your LLM calls, usage and errors in one place.

Streamlit: Rapid UI for Data and AI

Streamlit is an open-source Python framework that lets you create interactive web apps for data science and ML prototypes with remarkably little code. It’s beloved in the data science community for its simplicity: you write a Python script with Streamlit’s API (which feels like writing a simple script, not traditional web development) and Streamlit turns it into a reactive web interface. This is great for quickly standing up a demo or internal tool for your generative model.

For instance, with a few lines you can have text input boxes, buttons and charts. When building a generative AI app, you might use Streamlit to create a chat interface: a text input for the user question and then display the model’s answer in the app. Streamlit handles the layout and web server under the hood. It also auto-reloads when you update the script, making it very iterative.

Example: Here’s pseudo-code for a simple Streamlit app that connects to Azure OpenAI and an Azure Search index to answer questions:

[code example to come]

With this script, running streamlit run app.py would spin up a local web server and show a simple UI with a title, text box and the answer output. Under the hood, it’s doing retrieval (step 1) and generation (step 3).

Deploying a Streamlit app for wider use can be done in multiple ways. You could run it on a VM or container. On Azure, one neat way is to use Azure Container Instances or App Service with a Docker image. You’d create a Dockerfile that installs your Python deps and launches streamlit (which by default runs on port 8501). Then you deploy that container. Another approach is to use Streamlit Community Cloud for quick sharing (for smaller scale), though that wouldn’t be on Azure infrastructure.

There are other popular web application frameworks like Gradio or Chainlit. Our recommendation is to use Gradio when you want to expose a model with minimal UI work, specially for demos or if you have multiple input types (audio, image, etc.). Chainlit is better suited if you specifically want to build an application around an LLM or agent and would benefit from built-in chat UI components and debugging visuals (and you’re okay writing a bit of Python for the app logic.

Example: End-to-End Deployment of an AI Web App

To solidify these ideas, let’s outline a mini end-to-end example. Suppose we want to deploy the policy Q&A chatbot we sketched in Streamlit pseudo-code, on Azure, using Azure App Service with a container.

1. Develop the app locally: We write app.py using Streamlit (as above). We test it locally with some questions to ensure it retrieves from Azure Search and calls Azure OpenAI correctly. We ensure the Azure credentials (Search endpoint/key, OpenAI key, etc.) are set as environment variables for local testing.

2. Containerize the app: We create a Dockerfile such as:

FROM python:3.10-slim

# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Copy app code
COPY app.py .

# Expose Streamlit default port
EXPOSE 8501

# Set streamlit to listen on all interfaces and use port 80 (App Service expects web apps on 80)
ENV STREAMLIT_SERVER_PORT=80
ENV STREAMLIT_SERVER_ADDRESS=0.0.0.0

# Run the Streamlit app
CMD ["streamlit", "run", "app.py"]

We include in requirements.txt the needed libraries: streamlit, azure-search-documents, openai (for Azure OpenAI SDK or we use REST calls), etc.

3. Build and push container: Using Azure Container Registry (ACR) or Docker Hub, we build this container and push it. For example:

docker build -t myregistry.azurecr.io/policy-bot:v1 .
docker push myregistry.azurecr.io/policy-bot:v1

4. Create App Service for Container: We configure an Azure App Service to use a Container image. This can be done via the Azure Portal or Azure CLI. For example via CLI:

az webapp create --resource-group myRG --plan myAppServicePlan \
   --name my-policy-bot --deployment-container-image-name myregistry.azurecr.io/policy-bot:v1

We also need to configure the setting for ACR credentials (if the registry is private) and the environment variables for our app:

az webapp config appsettings set -g myRG -n my-policy-bot --settings \
   SEARCH_ENDPOINT="https://<your-search>.search.windows.net" \
   SEARCH_KEY="<search-admin-key>" \
   OPENAI_KEY="<azure-openai-key>" \
   OPENAI_ENDPOINT="<azure-openai-endpoint>"

In our app code, because we used os.getenv(), it will pick up these env vars.

5. (Optional) YAML for CI/CD: The user asked for YAML example, possibly meaning an Azure DevOps or GitHub Actions pipeline. Here’s an example GitHub Actions workflow (deploy.yml) that builds and deploys this on push:

name: Build and Deploy PolicyBot

on:
  push:
    branches: [ main ]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Login to Azure
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Build Container
        run: |
          az acr login --name <myregistry>
          docker build -t <myregistry>.azurecr.io/policy-bot:${{ github.sha }} .
          docker push <myregistry>.azurecr.io/policy-bot:${{ github.sha }}

      - name: Deploy to App Service
        uses: azure/webapps-container-deploy@v2
        with:
          app-name: 'my-policy-bot'
          images: '<myregistry>.azurecr.io/policy-bot:${{ github.sha }}'

This workflow logs into Azure, builds the container (tagged with the commit SHA), pushes to ACR, then tells App Service to deploy that image. The App Service was already created out-of-band with needed settings. (In a production scenario, we’d parameterize the image tag and perhaps use staging slots for zero-downtime deployment, etc., but this illustrates the idea.)

6. Access the app: Once deployed, the Streamlit app is running on Azure under the hood. Navigating to https://my-policy-bot.azurewebsites.net will show the Streamlit interface. The app is now live – users can type questions and get answers, with Azure Search and OpenAI doing the heavy lifting in the backend.

This example demonstrates how an AI model (GPT-4) + data index (Cognitive Search) + a Python UI (Streamlit) can be glued together on Azure with minimal fuss. We could have also implemented the UI in a more traditional web framework (React frontend, Flask/FastAPI backend). In fact, for a more polished product, one might separate the backend (just an API that given a question returns an answer with sources) and a frontend. But for many internal tools or prototypes, a Streamlit or Gradio app is sufficient and significantly speeds up development time. It’s not uncommon to iterate a lot in Streamlit/Gradio and then gradually refactor into a more robust web app if needed.

GitHub Copilot Models and Integrations

Finally, let’s turn to GitHub Copilot, the AI pair programmer that has become a staple for many developers and the models that power it and how it integrates into development workflows. While GitHub Copilot started off using OpenAI’s Codex model, it has significantly evolved. By 2025, GitHub Copilot uses a variety of models (OpenAI’s GPT-4 based code model and others like Anthropic’s Claude for certain features) and has extended far beyond just code completion in VS Code. We’ll overview the models (sometimes referred to as GitHub Copilot models, like GPT-4o, GPT-4, etc.) and the key integrations (editors, pull requests, CLI, etc.).

A notable and recent addition is Agent Mode, which transforms Copilot into an autonomous peer programmer. In this mode, Copilot can perform multi-step coding tasks, analyze codebases, propose edits across multiple files and even suggest terminal commands. It can detect and fix runtime errors autonomously, iterating until the task is completed.

Models Behind GitHub Copilot

GitHub Copilot initially (in 2021-2022) was built on OpenAI Codex, an offspring of GPT-3 tuned for code. It provided suggestions as you type. However, as the usage grew and competition emerged, GitHub (and parent Microsoft) have upgraded Copilot’s brain:

GPT-4 based models (“GPT-4o”): GitHub started using a variant of GPT-4 (possibly an OpenAI fine-tuned model optimized for coding, codenamed GPT-4o where “o” might stand for “OpenAI Orchestrator”. By mid-2024, GitHub Copilot Chat and certain features were powered by GPT-4o. This gave much better quality for multi-turn explanations and complex code generation. Copilot for Pull Requests (which writes summaries or reviews) also uses GPT-4.

GPT-3.5 and others remain for fast completion: Not every suggestion uses GPT-4, because of latency and cost. There are mentions of GPT-4.1 and o1 models. Possibly OpenAI o1 is an optimized 3.5-tier model for code (maybe analogous to GPT-3.5 Turbo fine-tuned on code). Copilot might choose a faster model for real-time single-line completions, and the heavy GPT-4 only when you engage Copilot Chat or ask for an explanation.

Anthropic Claude: GitHub integrated Anthropic’s Claude models as well. For example, Copilot X (especially in Copilot for Business) allows using Claude 3.5 and Claude 3.7 “Sonnet” models. These might be used for tasks like generating test cases or verifying code against natural language because Claude has different strengths (and maybe fewer licensing restrictions for some uses). The UI shows model options including Claude 3.7.

Google Gemini: Interestingly, GitHub’s UI snippet mentions Google Gemini 2.0 Flash​ as an available model. This suggests GitHub Copilot is model-agnostic to an extent and has integrated models from multiple providers. Gemini is Google’s next-gen model (the successor to PaLM, etc.). If Copilot offers Gemini, it means for some tasks (maybe if a user opts in) they can use Google’s model via Copilot.

Proprietary enhancements: GitHub might not train its own large model from scratch (it relies on partners), but it likely fine-tunes or at least orchestrates them cleverly. They also introduced specialized models for pull request diff summarization and vulnerability detection (could be smaller models or heuristics that work with the main LLM).

GPT-4.5 and beyond: The pricing plans showed that Pro+ tier includes “GPT-4.5”, implying that an upgraded GPT-4 (maybe GPT-4 Turbo or a new version) is accessible. This keeps Copilot at the frontier of model capabilities.

So, “GitHub Copilot Models” refers to this ensemble of models (OpenAI, Anthropic, Google) integrated into the Copilot service. The user typically doesn’t directly choose the model (unless toggling in settings), but these are working behind the scenes for different Copilot features.

Copilot Integrations

GitHub Copilot started in Visual Studio Code as an extension offering code completions. Now, it’s a suite of features across various aspects of development:

IDE Extensions (VS Code, Visual Studio, JetBrains): This is still the primary integration. As you type code, Copilot suggests the next line or entire function. You can also write a comment like “// function to reverse a string” and Copilot will generate the function. The integration is context-aware (it looks at the file, maybe neighboring files and the context of the comment). By 2025, these suggestions have gotten more accurate and can even incorporate info from your repository (like if you have a utils file, it might suggest using a function from it).

Copilot Chat in IDEs: Now developers can open a chat panel in VS Code, where Copilot behaves like an assistant. You can ask “Explain this code”, “How do I fix this error?”, or even general questions. This chat uses the more advanced model (GPT-4) and can reference your codebase. It’s like having Stack Overflow + an AI colleague, right in the editor. It can also execute simple queries like searching docs.

Pull Request Integration on GitHub.com: GitHub has introduced features where Copilot helps with PRs. For example, when you open a PR, Copilot can generate a description based on the diffs (summarizing changes). This uses AI to describe what the code change does in natural language. Also, Copilot for PR Reviews allows you to ask Copilot to review a PR and it will comment on potential issues or suggest improvements. This uses the model to reason about code differences and catch things like missing tests or performance concerns (not perfect, but a helper).

Copilot CLI: There’s a Copilot CLI tool that can turn natural language into shell commands or find commands. For example, you can type ?? find all JSON files containing “foo” and it will return the appropriate grep command. This integration helps with command-line productivity.

Copilot in Documentation (Labs): There have been experiments of Copilot in GitHub Docs or VS Code docs, where you can ask questions about how to use a function and it gives tailored answers (like an interactive doc assistant).

Copilot in Other Editors: Besides VS Code/VS, JetBrains (IntelliJ, PyCharm, etc.) now have Copilot, and Neovim via extension. Microsoft also integrated it in places like their Azure Data Studio (for SQL) and even in Windows Terminal (experimental).

Code Spaces integration: In GitHub Codespaces (cloud dev environment), Copilot is readily available. So any time you spin up a codespace, you have the power of Copilot in that environment, making cloud dev quite powerful.

Security integration (Copilot for CLI and code): There’s talk of Copilot integrating with security (perhaps in GitHub Advanced Security suite) where it can highlight insecure code or suggest fixes (like “there’s a SQL injection risk here, use parameterized queries”).

Training and Customization: While not an integration per se, GitHub did launch Copilot for Enterprises with some ability to set organization-wide policies. For instance, you can filter suggestions that resemble your internal code (to avoid license issues if concerned, though GitHub implemented a filter to avoid verbatim reproduction of large public code chunks). Possibly, enterprises can also point Copilot to internal docs or libraries to bias suggestions (there was something called “Copilot for Docs” in preview where you can add a repository of docs and Copilot would answer questions from it).

Copilot X Vision: GitHub’s vision named “Copilot X” encompassed all these: CLI, PRs, chat, etc. By 2025, many of these are realized. They also mentioned the possibility of voice (“Hey Copilot, deploy my app”) which might be in the works via Microsoft’s speech tech.

Extensions ecosystem: GitHub Copilot now supports an ecosystem of third-party extensions that can augment it. The UI snippet shows Copilot Extensions for Azure, Datastax, Docker, etc.​. These likely allow Copilot to interact with those services (e.g., Copilot extension for Azure might let it retrieve info about Azure resources or deploy something). This is akin to giving Copilot tools, similar to how ChatGPT plugins work. In an IDE context, an extension could allow Copilot to query a database schema (Datastax extension for Cassandra) to better assist with queries, etc. This is an exciting integration pattern: AI meeting DevOps. For example, a developer in VS Code could ask, “Copilot, what’s the status of our Azure deployment?” and via an extension and the Copilot chat, it could surface that info. It blurs the line between coding and operating.

Impact on development workflow:

Developers get suggestions for boilerplate code, unit tests, etc., speeding up coding.

They can ask Copilot to explain code (great for onboarding onto a new codebase).

Code reviews become augmented by AI – catching things a human might overlook (or at least highlighting them for inspection).

Writing documentation or release notes can be eased by Copilot’s summaries.

Multi-model advantage: If GPT-4 is slow or overkill for a trivial task, Copilot might use GPT-3.5 or Claude which is faster. They even allow users to choose model emphasis (some might prefer non-OpenAI for data residency or cost reasons; e.g., use Claude which might have a different pricing).

Concerns and best practices:

As an org, you might integrate Copilot but ensure you use the Copilot for Business which doesn’t use your code to improve the public model (GitHub has guarantees about not training on private code prompts).

Some regulated industries might have to avoid cloud-based Copilot or wait for on-prem (there’s none yet, though Azure OpenAI can do on-prem inference in containers for GPT, maybe future Copilot can point to that).

Integrations like the extensions need careful security. For example, if Copilot can run a Docker command via an extension, you want to ensure it’s not doing something harmful unless the user approves. GitHub is likely addressing these with sandboxing.

In summary, GitHub Copilot’s models and integrations have matured to become a holistic AI assistant across the software development lifecycle. It’s not just writing code; it’s helping plan, write, review, fix, and even manage code and related tasks. The use of multiple models (OpenAI, Anthropic, etc.) ensures high quality and flexibility. The tight integration into tools developers already use (IDE, GitHub, CLI) means it adds value without much friction.

For an Azure-focused perspective: GitHub Copilot is integrated into Azure DevOps environments as well (for instance, people use it in Azure DevOps repo contexts via VS Code). Also, Azure’s cloud makes heavy use of GitHub (e.g., for Infrastructure as Code, deploying via GitHub Actions, which can also use Copilot in editor when writing workflows. We might see in future even Azure DevOps (the product) integration with Copilot, although as of 2025, GitHub is the focus for these features.

Chapter 3. Model Selection and Prompt Engineering Best Practices

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the third chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

Selecting the right generative AI model is a foundational step in building effective AI applications. In this chapter, we’ll explore the key factors that influence model selection, ranging from performance benchmarks to deployment considerations, and how to navigate these choices using the Azure Model Catalog within Azure AI Foundry. Once a model is selected, its performance depends heavily on how you interact with it. That’s why we’ll also dive into prompt engineering best practices across different model families, and show how you can apply these techniques directly within AI Foundry to get the most out of your generative AI solutions.

Considerations for Model Selection

The process of model selection requires a structured approach to ensure alignment with your specific needs and use case. It begins by clearly articulating the specific task or problem you aim to solve. From there, it is essential to identify key features required for this task, such as multi-language support, input methods, or programmability. A well-defined use case should include the following factors:

Modalities: Determine whether the model needs to generate text, images, audio, video, or a combination of these (multimodal). For example, a chatbot may only require text generation, while an AI-powered design assistant may need text-to-image capabilities.

Customizability and Personalization: Fine-tuning allows organizations to tailor generative AI models to specific domains, improving accuracy, consistency, and relevance beyond standard prompting. It is particularly useful for adapting models to specialized industries (e.g., legal, healthcare) or aligning them with a brand’s tone and terminology. It is important to weigh the benefits of fine-tuning against costs, opting for it when high precision and domain expertise are essential, while considering lighter customization methods when flexibility and scalability are priorities.

Performance Expectations: Define the required accuracy, coherence, and quality of outputs. Some tasks, like creative text generation, may prioritize diversity and novelty, while others, like medical text summarization, demand high factual accuracy.

Latency Constraints: Identify acceptable response times. Applications requiring real-time interaction, such as AI assistants or customer support chatbots, need models optimized for low-latency inference. On the other hand, batch-processing tasks, like document summarization, may allow longer processing times.

Cost Considerations: Evaluate budget constraints, including licensing fees, cloud usage costs, and operational expenses. Some models, like open-source alternatives, may be free to use but require significant infrastructure investment, while most proprietary APIs charge per tokens ingested and generated.

Execution Environment: Decide whether the model will run in the cloud, on-premises, or at the edge, depending on your business and technical requirements. Cloud-based models offer scalability, high availability, and seamless integration with AI services, making them ideal for enterprises looking to leverage managed infrastructure and reduce operational complexity. On-premises or edge deployments may be preferred for scenarios requiring ultra-low latency, offline capabilities, or specific data residency requirements. Hybrid approaches can also be considered to balance performance, cost, and compliance needs.

Scalability Needs: Consider how well the model can scale to support growing workloads and business needs. Cloud-based AI models as a service are designed for seamless scaling, enabling enterprises to handle high traffic and fluctuating demand without requiring extensive infrastructure management. Open-source models, on the other hand, may need custom deployment strategies, such as distributed inference, optimized hardware utilization, and load balancing, to ensure scalability. Choosing the right approach depends on whether scalability is a key requirement and how much control you need over the deployment process.

Team Skills and Expertise: Assess your team’s familiarity with generative AI models and the expertise required to manage, optimize, and fine-tune them. Cloud-based AI services simplify model usage by abstracting infrastructure complexities, making them more accessible to teams with limited AI deployment experience. However, open-source models might require specialized skills in inference optimization, fine tuning, distributed computing, and security. Fine-tuning models for domain-specific tasks can be a powerful way to improve performance but demands expertise in data preparation, hyperparameter tuning, and continuous monitoring. Additionally, open-source deployments often necessitate rigorous security testing and the implementation of additional guardrails to ensure compliance, robustness, and safe usage in production environments.

Security and Compliance: Ensuring the security of generative AI models is critical, especially when handling sensitive data or deploying models in regulated industries. Cloud-based AI services often include built-in security measures such as encryption, authentication, and access controls, reducing the complexity of security management. However, organizations must still assess how data is processed, stored, and accessed to align with internal security policies. Open-source deployments require additional security considerations, as they often demand rigorous testing and validation before deployment. Teams might have to implement additional guardrails to prevent unauthorized access, adversarial attacks, and biased or harmful outputs. Security measures may include input sanitization, adversarial robustness testing, continuous monitoring, and applying red-teaming techniques to identify vulnerabilities. Additionally, organizations should consider risks such as model integrity, prompt injection attacks, and ethical safeguards to prevent misuse. Whether using cloud-based services or self-managed models, a proactive security strategy is essential to maintaining trust and reliability in generative AI applications.

Licensing and Compliance: Ensuring that the chosen AI model aligns with licensing requirements and legal constraints is crucial for both commercial and enterprise deployments. Some models are open-source and freely available, but they may have restrictions on commercial use, redistribution, or modifications. Proprietary models, on the other hand, often come with licensing fees and usage limitations, such as rate limits or data retention policies. Understanding these restrictions upfront helps avoid legal and operational issues down the line.Additionally, location requirements imposed by certain legislations must be considered. Some jurisdictions, such as the EU under GDPR, require that data processing and storage occur within specific geographic boundaries. Organizations operating in regulated industries, such as healthcare or finance, may need to ensure that their AI models comply with data residency laws and that cloud-based AI services provide region-specific hosting options. In cases where strict location policies apply, hybrid approaches, where sensitive workloads run on-premises or within designated regions, can help balance compliance and scalability.

Generative AI Benchmarks: When selecting a generative AI model, understanding its capabilities through standardized benchmarks is crucial. Benchmarks provide objective metrics to evaluate models across various tasks, ensuring alignment with specific application requirements. Below is an overview of some of the most prominent benchmarks in the field:​

MMLU (Massive Multitask Language Understanding): Evaluates models across 57 subjects, including humanities, STEM, and social sciences, to assess knowledge and reasoning abilities from elementary to professional levels.​

GPQA (Graduate-Level Google-Proof Q&A): Comprises 448 challenging multiple-choice questions in biology, physics, and chemistry, designed to be difficult for both AI models and humans, even with internet access.​

GAIA (General AI Assistant): Assesses AI assistants on real-world tasks requiring reasoning, multi-modality handling, web browsing, and tool-use proficiency, aiming to measure general intelligence akin to human capabilities.​

ARC-AGI-2 (Abstraction and Reasoning Corpus): Designed to test AI systems on tasks that are easy for humans but challenging for AI, emphasizing reasoning and generalization over memorization.​

HumanEval: A benchmark consisting of 164 Python programming problems designed to evaluate the code generation capabilities of language models.​

SWE-bench Verified: A human-validated subset of the SWE-bench benchmark, focusing on evaluating AI models’ ability to solve real-world software engineering issues by generating patches for GitHub issues.​

LiveCodeBench: Provides a holistic and contamination-free evaluation of large language models for coding tasks, focusing on code generation, self-repair, code execution, and test output prediction.​

MATH: A dataset of 12,500 competition-level math problems spanning seven difficulty levels, designed to assess mathematical reasoning and problem-solving abilities of AI models.​

MGSM (Multilingual Grade School Math): A benchmark of 250 grade-school math problems translated into 10 languages, testing large language models’ multilingual reasoning abilities.​

BFCL (Berkeley Function Calling Leaderboard): Evaluates large language models’ ability to accurately call functions and tools across various programming languages and scenarios, including multi-turn interactions.​

Custom task-specific benchmarks: Creating your own tailored benchmarks allows you to precisely evaluate model performance on tasks unique to your business or domain. By crafting representative test cases and metrics closely aligned with real-world requirements, organizations can effectively assess practical suitability, accuracy, and reliability beyond standardized evaluations.

By defining these factors in advance, you can better understand the landscape of potential model candidates and ensure the chosen model aligns with both technical and business constraints.

Choosing your model on the Azure AI Foundry

Model Catalog

Once your use case is well-defined, the next step is to evaluate available model options, leveraging tools like the Azure AI Model Catalog to streamline this process. This involves researching a variety of models, including open-source, proprietary, and third-party solutions, to identify potential candidates aligned with your specific requirements. Azure AI Model Catalog serves as a centralized hub where you can discover, assess, customize, and deploy a diverse array of AI models, currently supporting over 1800 different models. Depending on your needs, you might opt for a text-to-text generator for content creation, a text-to-image generator for visual content, or a large language model (LLM) for natural language processing tasks. For instance, if real-time response is critical, models optimized for low-latency inference, such as quantized transformer architectures, could be ideal. If budget constraints exist, smaller models (e.g., gpt-4.1-mini, gpt-4.1-nano) or those supporting self-hosted deployments (e.g., Phi-4-mini-instruct, Ministral-3B) may offer a more cost-effective solution.

Figure 3-1. AI Foundry Model Catalog UI

The Azure AI Model Catalog (see Figure 3-1) offers models across various modalities, including text, image, speech, and multimodal applications. You can filter models based on the model vendor/creator, the industry (if it is a industry specific model), the capabilities (ability of supporting other AI services such as Responses API or Agent Service), the deployment options, the inference tasks or modalities, the fine tuning tasks and the licenses, to find the best fit for your project. For instance, if your application requires text generation, you would select “text generation” under inference tasks and you will see the full list of models that could be used (see Figure 3-2).

Figure 3-2. Substet of available options to filter and search in the Foundry Model Catalog

To ensure the selected model meets your performance criteria, the catalog provides a model comparison tool, where you can compare benchmarks across different metrics (see Figure 3-3).

Figure 3-3. Comparison of different metrics

Each model has its own model card containing detailed information about containing in depth information about the results of the model in different benchmarks (with information like quality, estimated cost, latency and throughput), model versions and availability, supported data types and languages, responsible AI considerations, etc. In the example here (see figure 3-X) we can see the model card for Phi-4-mini-instruct which provides a detailed set of informations for the model containing its description, data, media and languages supported, release notes and use case examples but also access to benchmarks, the list of existing deployments and pricing and more. Through the model cards you can create deployments of the model by clicking on the “Use this model” button and for a set of models you will have the possibility to directly fine tune the model.

Figure 3-4. Model card for Phi-4-mini-instruct

Each model card contains its licensing information (see Figure 3-4), helping you navigate usage rights and restrictions to ensure compliance with legal and commercial requirements. Some models are open-source and freely available for modification, while others have proprietary licenses that may impose limitations on commercial use, redistribution, or fine-tuning. Understanding these licensing terms is essential for selecting a model that aligns with your deployment needs, whether leveraging fully managed services or customizing models for specific business applications.

Note

For rapid and cost-free experimentation, you could consider using GitHub Models. This platform offers free-tier access to both a playground and an API endpoint where you can query models from different providers, including GPT-4o and Llama 3. Using the Azure AI Inference SDK, you can call the GitHub Models endpoint directly from your code, test various models, tweak parameters, and observe responses in real time, helping you make an informed decision about which model best fits your application’s needs.

Once you’ve identified the right model and are ready to scale, you can seamlessly transition from GitHub Models to Azure AI Foundry by simply switching the endpoint in your code. This flexibility enables efficient prototyping and deployment within the Azure ecosystem, without requiring any code changes.

Model Deployment Types

Azure AI Foundry’s Model Catalog provides two main options for deploying models: Serverless APIs (Model as a Service or MaaS) and managed compute deployments. With MaaS, models are hosted on Azure’s managed infrastructure, providing seamless API access without requiring management of underlying resources. Managed compute deployments, on the other hand, offer greater flexibility by supporting a broader catalog of models while enhancing scalability, security, and operational efficiency. This approach is ideal for organizations that require more control over their AI models while still leveraging Azure’s robust infrastructure and integrations.

Serverless deployments in Azure AI Foundry can be configured in three ways, regional, data zone based, and global, allowing organizations to balance performance and compliance requirements.

Regional deployments place AI services in a specific Azure region, bringing them closer to end-users to minimize latency and improve responsiveness. This option is ideal for performance optimization in scenarios where compliance requirements are less restrictive.

Data zone deployments allow usage strictly within EU or US data zones. This is particularly valuable for organizations that must comply with regulations like GDPR, which impose strict data residency and processing requirements, while still benefiting from Azure’s managed infrastructure.

Global deployments take advantage of Azure’s worldwide infrastructure to dynamically route requests to the data center with the best availability at any given time. This delivers the highest default quotas, ensures consistent performance worldwide, and eliminates the need for manual load balancing.

By offering these three deployment modes, Azure AI Foundry enables organizations to choose the model placement strategy that best aligns with their operational and compliance priorities.

In addition to choosing where models are deployed, organizations can also decide how capacity is allocated through two serverless deployment modes: pay-as-you-go and provisioned throughput.

Pay-as-you-go provides on-demand access with standard capacity limits, ideal for variable or lower-volume workloads.

Provisioned throughput reserves dedicated processing capacity for a specific model, enabling much higher usage limits than the default and ensuring resources are available when needed.

Scalability is a critical factor for high-demand AI applications, and provisioned throughput is designed to address this. Available for a selection of models in the Azure AI Foundry catalog, it delivers consistent performance and predictable costs, particularly important for latency-sensitive or high-throughput workloads. By allocating Provisioned Throughput Units (PTUs), organizations can precisely align model capacity with workload demands, ensuring efficient resource utilization while maintaining reliability and performance.

It’s important to note that not all deployment or capacity options are available for every model in the Azure AI Foundry catalog. The choices, whether regional, data zone, global, pay-as-you-go, or provisioned throughput, depend on the specific model, so organizations should review each model’s documentation before planning their deployment.

Azure OpenAI Models

Azure AI Foundry enables the deployment of Azure OpenAI models as APIs in a serverless manner, allowing developers to integrate AI capabilities into their applications without managing infrastructure. The OpenAI models support multiple modalities, language, audio, image, video and embeddings, enabling a wide range of use cases.

Example 3-1. Python code example using the chat completions API

import os
from openai import AzureOpenAI

endpoint = "https://foundry-resource.cognitiveservices.azure.com/"
model_name = "gpt-5-mini"
deployment = "gpt-5-mini"

subscription_key = "<your-api-key>"
api_version = "2024-12-01-preview"

client = AzureOpenAI(
    api_version=api_version,
    azure_endpoint=endpoint,
    api_key=subscription_key,
)

response = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": "I am going to Paris, what should I see?",
        }
    ],
    max_completion_tokens=16384,
    model=deployment
)

print(response.choices[0].message.content)

For natural language processing, Azure OpenAI provides both classical GPT models and a newer generation of reasoning models. Classical models like GPT-3.5-Turbo, GPT-4, GPT-4.o, and GPT-4.1 are optimized for general-purpose language tasks. In contrast, the newer o-family of models, such as o1, o1-mini, o3, o3-mini, and o4-mini, are explicitly designed for advanced reasoning and problem-solving, offering improved performance on complex tasks and tighter integration with multimodal capabilities.

Developers can access Azure OpenAI models through multiple APIs tailored to different scenarios. The Chat Completions API is best suited for straightforward, multi-turn conversations, such as building chatbots or conversational interfaces that respond dynamically to user inputs (see Example 3-1). The Responses API (currently in preview) is designed for more advanced assistants that can maintain context across multiple sessions, follow complex instructions, and use built-in tools, such as web search, file search, and computer use, to retrieve real-time information, search documents, or perform computer operations on the user’s behalf (see Example 3-2). Unlike the Chat Completions API, it supports stateful interactions, allowing the AI to remember prior exchanges without requiring manual context management.

Example 3-2. Python code example using the code interpreter with the responses API

from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
)

client = AzureOpenAI(
  base_url = "https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
  azure_ad_token_provider=token_provider,
  api_version="preview"
)

instructions = "You are a personal math tutor. When asked a math question, write and run code using the python tool to answer the question."

response = client.responses.create(
    model="gpt-4.1",
    tools=[
        {
            "type": "code_interpreter",
            "container": {"type": "auto"}
        }
    ],
    instructions=instructions,
    input="I need to solve the equation 3x + 11 = 14. Can you help me?",
)

print(response.output)

In the audio modality, Azure AI Foundry offers a range of Azure OpenAI audio models designed for different scenarios:

Transcription – Whisper, GPT-4o-Transcribe, and GPT-4o-Mini-Transcribe deliver high-quality speech-to-text for both real-time and offline use cases.

Real-time voice interactions – GPT-4o-Realtime-Preview and GPT-4o-Mini-Realtime-Preview accept both text and audio as input and output, enabling fluid, fully conversational voice agents.

Non-real-time audio processing – GPT-4o-Audio-Preview and GPT-4o-Mini-Audio-Preview provide the same multimodal capabilities but are optimized for asynchronous tasks such as analyzing voice recordings or creating audio content like audiobooks.

Text-to-speech synthesis – GPT-4o-Mini-TTS, TTS-1, and TTS-1-HD generate natural-sounding voices in multiple personas and languages.

In addition, Azure AI Foundry also exposes Azure Speech Services, which provide STT, TTS, real-time conversational capabilities, and video avatars.

Beyond audio, Azure AI Foundry also exposes the gpt-image-1 and Sora models from Azure OpenAI. gpt-image-1 supports both text-to-image and image-to-image generation, enabling the creation or transformation of high-quality images. Sora produces realistic, dynamic videos from natural language descriptions and or a base initial images.

In the embeddings space, we can find several Azure OpenAI embedding models in the Azure AI Foundry catalog. Text-embedding-3-large and text-embedding-3-small are two of these models, designed to convert text into numerical vectors that capture semantic relationships between concepts, making them essential for applications requiring semantic understanding, such as retrieval-augmented generation (RAG), intelligent search, clustering, and content recommendation engines. Unlike generative models, embedding models are purpose-built for measuring and comparing meaning across text data rather than producing new content.

In summary, the Azure OpenAI models represent a core part of the first-party offerings in the Azure AI Foundry catalog, delivering enterprise-grade generative and embedding capabilities across text, audio, image, video, and semantic understanding. Whether used individually or in combination, they enable the creation of multimodal, context-aware, and highly interactive applications. Together, they illustrate the depth and versatility of Azure AI Foundry, setting the stage for developers and organizations to innovate at scale.

Additional Azure AI Foundry Capabilities

While model selection and deployment are central to building successful AI solutions, Azure AI Foundry also provides a range of complementary capabilities that simplify integration, enhance customization, ensure security, and accelerate experimentation. These features help teams move from prototyping to production with efficiency, governance, and flexibility.

The Azure AI Inference client library, supporting Python, C#, JavaScript, and Java, simplifies the integration of large language models into your applications by providing a unified, consistent API to access a broad range of models from providers such as OpenAI, Meta, Mistral, Microsoft, Cohere, and AI21 Labs, all deployed within Azure AI Foundry. It works seamlessly with models hosted via serverless APIs or managed compute endpoints and offers flexible authentication options through either API keys or Microsoft Entra ID. Additionally, the library includes built-in support for OpenTelemetry, enabling rich observability and tracing across your AI workflows for easier debugging and performance tuning.

To accommodate varying levels of expertise, Azure provides fine-tuning as a service for certain models, enabling teams to tailor AI models to specific data and requirements without extensive knowledge of underlying algorithms. This accelerates development and deployment, ensuring AI solutions align with organizational objectives. Azure AI Foundry supports fine-tuning across multiple model providers, granting access to the latest advancements in the field. For Azure OpenAI models, additional techniques such as Direct Preference Optimization (DPO) and Reinforcement Fine-Tuning (RFT) further enhance customization options. DPO enables efficient model alignment based on human preferences without requiring a separate reward model, while RFT enhances the o1-mini model’s reasoning capabilities in domains like science, coding, and mathematics through iterative feedback. These advancements provide organizations with greater flexibility in optimizing AI models for their specific needs while maintaining efficiency and performance.

Figure 3-5. Content Safety Filters in Azure AI Foundry

Security and responsible AI are central to Azure’s approach. AI Content Safety within Azure AI Foundry provides guardrails to detect and mitigate harmful content, ensuring AI applications meet ethical standards and protect user data. Content filters analyze both input prompts and AI-generated outputs to identify harmful material across categories such as violence, hate speech, sexual content, and self-harm. Prompt Shields defend against prompt injection attacks, whether direct jailbreak attempts or indirect attacks containing malicious instructions hidden in third-party content. These elements will be explored in more detail later in the book.

By leveraging Azure AI Foundry and its Model Catalog, along with the previously discussed factors, you will be able to efficiently identify, customize, and deploy AI models that best align with your technical requirements and business objectives, ensuring optimal performance and scalability for your applications.

We have seen that continuously gathering feedback and refining model selection ensures ongoing performance improvements. Implementing user feedback mechanisms and remaining adaptable to emerging models and technologies enable continuous optimization. Since choosing the right generative AI model is an iterative process, seeking expert consultation can help align business needs with the most suitable technologies, ensuring efficiency, scalability, and long-term success.

Prompt Engineering Best Practices

Laying the Foundations for Effective Prompts

Once you’ve selected the right model, the next critical factor in unlocking its full potential is how you interact with it, this is where prompt engineering comes in. Prompt engineering has rapidly emerged as a critical discipline in modern AI, combining creativity and structured methods to effectively guide language models toward accurate and contextually relevant outcomes. This dynamic practice involves carefully designing, testing, and iteratively refining input prompts, including system instructions, user requests, and specifying tone or style, to precisely steer model behavior toward targeted results. Prompt engineering continuously evolves alongside advancements in language models, from early foundational architectures to today’s sophisticated reasoning systems.

Clear communication is essential in prompt engineering. Starting with simple and direct prompts and gradually improving them based on the model’s responses allows for consistent progress. Providing detailed descriptions of the desired result, context, limitations, and expectations greatly reduces confusion. Clear prompts help ensure the model produces relevant and accurate answers.

The way instructions are worded also significantly affects the model’s responses. Using clear and positive language works better than negative or unclear instructions, which might confuse the model or lead to unexpected outcomes. Regular testing and adjustment help continuously refine the prompts, keeping them aligned with the desired results.

Structured language for your prompts is another important aspect. Using simple formats like Markdown or JSON helps maintain clarity and consistency. Structured prompts make it easier for the model to understand and follow instructions, resulting in more reliable responses.

Another important consideration is tailoring prompts to specific tasks or applications. Different tasks may require varying levels of detail or specific phrasing to achieve optimal performance. It is also beneficial to monitor and document the outcomes of different prompt approaches, creating a reference for future use.

When interacting with AI models conversationally (such as chat models), each message type plays a critical role. Typically, there are system messages, user messages, and assistant messages. System messages, often hidden from the user, set the initial context or behavior of the AI, such as defining its persona, tone, or special instructions (e.g., “You are an AI lawyer who responds politely with law references”). Because the system message influences every response, refining it can greatly affect the model’s output, often more significantly than adjusting user prompts.

In chat models specifically, prompt engineering may also involve modifying user messages by adding contextual elements either before or after the user’s actual prompt. This approach helps to clarify or reinforce context, guide the model’s focus, or explicitly state certain conditions the model must consider in its response.

Assistant or system messages are sometimes used in conversational prompts to provide examples of desired responses. By preloading conversation history with specific examples, you guide the model toward particular behaviors or response styles. This approach, is an example of few-shot prompting, and it helps the model understand how to answer future queries similarly. For instance, providing an example dialogue can illustrate the expected briefness or structure of responses.

Adjusting model parameters also significantly influences the outcomes. For instance, the temperature parameter controls randomness or creativity of responses. Lower temperatures produce more conservative, factual responses, while higher temperatures lead to more creative or varied outputs. Similarly, top-p (nucleus sampling) selects words based on cumulative probability, narrowing the possible word choices. Lower values restrict the model to highly probable words, making outputs predictable; higher values increase output diversity. Stop sequences define tokens that indicate where the model should end its response, helping produce clean, relevant outputs without unnecessary text.

Another powerful prompt engineering method involves function calling and tool integration. Modern AI models, like GPT-4o, can output structured function oriented data to invoke predefined external functions. Properly describing available functions and their parameters within the prompt or system message enables the model to use these tools accurately. For example, defining a function clearly (e.g., getting the weather for a specific location) allows the model to recognize when to utilize it and ensure the output aligns with the required schema.

Some models, such as those from OpenAI, provide parameters to enforce structured outputs directly. For example, developers can define a JSON Schema during an API call, which guides the model to produce outputs that strictly match the schema.Structured output techniques are particularly valuable when building applications that rely on machine-readable formats such as JSON, XML, or key-value pairs. By combining well-designed prompts with schema enforcement, developers can ensure consistency, avoid hallucinated or malformed outputs, and simplify downstream processing. This is especially useful when integrating LLMs into pipelines where their output needs to be parsed by other systems or used in decision-making.

For models that do not offer built-in structured output capabilities or that native schema enforcement is unavailable, there are libraries like Guidance that help guide and constrain model outputs using templated generation. Guidance allows developers to write prompts using a mixture of natural language and embedded control logic (such as variable placeholders, loops, and conditionals), effectively turning prompts into lightweight programs. This enables step-by-step, structured generation and can also help reduce hallucinations by constraining what the model can generate at each stage. It’s particularly helpful in use cases that require high output fidelity, like form-filling, API responses, or structured document creation.

Fine-tuned models are trained on specific data, like medical or legal texts, and set up to follow clear instructions. They understand simple, natural prompts easily because of their better grasp of context. They also use fewer tokens to complete tasks, making them more efficient. Non fine-tuned models that are not fine-tuned often need more detailed prompting. They usually work better with structured formats, clear output rules, or several examples to guide their responses. Without special tuning, these models may require extra in-prompt guidance. It is important to note that fine-tuning is not a cure-all for prompt engineering challenges. The process is complex and expensive, and even fine-tuned models need clear, well-organized prompts since prompt engineering cannot give a model information it has not learned.

Image and video generation models follow similar principles of prompt engineering as text-based models. Recent techniques include collaborating with LLMs to create advanced JSON context profiles (structured description files) that enable the generation of richer and more complex images or videos. Additionally, it has been discovered that the latest generation of image-to-video models exhibits an emerging capability to interpret annotations directly on image prompts, allowing creators to guide motion, style, and other aspects of the generated video more precisely.

Prompt engineering techniques

Additionally to all the best practices we have discussed so far, there are a series of prompt engineering techniques that are applicable to any complex task to further optimizing performance:

Zero-shot prompting: Involves directly querying the model without providing examples, relying solely on clear, precise instructions. This method is quick and efficient for straightforward tasks.

Few-shot prompting: Offers the model a small number of example responses to guide its output, enhancing accuracy through illustrative context and clearer expectations.

Prompt chaining: Breaks down complex tasks into smaller, sequential prompts, where the output from one prompt becomes the input for the next. This method systematically addresses multifaceted challenges by guiding the model step-by-step.

Chain of Thought (CoT): Encourages models to explicitly outline their reasoning steps, improving logical coherence and accuracy in complex reasoning or problem-solving tasks. Just using phrases like “let’s think step by step” could help the model generate a more structured, transparent, and logically coherent response, increasing overall accuracy and interpretability, especially for intricate problems requiring multi-step thinking.

Self-reflection: Directs the model to critically assess its own responses, identifying errors or suggesting areas for improvement. This fosters continuous improvement in model outputs.

Reflexion: Builds upon self-reflection by prompting iterative evaluation and adjustment, allowing the model to refine its responses based on internal feedback loops.

ReAct (Reasoning and Acting): Integrates reasoning steps with actionable outputs, enabling the model to both think through problems and take external actions such as retrieving data from APIs or external databases.

Constrained prompting: Explicitly sets strict boundaries or criteria for responses, refining precision, relevance, and adherence to specific requirements.

Ensemble prompting: Combines multiple prompting strategies to aggregate strengths from various approaches, resulting in more robust and accurate outputs.

Persona prompting: Assigns specific roles or personas to the model, enabling it to generate responses from defined perspectives or contexts, adding depth and nuance.

While these techniques can significantly enhance the quality, precision, and reliability of model outputs, their effectiveness often depends on the specific model, its architecture, and its training data. Some models respond exceptionally well to certain strategies, like Chain of Thought or ReAct, while others may show little improvement or even degrade in performance. Experimentation, measurement, and iteration are therefore essential to identify which techniques best align with your chosen model’s capabilities and the task at hand.

Prompt engineering Key Takeaways

Iterative testing plays an essential role in prompt refinement. Regularly evaluating and adjusting prompts based on output quality ensures continuous improvement and alignment with desired outcomes. Iterative cycles of testing, feedback, and refinement significantly boost overall efficacy and adaptability in real-world applications.

As mentioned earlier, prompt strategies might also differ based on the model type. When it comes to reasoning models, prompt engineering requires a well defined balance between guiding the model through complex problem-solving and allowing it enough autonomy to leverage its built-in reasoning capabilities. For instance, while traditional models might benefit from explicit, step-by-step instructions (like “let’s think step by step”) or few shot prompting, reasoning models often perform better with simple, more concise, zero-shot prompts. This is because these models are specifically designed to allocate computational resources toward internal deliberation, breaking down problems into manageable sub-tasks without the need for overly detailed guidance

In practical applications, this means that prompt engineering for reasoning models must be adaptive. Developers are advised to start with straightforward precise prompts and only incorporate additional context or structure if the initial output fails to meet quality standards.

As models evolve and incorporate hybrid reasoning capabilities, the art of prompt engineering will shift towards facilitating the model’s natural thought processes rather than micromanaging them. This evolution not only reduces the cognitive load on the prompt engineer but also paves the way for more intuitive, user-friendly interactions with AI systems, making advanced reasoning accessible without specialized knowledge.

Prompt engineering is a continuously developing practice shaped by ongoing advancements in AI. It requires clear communication, precise instructions, structured approaches, and collaborative learning, ensuring each interaction with language models is effective, intentional, and productive. Collaboration and sharing best practices among practitioners can further enhance the effectiveness of prompt engineering. Learning from others’ experiences and approaches can introduce new ideas and improvements, speeding up the learning process and innovation.

In conclusion, effective prompt engineering involves a thoughtful combination of clear, structured instructions, explicit reasoning cues, and iterative refinement. However, to operationalize prompt quality at scale, it should be integrated with robust software engineering practices such as version control, continuous testing, and proactive monitoring. Version control ensures traceability and ease of collaboration, continuous testing validates prompt effectiveness across scenarios, and monitoring provides insights into prompt performance and user interactions over time. Combining these approaches ensures prompt robustness, reliability, and continuous improvement in production environments.

Note

In recent months, a new term called context engineering has gained significant traction. It extends from the concepts of prompt engineering, moving beyond the crafting of individual instructions to take a holistic, system-level approach to what the model actually sees. Context engineering is about carefully managing all the content sent to the LLM, including memory history, retrieved documents, system prompts, tool calls, and user information, curated and structured to fit within context window constraints. This discipline ensures that models receive the right information, in the right format, at the right time, scaling far beyond individual prompts and helping improve both reliability and performance.

Prompt Engineering in Azure AI Foundry

Azure AI Foundry offers a comprehensive platform to refine and elevate your prompt engineering practices through an integrated suite of tools and interfaces. For instance, the platform’s chat playground interface enables you to quickly test and iterate your prompts in a controlled environment (see Figure 3-6). This immediate feedback loop allows you to assess prompt performance, troubleshoot issues, and gain initial insights before scaling your experiments further.

Figure 3-6. AI Foundry Chat Playground

In addition to the chat playground, Azure AI Foundry includes a built-in evaluation experience for both manual and automated reviews. In the portal, you can create evaluation runs against a model or a test dataset, choose quality and safety metrics, map your dataset columns, and submit batch tests. Supported metric families include AI-assisted quality (for example groundedness, relevance, coherence, fluency, GPT similarity), classic NLP metrics (F1, ROUGE, BLEU, GLEU, METEOR), and risk and safety checks (self-harm, hate or unfairness, violence, sexual content, protected material, indirect attack). Datasets can be CSV or JSONL, and you can evaluate either generated outputs you already captured or have the system generate them during the run.

Figure 3-7. AI Foundry Testing Criteria

Results are visualized in the Evaluation page, where you can inspect sample-level metrics, give quick human feedback with a thumbs up or down, and compare runs side by side (see Figure 3-8). The comparison dashboard lets you set a baseline, view deltas, and focus only on differing rows to speed up iteration. There is also guidance for measuring jailbreak vulnerability by comparing defect rates on adversarial datasets.

Figure 3-8. AI Foundry Metric Dashboard for AI Quality metrics

For code-driven workflows, the Azure AI Evaluation SDK lets you run the same evaluators locally or in the cloud, log results back to your project, and extend with custom evaluators. Built-in evaluators cover general quality, RAG-specific metrics like groundedness and document retrieval, agentic behaviors such as tool-call accuracy, and content safety. You install the azure-ai-evaluation package, run on single items or large JSONL datasets, and track runs in your project for consistent reporting.

Taken together, the playground for quick trials, the portal’s evaluation runs for structured testing, and the SDK for automated batch and CI flows give you an integrated loop for prompt iteration, measurement, and regression checking inside Azure AI Foundry.

Summary

Selecting the right model is the first strategic decision in building a generative AI solution, as it defines the capabilities, limitations, and operational parameters of your application. Throughout this chapter, we explored how to navigate the Azure AI Foundry Model Catalog, compare performance benchmarks, and evaluate trade-offs such as latency, cost, and compliance requirements. We also reviewed the different deployment options available, from serverless endpoints to provisioned throughput, and highlighted additional AI Foundry features that can streamline experimentation and integration.

Prompt engineering plays a pivotal role in shaping the behavior and outputs of language models. We have seen that designing prompts is only part of the equation, understanding how to assess their performance is equally critical. Evaluations allow us to systematically measure prompt quality, optimize outcomes, and ensure alignment with intended use cases. This leads us into the broader discipline of GenAIOps, where evaluation becomes an integral part of the lifecycle of generative AI systems.

In the next chapter, we’ll dive deeper into how to evaluate prompts and model responses at scale, automate quality control, and implement GenAIOps best practices to operationalize generative AI responsibly and efficiently.

Chapter 4. Retrieval Augmented Generation (RAG)

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fourth chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

In this chapter, we will dive into one of the most impactful applications of generative AI: Retrieval-Augmented Generation (RAG). We will examine its core principles, explore different implementation variants tailored to specific use cases, and focus on Azure-based solutions for deploying RAG effectively.

Traditional generative AI models, despite their impressive capabilities, have inherent limitations that often hinder their performance in practical, real-world scenarios. These models rely exclusively on static knowledge acquired during training, typically having a knowledge cutoff date beyond which no new information is incorporated. Consequently, they are unable to access or utilize information that emerges after training concludes. Additionally, traditional generative models are susceptible to producing plausible-sounding but incorrect information, commonly referred to as hallucinations, particularly when responding to detailed or highly specialized queries.

Addressing these limitations, Retrieval-Augmented Generation (RAG) offers a transformative solution by integrating external knowledge sources directly into the generative process. Unlike standalone generative models, RAG dynamically retrieves relevant and current information at the moment of inference, significantly enhancing the accuracy and timeliness of the responses. By grounding outputs in real-time data, RAG effectively reduces the risk of outdated or incorrect information.

Fundamentals of RAG

The concept of retrieval itself has deep historical roots, playing a crucial role in information retrieval systems and search engines for decades. Retrieval methods have evolved significantly from basic keyword searches to advanced semantic searches utilizing vector databases and embeddings. Generative models have also undergone substantial evolution, moving from simple statistical methods to sophisticated transformer-based Large Language Models (LLMs). The convergence of these two historically distinct fields, retrieval and generative modeling, has paved the way for RAG, marrying the strengths of both to deliver superior outcomes.

The benefits of RAG extend beyond simply enhancing accuracy. Retrieval-augmented systems offer improved transparency and explainability, as generated outputs can be traced directly back to specific, authoritative sources, further emphasizing the importance of grounding in reliable information. Furthermore, RAG architectures offer greater flexibility and scalability, enabling easy updates and expansions to the knowledge base without the need for retraining the generative model. This modular design approach significantly reduces computational overhead and simplifies maintenance, making RAG particularly valuable in enterprise applications where accuracy, agility, and efficiency are critical.

The RAG Process

To achieve effective grounding, the RAG process begins with the indexing phase. During this phase, documents are optimized by segmenting them into coherent chunks, a process known as chunk optimization. Techniques range from fixed-size chunking with overlaps to more sophisticated semantic chunking, which preserves logical structures like paragraphs, tables, or code blocks.

The most fundamental approach is fixed-size chunking with overlap. The text is divided into regular token windows (commonly 150-400 tokens), while repeating the last 10-20 percent of each window at the start of the next. Overlap preserves cross-sentence context so that information split across a boundary can still be retrieved, which tends to raise recall. Window size presents a classic trade-off: small slices speed up search and shrink storage but disperse related ideas over many hits, whereas large slices keep concepts together at the cost of higher latency and noisier embeddings. Both window and overlap sizes are therefore tuned empirically against retrieval quality metrics.

Structure-aware chunking takes a more linguistically and layout-informed approach, dividing documents at meaningful textual boundaries such as sentences and paragraphs. This strategy enhances semantic clarity while also making it easier to isolate and extract document elements like tables, lists, or figures. Because sentences and paragraphs are usually self-contained units of meaning, this method tends to produce coherent chunks that align well with the original intent of the author. However, long sections must still be carefully handled, typically by recursively splitting them into smaller paragraphs or groups of sentences to respect token limits, while preserving their internal coherence.

Semantic chunking builds on structure-aware methods by analyzing the content of adjacent sentences or paragraphs to determine whether they belong together. This is often achieved by generating embeddings, for each sentence or span and comparing their similarity. A sharp drop in similarity can indicate a natural breaking point, while closely related spans are grouped into a single chunk. This technique creates semantically cohesive units that improve the precision of retrieval, especially in content-rich or multi-topic documents. It does, however, come with higher preprocessing costs and increased complexity, as changes in the original text may require recomputing embeddings and re-evaluating chunk boundaries.

Following chunk optimization, indexing organizes the content efficiently, typically involving storing metadata and document text in structured or semi-structured databases. Usually, embeddings are also generated for each chunk and stored alongside traditional indexing data within hybrid databases capable of both classical and vector search.

Query manipulation or transformation methods further enhance retrieval accuracy by modifying user queries. This can involve expanding queries with synonyms, correcting spelling errors, or even generating multiple queries from a single user input to ensure comprehensive retrieval across different database segments or indexing methods.

Another important aspect of pre-retrieval is query routing, where the processed queries are directed toward the most appropriate database or index. This becomes crucial when multiple specialized indexes exist, requiring intelligent decisions about which index is most relevant to a particular query. Additionally, query routing might involve sending queries to different LLMs based on factors such as complexity, specificity, or the type of answer required.

The retrieval phase focuses on fetching relevant content based on the processed queries. Typically, a fixed number of top documents or chunks (n documents) are retrieved, initially based on their relevance scores. Retrieval can involve classical search methods which rely on precise keyword matching. Alternatively, vector search methods use semantic embeddings to identify content that is contextually similar to the user’s query, even without exact keyword matches. Hybrid search combines classical and vector methods to leverage the strengths of both approaches. These retrieved documents or chunks are then ranked based on their relevance. Retrieval can be further refined using filtering techniques based on metadata such as dates, document types, or reliability indicators, ensuring the returned content meets specified criteria.

After the retrieval phase, additional refinements can be applied to further enhance the results. Re-ranking retrieved results based on their relevance is one possible enhancement but is not always necessary. Another optional improvement involves prompt compression, condensing the retrieved information to fit within the token constraints of the LLM, maintaining the essential context while reducing redundancy.

Finally, during the generation phase, the carefully retrieved and refined information is combined with the user’s query and presented to the LLM. The LLM generates responses grounded exclusively in the shared information, effectively ensuring accuracy and relevance. In this phase, prompt engineering is instrumental, guiding the model to produce specific and targeted responses by carefully structuring and wording the prompts to achieve the desired outcomes.

Figure 4-1. Classical Azure RAG High Level Architecture. Ref: https://learn.microsoft.com/en-us/azure/developer/python/get-started-app-chat-template

RAG in Azure

You have just seen that Retrieval-Augmented Generation (RAG) improves large language model outputs by bringing in relevant external data. In Azure, implementing RAG involves data ingestion, indexing, orchestration, and other components. However, when it comes to managing the data that supports retrieval, there are several key database and vector store options available in Azure, each with its own strengths and ideal use cases.

The most commonly used and recommended service for RAG on Azure is Azure AI Search. It combines full-text, vector, and semantic search in one engine, making it very effective for finding the most relevant documents or content chunks based on a user query. It supports hybrid search, which blends keyword matching and vector similarity, and has features like semantic ranking to boost the relevance of results. Azure AI Search also offers integrated capabilities to automatically chunk documents and generate embeddings using Azure OpenAI models, reducing the need for separate data preparation pipelines. This integration, combined with a scalable and managed architecture, makes it a common choice for most RAG implementations in Azure. We’ll explore this service in more detail later.

Another option is Azure Cosmos DB, a globally distributed NoSQL database that now supports native vector search. This means that you can store embeddings directly alongside JSON data and run similarity queries with high performance. It’s particularly useful when your application already uses Cosmos DB for operational data and you want to bring in semantic search without adding another database. Cosmos DB supports multiple indexing strategies such as HNSW (Hierarchical Navigable Small World) and DiskANN, giving flexibility depending on the size and nature of your data.

Note

HNSW (Hierarchical Navigable Small World) and DiskANN are approximate nearest neighbor (ANN) indexing strategies. HNSW runs in memory, offering very fast queries for smaller to medium datasets, while DiskANN leverages SSDs to handle much larger datasets, making it better suited when data doesn’t fit in memory.

For teams already using PostgreSQL, Azure Database for PostgreSQL offers a way to add vector capabilities using the open-source pgvector extension. With pgvector, you can store and search embeddings within your relational data, allowing you to mix SQL logic with vector similarity in a single query. It’s a popular choice for developers who want to stay within a familiar open-source stack but still add semantic search functionality.

Finally, Azure SQL Database has introduced native vector support in preview. This feature adds a new VECTOR data type and functions for similarity search, enabling you to run vector-based queries inside a traditional SQL environment. This is especially helpful for enterprises that have standardized on Azure SQL and want to integrate RAG features without migrating their data to a new system. While still in early stages, this integration allows for hybrid scenarios that mix text search and vector similarity within the same engine.

Each of these options offers a different approach to managing and retrieving knowledge in RAG systems. Azure AI Search remains the most complete and production-ready solution, but Cosmos DB, PostgreSQL with pgvector, and Azure SQL Database provide flexible alternatives depending on your architecture, data, and performance needs. Refer to Figure 4-2 to better understand what would be the main factors influencing what type of database you would use for your RAG system.

Figure 4-2. Azure RAG Database & Search Decision Tree. Ref: Microsoft Learn.

In the following sections of this chapter, we will explore how to implement Retrieval-Augmented Generation (RAG) with the ‘Azure OpenAI on Your Data’ capability, and directly integrating with Azure AI Search.

Azure OpenAI on your Data

Azure OpenAI on your data is a managed feature of the Azure OpenAI service that enables retrieval-augmented generation (RAG) by grounding responses on your private knowledge sources. In practical terms, you can chat with your data, ask questions and get answers with citations from your content.

Azure OpenAI on your data works by intercepting your chat requests and performs an internal RAG flow: it searches your indexed data for relevant information and injects those results into the prompt that the GPT model sees. This way, the model’s responses are augmented with factual snippets from your data, improving accuracy and grounding. You don’t have to orchestrate the search and prompt construction yourself, the Azure OpenAI service handles it and returns a response with references. You can access Azure OpenAI on Your Data via a REST API, SDK, or the Azure AI Foundry web interface (see Figure 4-3).

Figure 4-3. Screenshot of Azure AI Foundry Chat playground with a red arrow highlighting ‘Add a data source’ under ‘Add your data’ in the left panel

You can ingest data in various formats (text, PDF, Office files, etc.) and from various sources using built-in connectors. Azure OpenAI on your data allows you to pull content from an Azure Blob Storage container or a public URL, or otherwise connecting it directly to an existing index in Azure AI Search, Elasticsearch, Azure Cosmos DB for MongoDB, or MongoDB Atlas. If the raw data needs to be indexed, it is transformed into chunks of text. Each chunk is transformed into an embedding vector using an Azure OpenAI embedding model (like text-embedding-3-large) and stored in a search index (either a new one it creates in your Azure AI Search resource, or an existing index you link). Additional metadata fields (e.g., title, URL) can be indexed and linked to retrieved results, allowing clear source references in answers.

Once you have connected a data source and built an index, using Azure OpenAI on Your Data is straightforward. For example, with the Azure OpenAI Chat Completions API, you can specify the attached data source in your request. The service will then automatically do the RAG steps before returning the answer. In code, this looks like the following (in Python):

import os
import openai

# Set Azure OpenAI endpoint and API key
openai.api_type = "azure"
openai.api_base = os.environ["AZURE_OPENAI_ENDPOINT"]  # e.g. "https://your-resource.openai.azure.com/"
openai.api_version = "2024-06-01"  # a version that supports the extensions
openai.api_key = os.environ["AZURE_OPENAI_API_KEY"]

# Define the deployment name of your Azure OpenAI chat model (e.g., gpt-35-turbo or gpt-4)
deployment_id = os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"]

# Call the Azure OpenAI Chat Completions API with a data source (Azure AI Search index)
response = openai.ChatCompletion.create(
    deployment_id=deployment_id,
    messages=[ {"role": "user", "content": "What are my available health plans?"} ],
    dataSources=[                           # Attach the Azure AI Search index as a data source
        {
            "type": "AzureCognitiveSearch",
            "parameters": {
                "endpoint": os.environ["AZURE_AI_SEARCH_ENDPOINT"],
                "key": os.environ["AZURE_AI_SEARCH_API_KEY"],
                "indexName": os.environ["AZURE_AI_SEARCH_INDEX"]
            }
        }
    ]
)
answer = response['choices'][0]['message']['content']
print(answer)

The snippet above calls the chat model deployment with a user’s question and provides an Azure AI Search data source, including the search service endpoint, API key, and index name. The Azure OpenAI service uses these parameters to query the index for relevant chunks (e.g. top N documents) and then includes those in the prompt to the model. The final answer (with references) is returned in the answer variable. We didn’t have to manually perform any search, the model call with dataSources triggers the retrieval augmentation internally. This is a simple example; in practice you can refine it by specifying options like how many documents to retrieve or a “strictness” setting for search matching. Azure’s SDKs for .NET, Java, and Python provide higher-level interfaces to configure data sources as well.

Figure 4-4. Chat playground with Azure AI Search connected—grounded answers with document references.

In summary, OpenAI on Your Data in Azure provides a managed RAG solution: you bring your data, and Azure takes care of indexing it and orchestrating the retrieval+generation loop. It’s ideal for quickly creating a ChatGPT-like assistant over private knowledge with minimal development overhead. However, it trades some flexibility for convenience. In the next section, we explore a more custom approach – building a RAG system using Azure AI Search directly, which offers more control and tailoring.

RAG with Azure AI Search

Figure 4-5. Azure AI Search: end-to-end data flow-ingest, index, query, deliver. Ref : https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search

While Azure OpenAI on your data is an out-of-the-box solution, you may want to build a custom RAG architecture using Azure’s building blocks for greater flexibility. Azure AI Search is the core component for the retrieval piece. In a RAG system on Azure, Azure AI Search typically serves as the vector database and search engine for your documents, while Azure OpenAI provides the generative AI that produces answers. As the developer, you’ll need to create the integration layer that connects the two: Indexing the data, issuing queries to the search index, feeding results into the OpenAI model prompt and optimizing the answer generation. This section walks through how to build such a system, covering the architecture, data ingestion and indexing, embedding generation, and query-time retrieval and generation.

Data Ingestion and Indexing

Figure 4-6. Stages of Indexing. Ref: https://docs.azure.cn/en-us/search/search-indexer-overview

In order to implement a RAG pipeline, you first need to populate a search index with your content. An Azure AI Search index is a schema-defined store optimized for text search and retrieval. Each document in the index will typically have fields like id, content (the text), embedding (the vector embedding of that text), and any metadata (title, source, etc.).

Begin by defining the fields that will make up your index. For a RAG scenario, include one or more searchable text fields and a vector field to hold the embeddings. You can build the index directly in the Azure Portal or automate the process with code or the CLI.

Figure 4-7. Creation of an Index in the Azure Portal. Ref: https://docs.azure.cn/en-us/search/search-get-started-portal

In the following example we will be using the Python SDK to define an index with the fields id (key), content (searchable text), contentVector (embedding vector for the content and sourceUrl (filterable metadata).

from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import SearchIndex, SimpleField, SearchableField, VectorSearch, VectorSearchAlgorithmConfig

index = SearchIndex(
    name="my-knowledge-index",
    fields=[
        SimpleField(name="id", type="Edm.String", key=True),
        SearchableField(name="content", type="Edm.String"),            # text content
        SimpleField(name="contentVector", type="Collection(Edm.Single)",
                    searchable=True, vector_search_dimensions=1536,    # vector dim
                    vector_search_configuration="my-vector-config")
    ],
    vector_search=VectorSearch(algorithm_configurations=[
        VectorSearchAlgorithmConfig(name="my-vector-config", kind="hnsw")  # using HNSW algorithm for ANN search
    ])
)
index_client = SearchIndexClient(endpoint, AzureKeyCredential(api_key))
index_client.create_index(index)

Once you have defined the schema for your index, the next step is to load your data into it. Azure AI Search provides two primary approaches to ingest data, depending on your level of control and automation needs:

Push approach: This method involves programmatically sending your data into the search index using Azure AI Search REST APIs or SDKs. It offers full flexibility, allowing you to preprocess, transform, and control how and when data is indexed. This is typically used when your data resides in custom applications, dynamic sources, or when complex ingestion logic is required.

Pull approach (indexers and skillsets): In this scenario, Azure AI Search indexers connect to supported data sources (such as Azure Blob Storage, Azure SQL Database, Cosmos DB, etc.) and automatically crawl and extract content to populate the index. You can enrich the data during ingestion using skillsets, which apply AI-powered transformations like OCR, language detection, entity recognition and other cognitive skills

One of the key additions of using the pull approach is the Azure OpenAI Embedding skill: the indexer can invoke a deployed embedding model to vectorize each document or chunk on the fly, writing the resulting vectors directly into the index’s vector field. This built-in “integrated vectorization” means you no longer need a separate preprocessing script for embeddings. For example, an indexer pointed at a Blob Storage container full of PDFs might chain an OCR skill (to read scanned pages), the Text Split skill (to chunk large documents), and the Embedding skill (to generate a vector per chunk). The entire pipeline is described in a skillset JSON file and can run on demand or on a schedule, making it ideal for static or semi-structured content sources that need to be RAG-ready out of the box.

Figure 4-8. Indexer Configuration in the Azure Portal.Ref: https://docs.azure.cn/en-us/search/search-get-started-portal

Behind the scenes, adding a vector field prompts Azure AI Search to build an HNSW (Hierarchical Navigable Small World) graph that makes nearest-neighbour look-ups fast. The default graph density and search breadth are tuned for a good recall-versus-latency balance, but you can push either dial through the REST API: a denser graph or broader search window yields higher accuracy at the cost of indexing time or query latency, while trimming them has the opposite effect. In most cases the out-of-the-box settings are enough, so you only tweak them when you have hard latency targets or must squeeze out a few extra points of recall.

In summary, by the end of the ingestion and indexing step, you should have an Azure AI Search index that contains all your content in a vectorized form and is ready to be queried for relevant chunks. Whether you let an indexer-and-skillset pipeline handle everything end-to-end, or write a custom push workflow, the outcome is the same: a vector-ready Azure AI Search index that your RAG application can query immediately

Retrieval and generation workflow

Once your data has been indexed, the query and response generation workflow becomes the core engine of your Retrieval-Augmented Generation (RAG) system. This workflow bridges the user’s natural language question with the relevant content in your knowledge base, transforming retrieved information into a coherent, context-aware answer. The typical steps in this process include: query understanding and enhancement, retrieval, retrieval post-processing, and response generation.

Figure 4-9. Querying Azure AI Search in Search Explorer with JSON results and source citations. Ref: https://docs.azure.cn/en-us/search/search-explorer

In the query understanding and enhancement phase, the system may preprocess or refine the user’s original question to improve retrieval accuracy and relevance. Azure OpenAI can assist in this step by rephrasing verbose, ambiguous, or conversational questions into more focused and search-optimized queries, such as generating key phrases or summarizing the user’s intent into concise prompts.

Additionally, the system can enrich the query by incorporating metadata from other enterprise systems (such as user profile data, session context, or application state) or by inferring user-specific preferences and context dynamically. This allows the system to personalize and tailor the retrieval scope. In some implementations, certain elements of the query, such as timeframes, locations, or specific entities, can also be extracted and used as filters during retrieval, narrowing down the document set and increasing precision. However, due to Azure AI Search’s natural language, semantic, and vector search capabilities, such preprocessing remains optional for many scenarios. More advanced workflows might also leverage large language models to classify the query, detect user intent, or decide dynamically whether a search is required at all, for example in cases where the user’s query is purely conversational or requires direct generation without retrieval.

During the retrieval step, the system queries the Azure AI Search index to identify the most relevant documents or passages that can enrich the generative response. Azure AI Search supports several complementary retrieval methods, each suited for different scenarios and levels of sophistication:

Keyword Search: This traditional approach leverages full-text search techniques to retrieve documents containing exact terms or phrases. It excels in cases where precision is paramount, such as looking up product codes, technical terms, or specific named entities.

Semantic Search: Using Azure AI Search’s semantic ranker feature, results from keyword or hybrid searches can be re-ranked based on Microsoft’s proprietary language understanding models. This step enhances result relevance even when exact keywords are missing, enabling the system to better understand user intent and return contextually appropriate documents.

Vector Search: In this approach, both documents and queries are represented as high-dimensional vectors using embeddings. Retrieval is based on semantic similarity, measured by the closeness of vectors in the embedding space. Vector search is particularly powerful for handling ambiguous, open-ended, or concept-heavy queries where the exact wording might differ significantly from the stored content.

Hybrid Search (Keyword + Vector): This method combines keyword and vector searches by executing both in parallel. The results are merged using algorithms such as Reciprocal Rank Fusion (RRF), which blends the strengths of lexical matching and semantic similarity, ensuring a more comprehensive set of relevant results.

Hybrid Search with Semantic Ranking: The most advanced option, this approach builds upon hybrid search by applying the semantic ranker on top of the merged results. This multi-layered refinement process yields highly relevant and context-aware documents, providing the best of exact matching, semantic understanding, and nuanced ranking.

Selecting the appropriate retrieval method depends on the specific use case, data type, and user expectations. In practice, many enterprise-grade RAG systems employ hybrid search with semantic ranking as the default pattern, balancing performance, accuracy, and user satisfaction.

query = "What are the symptoms of long COVID?"  # user question
# Get embedding for the query (must use same model as index)
query_vector = openai.Embedding.create(input=query, engine="embedding-model")["data"][0]["embedding"]
results = search_client.search(
    vector_queries=[{
        "kind": "vector",
        "value": query_vector,
        "fields": ["contentVector"],  # search in the contentVector field
        "k": 5                        # retrieve top 5 nearest chunks
    }]
)
top_chunks = list(results)

In this snippet, we embed the user’s query and ask Azure AI Search for the 5 most similar content vectors. The results come back as an iterable of documents, which we convert to a list. Each result likely contains the content text and maybe metadata.

Once the relevant documents or passages are retrieved, the system moves into the retrieval post-processing and response generation phase. In this step, the retrieved content can be further filtered, cleaned, or enriched before being injected into the prompt for the generative model. Post-processing techniques might include deduplication, removal of certain content, or applying business logic to prioritize certain sources over others.

The cleaned and enriched content is then assembled into a structured prompt that provides the generative model with the necessary information to answer the user’s question accurately and in a context-aware manner. This step is critical, as the way the instructions and content are framed will directly influence the quality, completeness, and trustworthiness of the model’s response. Various prompt engineering techniques should be applied, such as including explicit instructions to the model to cite its sources, presenting the retrieved snippets in a clear and consistent format, and adding guidance that limits the model’s response to cases where sufficient evidence is present.

Handling citations is a key part of this process. The prompt should clearly instruct the model to include citations alongside the generated answer, using patterns like “[Source: Document 1]” that refer back to the retrieved snippets. To support this, retrieved passages should be presented with their accompanying metadata, such as document titles, authors, dates, or source IDs, making it easier for the model to incorporate accurate references. However, because language models may sometimes omit, misplace, or hallucinate citations, some implementations might include a post-processing step where the system programmatically attaches the citations. This ensures that the final answer is not only informative but also transparent and easily traceable to the original sources, an essential requirement in enterprise, legal, or regulated scenarios.

Once you have the prepared prompt, send it to the model, typically via the Azure OpenAI API, and receive the generated answer, which is then delivered back to the user. By combining structured prompting, metadata inclusion, and post-generation citation handling, the system ensures that responses remain grounded, trustworthy, and aligned with enterprise-grade standards of reliability and accountability.

[Code example to come]

Note

Designing a custom RAG pipeline with Azure AI Search and Azure OpenAI allows you to finely tune system behavior, but it also requires careful attention to responsible AI practices. It’s essential to incorporate safeguards such as content filtering to moderate both user queries and model responses, especially to detect and block harmful content or personally identifiable information (PII). Additionally, implementing document-level access controls ensures that users only receive information they are authorized to view, an important consideration for enterprise and multi-tenant applications. See the GenAIOps chapter for a deeper exploration of these practices, where you’ll examine how to scale and operationalize LLM-based systems responsibly.

Comparison: OpenAI on Your Data vs. Azure AI Search RAG

Azure OpenAI “On Your Data” provides a fully managed RAG experience, automating ingestion, chunking, vectorization, ranking, prompt orchestration, security trimming, and inline citations behind a single REST/SDK endpoint, allowing you to focus primarily on the user experience. In contrast, building a RAG solution with Azure AI Search and the Azure OpenAI API gives you fine-grained control over every component such as chunk size, embedding model, classic, vector or hybrid search, semantic ranker or prompt templates; but requires you to design, operate, and maintain each part of the pipeline independently. See Table 4-1 for a comparison of each option’s capabilities.

Table 4-1. Comparison between Azure OpenAI on your Data vs DIY Azure RAG

Capability / Aspect | Azure OpenAI “On Your Data” (managed) | DIY RAG with Azure AI Search + Azure OpenAI
Integration approach | End-to-end managed chat extension that couples Azure AI Search, retrieval, ranking and GPT inference under one API. | You assemble the pipeline: index docs in AI Search, retrieve top passages, then pass them as context to the Chat Completions API.
Ease of setup | “Few-click” setup in Azure AI Foundry portal or one API call; sensible defaults picked. | Must create data source, index, skillset and orchestration code; higher initial effort.
Ingestion & chunking | Automatic extraction, 1024-token default chunks, vectorisation run by the built-in indexer. | Choose push or pull (indexer) mode and define your own chunking strategy or Text Split skill.
Embedding generation | Uses the managed Azure OpenAI embedding model you select. | Full choice: call any OpenAI or custom model inside a skillset or pre-compute vectors offline.
Index creation & tuning | Index created with sensible defaults; limited knobs exposed. | Control vector dimensions, HNSW parameters, BM25, synonym maps, RRF weighting, semantic ranker, etc.
Prompt engineering | Service auto-constructs retrieval-augmented prompts and injects citations; you mainly send the user message. | You design the entire prompt template (system/instruction, retrieved passages, user input, tool calls, etc.).

In summary, Azure OpenAI on Your Data provides an out-of-the-box, managed experience that quickly connects your enterprise data to powerful language models with minimal fuss. In contrast, using Azure AI Search offers a more DIY approach for RAG, where you manually integrate search results with LLM-generated responses, granting you tighter control over each part of the pipeline at the cost of additional development work

Iterative Optimization of your RAG pipelines

Improving a retrieval-augmented generation pipeline on Azure is a never-ending, iterative journey. You can use Azure AI Foundry to score each answer for retrieval accuracy, relevance, groundedness (the extent to which a model’s response is accurate and verifiably supported by the retrieved sources), content safety, estimated cost, and more (see Figure 4-10). Alongside these metrics, you can collect human feedbackthrough a thumbs-up / thumbs-down control, ideally an optional free‑text box or a checkbox that lets users flag missing sources, hallucinations, or incomplete answers. Examining the quantitative scores together with this qualitative feedback quickly reveals which of the four recurring pain points deserves attention next: quality and grounding (ensuring responses are factually accurate and supported by retrieved sources), relevance tuning and retrieval effectiveness (making sure the right documents are surfaced), latency (minimizing response time to maintain usability), or cost (minimizing tokens use and retrieval expenses). You can then deploy prompt refinements, chunk-size adjustments, and index reconfigurations then measure again, forming an iterative improvement loop that steadily sharpens the system’s responses.

Figure 4-10. Automated Evaluations in AI Foundry

Quality and grounding issues are often the first symptoms to surface when a RAG system underperforms. These problems typically appear as low groundedness scores in Azure AI Foundry or as user reports flagging hallucinations (responses that seem plausible but aren’t supported by the retrieved content). Azure AI Content Safety’s groundedness detector highlights when answer sentences do not align with the source passages, and it even goes a step further, identifying ungrounded segments and suggesting corrected formulations, helping teams bypass the need for a separate verification step.

When faced with a grounding issue, begin by confirming that the knowledge actually exists in the index. If the needed content was never indexed, the model has no chance of producing a grounded response. This might happen if the corpus is incomplete, outdated, ambiguous, contradictory, difficult to interpret or lacking in domain coverage. Addressing this might require reviewing and refining the source material, or layering additional clarification logic during the generation phase.

If you’re confident the information does exist in the index, the next step is to determine whether it was actually retrieved. Retrieval effectiveness depends on several factors: the chunk size, the number of documents or chunks returned, the quality of the vector and keyword matching, and the structure of the query itself.

In some cases, you might have an incorrect chunking strategy. If chunks are too short, critical context might be lost; if they’re too large, relevant details can be buried or excluded due to token limitations. Chunk size optimization is essential here, ideally balancing comprehensiveness with precision.

Also, if the retrieval step failed to surface the right content it might be because it is retrieving too few results, which could prevent key evidence from reaching the model. Increasing the number of retrieved chunks, refining query generation, or enabling hybrid search with semantic ranking in Azure AI Search can improve the likelihood of surfacing the right information.

Weak semantic matching or incomplete synonym handling could also cause relevant documents to be overlooked. A surprisingly powerful but underused lever is the choice of analyzers. Assigning a language-specific or custom analyzer to key fields improves tokenization, stemming and synonym handling, which in turn raises the chance that the right passages are pulled into the top-k set. Additionally Azure AI Search semantic ranker could be used to reorder the documents, increasing the recall for RAG workloads.

If the relevant content was retrieved, yet the model still hallucinates or gives an incomplete answer, the issue likely lies elsewhere. One possibility is prompt design: the model may not have been clearly instructed to rely exclusively on the retrieved content. A well-structured prompt should explicitly direct the model to answer based only on the provided context and to admit when the answer is not found.

Structuring the analysis of grounding issues using both telemetry and user feedback, checking index completeness, retrieval success, and model adherence, enables teams to quickly isolate and resolve the root cause. Combined with Azure AI Foundry’s metrics and the Content Safety groundedness detector, this approach provides a systematic path toward improving the factual reliability of responses in production RAG systems.

Another aspect that might be difficult to to optimize is cost. Cost control is an art involving the optimization of search units and model tokens used. On the search side, Azure AI Search charges only for the replicas and partitions you allocate. Start with a single replica-partition pair and add replicas only when 95-percentile latency climbs, or partitions when you near the index-size limit. Before scaling up, squeeze more value out of what you already pay for: choosing language-appropriate or custom analyzers improves recall without touching capacity, because better tokenisation and stemming help the right passages surface in the first place.

Tokens, on the other side, drive the Azure OpenAI bill costs. Smaller GPT model usually costs a fraction of larger ones, so many teams might route routine questions to smaller models and invoke bigger models only for complex cases, trimming spend by an order of magnitude. Azure AI Foundry Model Router (announced at Microsoft Build 2025) is a deployable chat model trained to select the best large language model (LLM) to respond to a given prompt in real time.

Another way to further reduce your token usage is by keeping chunk sizes compact (≈ 250–500 tokens) or by limiting the number of chunks retrieved. Finally, cache embeddings and even full answers avoids recomputing the same work repeatedly, saving tokens and latency for repeated or similar queries.

Ultimately, a RAG pipeline on Azure becomes robust only when every insight above, index completeness, retrieval effectiveness, prompt fidelity, model selection, cost hygiene, and continual monitoring, is folded into a single, disciplined feedback loop. Instrument the stack with Azure AI Foundry metrics, overlay human ratings, and revisit the corpus, search settings, chunking strategy, and model routing whenever groundedness or latency drifts. Teams that make this inspection‑improvement cycle routine transform a fragile demo into a resilient, self‑correcting production system that stays aligned with both factual accuracy and cost targets.

Advanced RAG Scenarios

Retrieval-augmented generation (RAG) started out as a straightforward “retrieve, then generate” wrapper around large language models and soon became the canonical way to ground LLMs in enterprise data, sharply reducing hallucinations by forcing the model to reason over evidence surfaced at query time. As deployments scale, however, that once simple pipeline is evolving into a rich family of patterns that expand the retrieval substrate, embrace new modalities and tasks, connect directly to data planes, and in several cutting-edge implementations, embed autonomous agents to orchestrate the entire loop.

Multimodal RAG

Multimodal RAG allows language models to use more than just text when answering a question. The system can also consider screenshots, diagrams, tables, audio transcripts, and other types of data that help clarify the answer. This gives users responses that are better grounded in the format that carries the most meaning for the question, whether that’s a paragraph, an image, or both.

On Azure, this is supported directly through Azure AI Search. You can use built-in skills like OCR, Image Analysis, and Vision Embedding to extract visual information such as tags, captions, or layout details, while text content is processed with standard Text Split and Embedding skills. You can also add audio transcripts or custom metadata using custom skills. Everything stays in one index with the same filters, access control, and enrichment process. With the recent release of multivector search (public preview, May 2025), a single query can now search across multiple types of content (text, image, and layout) and combine the results using BM25 and semantic ranking. This allows you to build powerful, multimodal RAG pipelines without needing separate vector databases or complex setup.

GraphRAG

Beyond working across different media, RAG can also tap into the structured knowledge graphs many organisations already maintain for product metadata, policy links, or data lineage. In graph-based RAG, a quick graph walk is combined with the usual vector search so the system only returns information tied to entities the caller can access or that truly match the question. This keeps answers on-topic and cuts hallucinations without extra post-processing.

A more advanced approach that deeply integrates structured data with RAG is GraphRAG, developed by Microsoft Research. This method constructs a comprehensive knowledge graph from the source corpus using large language models (LLMs) to extract entities, relationships, and assertions. These elements are then organised into hierarchical communities, each summarised to capture the semantic themes of the dataset. At query time, GraphRAG leverages these community summaries to provide contextually rich and accurate responses, particularly excelling in handling complex, multi-hop queries that require understanding across different parts of the data. This structured approach enables the system to deliver more coherent and comprehensive answers compared to traditional RAG method. On Azure, both the graph and its vectors can live in Cosmos DB or PostgreSQL, where vector similarity sits behind the same security envelopes and role-based permissions that guard the graph itself, so lineage, access control, and retrieval all stay in one place. The graph logic can be run as needed in Azure Functions or containers, making this a scalable and infrastructure-friendly solution for production environments.

However, maintaining a complete and continuously updated knowledge graph can be costly and complex, especially at scale. This is where Lazy GraphRAG, a method introduced by Microsoft Research, offers a practical alternative. Rather than building and storing the entire graph upfront, Lazy GraphRAG starts with a standard vector search to retrieve top candidates and then constructs a small, temporary graph on the fly using entity co-occurrence, citation links, or metadata relationships. A short traversal over this mini-graph helps re-rank results based on structural signals such as edge density or community overlap. Despite the lightweight approach, Lazy GraphRAG delivers answer quality close to full GraphRAG while reducing index maintenance and storage costs by over 90%.

NL2SQL

In many enterprise scenarios, the information users seek isn’t found in documents, it’s buried in relational databases. This is where NL2SQL (natural language to SQL) becomes an essential extension of RAG. Rather than retrieving static passages, the system generates structured queries to fetch live data directly from tabular sources. On Azure, this pattern is supported through Microsoft Fabric agents and the Azure AI Agent Service, which allow agents to generate SQL queries based on the user’s prompt, execute them securely against OneLake or other Fabric-connected databases, and return the results as part of the final response. These agents can be augmented with metadata such as schema, sample rows, and query examples to improve accuracy and reduce ambiguity. Because the execution happens under the user’s identity and adheres to native data access policies, this approach combines the flexibility of natural language interfaces with enterprise-grade governance. NL2SQL bridges the gap between language models and structured data, turning RAG into a fully integrated front end for operational analytics and real-time decision-making.

Agentic RAG

Another complementary evolution of RAG involves the use of autonomous agents, often referred to as Agentic RAG.. This represents a significant evolution in RAG. These autonomous agents introduce dynamic reasoning, decision-making, and multi-step task execution capabilities, enabling the system to handle complex queries more effectively. Unlike traditional RAG systems that follow a fixed sequence of retrieving and generating responses, Agentic RAG systems can plan, evaluate, and adapt their actions based on the context and quality of the information retrieved.

A key feature of Agentic RAG is its ability to perform multi-query generation. When faced with complex or multifaceted questions, the agent can decompose the original query into multiple sub-queries, each targeting a specific aspect of the user’s request. These sub-queries are then executed in parallel, retrieving diverse pieces of information that are subsequently synthesized into a coherent and comprehensive response. This approach enhances the system’s ability to provide accurate and contextually relevant answers, especially for queries that encompass multiple topics or require information from various sources.

Another powerful capability is autonomous multi-turn querying, where the agent can iteratively refine its understanding of the user’s need by issuing follow-up queries based on earlier retrievals. For example, if a user asks a broad or underspecified question, the agent might first gather general information, then drill down with more focused queries until all parts of the question are answered. This iterative strategy is especially valuable in domains like legal, medical, or technical support, where resolving a single query often requires chaining together multiple retrieval steps.

In the Azure ecosystem, the recently introduced Agentic Retrieval feature in Azure AI Search enables intelligent multi-query generation. It uses large language models to analyze the user query and conversation history, then generates several focused sub-queries that are run in parallel over both text and vector indexes. Results are then merged and reranked using Azure’s semantic ranking capabilities to produce a single coherent response. While Agentic Retrieval itself does not yet perform fully autonomous multi-turn querying, this behavior can be implemented by combining Semantic Kernel with Azure AI Search, allowing developers to design agents that reason over intermediate results and continue querying until the task is complete.

RAFT

By integrating these agentic capabilities, RAG systems become far more flexible and intelligent. Agentic RAG is particularly well-suited for use cases that demand depth, reasoning, and the ability to navigate complex, multi-layered information landscapes; making it a strong fit for enterprise applications, decision support systems, and advanced conversational AI.

When deploying RAG systems in specialized domains, such as biomedical research, law, or enterprise operations, it becomes clear that even LLMs equipped with retrieval capabilities can struggle to distinguish relevant information from noise. Many models treat all retrieved passages equally, leading to diluted or incorrect responses when distractor documents are included. To address this challenge, researchers at UC Berkeley proposed an approach called Retrieval-Augmented Fine-Tuning (RAFT), which combines supervised fine-tuning with retrieval-augmented generation to teach the model how to focus on truly relevant content while ignoring misleading or irrelevant information.

Rather than modifying the retriever itself, RAFT enhances the generator. It does so by fine-tuning the model on carefully constructed examples that include a user question, a mixture of relevant and irrelevant (distractor) documents, and a chain-of-thought answer that explicitly cites the correct sources. Through this process, the model learns not just to answer questions but to reason over the retrieved context, prioritizing documents that contribute to the final response. This results in improved accuracy, groundedness, and faithfulness, particularly in complex question-answering scenarios. Studies have shown that RAFT-trained models outperform baseline RAG systems across datasets like PubMedQA, HotpotQA, and Gorilla, confirming its effectiveness in high-stakes, knowledge-heavy environments.

On Azure, RAFT can be applied using Azure AI Foundry, which supports fine-tuning of Azure OpenAI and open source models. This enables enterprises to adapt any general-purpose model to their internal knowledge and retrieval dynamics, producing systems that not only ground their answers better but also align with enterprise governance standards.

Conclusions

As the evolution of retrieval-augmented generation unfolds, it’s clear that RAG is no longer a monolithic architecture but a dynamic design space, capable of adapting to structured and unstructured data, navigating multimodal inputs, and orchestrating complex reasoning through agents. Azure’s expanding suite of services, from AI Search and Semantic Kernel to Agent Service and AI Foundry, enables enterprises to compose these advanced RAG variants in a modular, secure, and scalable fashion. Each pattern, be it GraphRAG, LazyGraphRAG, NL2SQL, or Agentic RAG, represents a step toward more adaptive, domain-aware systems that can reason, retrieve, and respond with increasing nuance and reliability.

Looking ahead, the next frontier lies in composability: blending these advanced patterns into unified architectures that are context-sensitive, continuously learning, and tightly governed. Whether it’s enabling deeper domain adaptation through RAFT, orchestrating agent workflows for multi-turn reasoning, or seamlessly bridging SQL backends with vector stores, Azure empowers teams to build retrieval-augmented systems that are not just reactive, but intelligent collaborators in decision-making. As generative AI becomes a foundational layer across industries, mastering these advanced RAG patterns will be key to building grounded, trusted, and enterprise-grade AI solutions.

Summary

In this chapter, we explored the foundations and practical applications of Retrieval-Augmented Generation (RAG). We began with the fundamentals of RAG and the typical process of combining retrieval with generation, before turning to how these concepts can be implemented on Azure. We looked at OpenAI on Your Data as a managed entry point and contrasted it with building a RAG pipeline directly on Azure AI Search, highlighting the differences in flexibility, control, and effort between the two approaches.

From there, we discussed how RAG systems can be continuously improved through iterative optimization. Techniques such as prompt refinement, chunking strategies, index reconfiguration, and feedback loops were presented as practical levers to enhance accuracy, grounding, cost efficiency, and latency.

Finally, we explored advanced directions in RAG. These included extending RAG beyond text with multimodal inputs, enriching retrieval with knowledge graphs (GraphRAG), bridging unstructured and structured data with natural language to SQL, introducing autonomous agents through Agentic RAG, and improving factual consistency through reinforcement techniques such as RAFT. Together, these approaches point toward the future of RAG: systems that are more adaptive, multimodal, and capable of handling increasingly complex tasks.

Chapter 5. Fine-Tuning Generative AI Models in Azure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the fifth chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

Fine-Tuning Foundations

In this chapter we will discuss when fine-tuning generative AI models is necessary and walk through the different fine-tuning approaches available in Azure, helping you determine the best strategy for optimizing model performance.

Fine-tuning represents a powerful method for adapting large language models to specialized tasks and domains by updating their internal parameters based on new data. Positioned between prompt engineering and retrieval-augmented generation (RAG), fine-tuning offers a means of embedding knowledge directly into the model, allowing it to internalize patterns, terminology, or behaviors that may not be well represented in its original training corpus. This approach enables more precise control over outputs, improved performance on domain-specific tasks, and reduced dependence on lengthy prompts or external retrieval mechanisms. Unlike prompting, which relies on instructive input without altering the model, or RAG, which dynamically incorporates external context at inference time, fine-tuning permanently refines the model’s behavior. In this section, we will explore the conditions under which fine-tuning is the most appropriate choice.

The Fine-tuning process begins by continuing the training of a pre-trained model on a new, typically smaller dataset that reflects the desired application domain or task. This is achieved by adjusting the model’s parameters to better capture the linguistic patterns, knowledge, or behaviors relevant to the target use case. There are two principal approaches to fine-tuning: full fine-tuning and parameter-efficient fine-tuning (PEFT). In full fine-tuning, all of the model’s parameters are updated, which can yield significant performance improvements but often requires substantial computational resources and careful regularization to prevent overfitting. In contrast, PEFT techniques introduce a minimal number of trainable parameters, usually by inserting lightweight components such as adapter layers or training task-specific vectors that act as soft prompts, while keeping the vast majority of the model frozen. These methods significantly reduce the computational and data requirements, making fine-tuning more accessible and efficient. Together, these strategies offer flexible pathways to adapt large models while balancing performance, scalability, and resource constraints.

When to Fine-Tune your Generative AI Models?

While fine-tuning requires additional investment in training time, data preparation, and model management, the returns can be significant. The following scenarios illustrate why teams may choose to fine-tune a pre-trained model rather than rely solely on prompt engineering or retrieval-based approaches:

Domain alignment. Fine-tuning enables models to absorb the specific terminology, conventions, and reasoning patterns of specialised fields such as law, medicine, or engineering, areas where general-purpose models might fall short.

Narrow task optimisation. For structured, highly specific tasks, such as intent classification, form validation, or function calling, fine-tuning allows precise control over model outputs, aligning them with well-defined application requirements.

Customisation with limited or proprietary data. Fine-tuning offers a way to adapt models using small but highly relevant datasets, including proprietary data, domain-specific corpora, or synthetic examples. This strategy is especially effective when training with parameter-efficient methods, which minimise the need for large-scale resources. The same approach also enables personalisation, allowing models to adapt to use cases by integrating contextually relevant data, all while preserving data privacy and reducing prompt complexity.

Language coverage and localisation. When working with low-resource languages or regional dialects, fine-tuning can significantly improve comprehension and generation quality. It also facilitates localisation by embedding cultural nuances, local terminology, and region-specific standards directly into the model’s internal representation.

Latency and edge deployment. A fine-tuned, smaller model can meet application-level performance requirements with significantly lower latency and resource consumption, making it ideal for edge devices or constrained environments. Techniques like LoRA further enhance this by reducing the overhead of adaptation.

Cost and token savings. Fine-tuning can reduce inference costs by shortening prompt lengths and enabling the use of smaller models, especially in high-throughput or production settings where predictability and efficiency are critical.

IP ownership and compliance. Once domain-specific knowledge is embedded into a model through fine-tuning, sensitive information no longer needs to be sent with each request. This reduces exposure risk and simplifies compliance with security and governance standards.

Prompt Engineering vs RAG vs Fine-Tunning

Selecting between prompt engineering, retrieval-augmented generation (RAG), and fine-tuning involves balancing trade-offs based on the specific requirements of the application, such as accuracy, latency, operational complexity, and available resources. In practice, it is often advisable to begin with prompt engineering, given its simplicity and immediate applicability. From there, RAG can be introduced to incorporate dynamic, document-level context with minimal model changes. As demands for accuracy, latency, or domain adaptation increase, fine-tuning becomes a viable next step, enabling deeper control and efficiency by embedding task-specific knowledge directly into the model. The table below provides a comparative overview of these strategies to help guide informed decision-making:

Criterion | Prompt Engineering | Retrieval-Augmented Generation (RAG) | Fine-Tuning (Full or PEFT)
Implementation Effort | Minimal: requires no model changes or training | Moderate: requires setting up a retrieval system and integrating it with the model | High: involves data curation, training pipelines, and evaluation processes
Data Requirements | None or a few examples embedded in prompts | Access to well-structured external documents or databases; no labels required | Training data sets (small for PEFT; larger for full fine-tuning)
Adaptability | Limited to what the base model can do with prompt instructions | High: can incorporate real-time, domain-specific information through retrieval | Very high; embeds new knowledge and behavior directly into the model
Knowledge Freshness | Static; limited to pre-training cut-off | Dynamic; reflects the latest data as long as the retrieval corpus is kept up to date | Static; updates require re-training
Performance |  | Utilizes external resources, suitable for questions demanding factual or knowledge-based information.. | Fine-tuning enhances performance through context-aware responses, specific task execution, and specialized workflow optimization.
Latency | Low inference latency (may require long prompts which might increase the latency) | Slightly higher latency due to retrieval operations | Low latency, especially with smaller fine-tuned models
Cost Considerations | No training cost, just token usage costs | Moderate; additional infrastructure and retrieval compute | Higher upfront training cost and inference costs depend how the model are inferred
Security & Privacy | Sensitive information can be included in the prompt at runtime | Document access controlled, but retrieval mechanisms must be secured | Sensitive data embedded in the model: less need to send it during inference
Best Use Cases | General-purpose Q&A, and simple tasks | Factually based response generation, FAQs, knowledge access | Domain-specific assistants, structured outputs, function calling, etc.

To conclude, fine-tuning stands as a powerful tool in the broader landscape of generative AI adaptation strategies. While it requires more upfront effort than prompting or retrieval-augmented generation, it offers unmatched control, precision, and integration of domain-specific knowledge. Understanding when and how to apply fine-tuning is essential for building scalable, high-performing solutions that meet real-world requirements. In the next sections, we will explore how these concepts are implemented in practice using Azure’s ecosystem, beginning with fine-tuning capabilities in Azure OpenAI, followed by approaches for fine-tuning other generative models through Azure AI Foundry.

Fine-Tuning Taxonomy

In the previous sections, we explored the rationale for fine-tuning: what it is, how it compares to alternatives like prompt engineering and retrieval-augmented generation, and when it makes sense to invest in adapting a model beyond zero-shot or few-shot capabilities. With those foundations in place, we now turn from why to how. Fine-tuning isn’t a single method but rather a family of strategies that vary in complexity, efficiency, and purpose. From classic supervised learning to lightweight adaptations, from alignment via human preferences to model compression and deployment optimization, these approaches offer a diverse and evolving toolkit for shaping foundation models into practical, task-specific systems.

Supervised fine-tuning is the most direct approach within that toolkit. It adapts a pre-trained model to a specific task by training it on labeled input–output pairs, minimizing a loss function that reflects the difference between its predictions and the expected outputs. In its full form, this involves updating all of the model’s parameters, which provides maximum flexibility but also comes with high computational cost and an increased risk of overfitting, especially when the dataset is small or highly specialized. Another risk is catastrophic forgetting, where the model’s performance on its original broad domain degrades after being fine-tuned for a narrow task. Techniques like regularization, data mixing, or the use of parameter-efficient tuning methods can help mitigate this effect.

While supervised fine-tuning is most commonly associated with text models, such as adapting a language model to follow specific instructions or use domain-specific terminology, it also applies to other modalities. Visual models, for example, can be fine-tuned on small, curated datasets to adapt their capabilities to a specific style, subject, or domain. This allows a general-purpose vision model to specialize in recognizing particular types of images with greater accuracy.

Supervised fine-tuning is also frequently used to enable or refine function calling, where a model is trained to respond to user inputs by generating structured outputs such as API calls. This may be used to equip a model with function-calling capabilities from scratch or to improve how reliably and accurately it performs tool use in narrow domains. By exposing the model to concrete examples of prompts and their corresponding structured outputs, fine-tuning helps it learn both the intent behind a request and the proper operational response.

As model sizes have scaled into the billions of parameters, the cost of full supervised fine-tuning, both in memory and training time, has become a limiting factor for many practical scenarios. Parameter-efficient fine-tuning (PEFT) techniques emerged as a response to this challenge. Rather than updating the entire model, PEFT strategies adjust only a small subset of parameters or introduce lightweight, task-specific components while keeping the core model frozen. This drastically reduces the number of trainable weights, speeds up fine-tuning, and lowers hardware requirements, all while maintaining most of the model’s original capabilities

One of the most widely adopted approaches is LoRA (Low-Rank Adaptation), which inserts small trainable matrices into the model’s architecture to learn compact task-specific updates. These matrices can later be merged with the original weights or kept separate as modular add-ons. Building on this idea, QLoRA combines LoRA with quantization ( reducing the numerical precision of a model’s parameters, such as converting 16- or 32-bit floating-point values into lower-bit representations like 8-bit or 4-bit integers) to enable fine-tuning of even very large models in a more efficient way, making high-performance customization far more accessible.

These techniques all involve trade-offs between memory savings, training speed, and ultimate performance. In practice, they are often combined, for example, using LoRA on top of a quantized model, to push efficiency even further. A major benefit of PEFT methods is their modularity: by keeping the base model intact, you can maintain a single foundation model and selectively activate different fine-tuned components for each use case.

While supervised fine-tuning focuses on task completion, it doesn’t necessarily ensure that model outputs align with user expectations in terms of helpfulness, tone, or factual correctness. Preference-based alignment techniques address this by fine-tuning models based on human feedback rather than task-specific labels. These methods use comparison data to guide the model toward responses that users are more likely to prefer.

One of the most common approaches is Reinforcement Learning from Human Feedback (RLHF). It starts by collecting human comparisons between different model responses for the same prompt. These preferences are used to train a reward model that scores the quality of each output. The generative model is then fine-tuned using reinforcement learning, typically with an algorithm like Proximal Policy Optimization (PPO), to produce outputs that maximize this reward. PPO iteratively adjusts the model to favor responses that receive higher scores, helping it align more closely with what users prefer.

To reduce the need for human raters, Reinforcement Learning from AI Feedback (RLAIF) replaces human comparisons with judgments from another language model. Instead of human annotators, an automated model decides which outputs are better and generates preference labels. These labels are used in the same way as in RLHF, enabling scalable alignment with far less manual effort.

Direct Preference Optimization (DPO) takes a simpler, supervised learning approach. Instead of building a reward model and using reinforcement learning, DPO updates the model directly based on preference comparisons. If output A is preferred over B for a given prompt, the model is trained to assign higher probability to A. This avoids the complexity of reward modeling and iterative policy updates. While DPO is mathematically grounded in similar objectives as RLHF, it achieves alignment in a single, stable training loop. It’s often more sample-efficient, too, since each comparison is used immediately to refine the model. The main trade-off is that DPO works best with clear binary preferences and doesn’t naturally handle more nuanced feedback.

Ultimately, the main difference between these techniques is how they incorporate preferences. RLHF and RLAIF involve learning a reward signal and optimizing through exploration, which provides flexibility but adds training complexity. DPO, on the other hand, directly uses comparison data to guide model updates in a simpler pipeline. In practice, alignment workflows may combine these approaches, for example, using DPO for initial alignment and RLAIF to supplement the training data, offering a balance between efficiency, scalability, and output quality. Preference-based alignment is a critical tool for making generative models more responsive to real-world user expectations.

Fine-Tuning in Azure AI Foundry

We have seen that fine-tuning large language models (LLMs) enables organizations to tailor general-purpose models to specific domains, tasks, or stylistic requirements. Azure AI Foundry provides a path to adapt base models from multiple providers into dependable, domain-ready variants.

Fine-tuning is available for classic text-to-text across all supported LLMs, and, visual fine tuning is available for a set of multimodal models. Modality and limits are model-specific; always check the model card for the exact schema and constraints.

Azure AI Foundry offers two ways to run training and host the resulting model variant: a fully managed serverless path and a bring-your-own-infrastructure managed compute path. Both produce the same kind of tuned artifact, the difference lies in how much operational control you want (and need) over GPUs, hyperparameters, and runtime versus how much you prefer a streamlined, consumption-based experience.

Serverless: Azure provisions and operates the training and hosting environment for you, no GPU quota to manage, consumption-based pricing, and sensible defaults that let you start quickly. This path is ideal when you prioritize simplicity and time-to-value, or when you want access to curated model families without dealing with low-level configuration; you trade a bit of fine-grained control for a streamlined workflow and autoscaling operations. It’s important to note that fine-tuning Azure OpenAI models is exclusively available through the serverless fine-tuning option.

Managed compute: You bring your own Azure GPU capacity and choose SKUs, giving you deeper control over hyperparameters and runtime (e.g., batch size, learning rate, epochs) and the flexibility to target models or variants not offered in serverless. This mode suits advanced teams with capacity planning and MLOps in place who need strict performance/cost control or specialized training behavior, at the cost of additional operational ownership.

Selecting the right optimization method depends on your goal (format fidelity, style alignment, or reasoning gains), the kind of data you have (labeled targets, preference pairs, or rewardable outcomes), and the measurability of success. In practice, many teams layer techniques over time: start with supervised updates to lock structure and tone, add preference optimization to steer qualitative traits, and reserve reinforcement-style training for tightly scoped, objectively gradable tasks.

Supervised fine-tuning (SFT): Provide high-quality input/output examples to teach the model target behavior, domain phrasing, structured extraction, response schemas, or brand tone. SFT is the default choice when prompt engineering plateaus and you need repeatable outputs; start with a modest but clean dataset, hold out evaluation data, and iterate to avoid overfitting while locking in format and style reliability.

Direct Preference Optimization (DPO): Instead of a single “gold” answer, supply pairwise preferences (“A better than B”) so the model learns qualitative traits such as helpfulness, harmlessness, politeness, or on-brand style without building a separate reward model. DPO is effective when quality is subjective and best expressed as choices; its success depends on well-curated comparisons that reflect the judgments you want the model to internalize.

Reinforcement fine-tuning (RFT): Optimize against an explicit reward signal for tasks with verifiable correctness (math, logic, code tests) or measurable objectives (tool-use success, constraint satisfaction). RFT offers strong gains on reasoning-centric problems but introduces higher ML/ops complexity, defining stable rewards, preventing reward hacking, and running controlled experiments; so it’s best applied to tightly scoped tasks with reliable automated scoring.

Each of these techniques offers unique advantages and is suited to different scenarios. Selecting the appropriate fine-tuning method depends on the specific goals, data availability, and constraints of your application.

Approach fine-tuning as an iterative experimentation cycle: quantify the baseline gap, choose serverless or managed based on simplicity versus control, select the technique that matches your data and success criteria, and iterate with held-out tests until the tuned model consistently beats baseline on target metrics. When ready, deploy the tuned model as a serverless endpoint or on your managed runtime, monitor quality, latency, and cost, and plan periodic refreshes as your data and requirements evolve. Next, we’ll look at provider-specific guidance for fine-tuning Azure OpenAI models, followed by a section on fine-tuning other providers’ models in Azure AI Foundry.

Fine tuning Azure OpenAI Models

As noted earlier, Azure OpenAI models are tuned and hosted through the Azure AI Foundry serverless experience. You don’t manage GPUs or training runtimes; you prepare data, select a base model and training method, and Azure runs the job and serves the resulting custom model as a managed deployment. Azure OpenAI training options vary by region, so always check the models page before you start.

Azure OpenAI supports supervised fine-tuning (SFT) for non-reasoning chat models, Direct Preference Optimization (DPO preview) for a subset of models, and Reinforcement Fine-Tuning (RFT preview) for select reasoning models. Certain Azure OpenAI models also support vision fine-tuning (teach the model from images + text) and tool-calling fine-tuning (teach when and how to call functions/tools, and how to interpret tool outputs). The table below summarizes common base models and which techniques are currently supported. Availability changes quickly and sometimes by version, so treat this as a snapshot and verify against the documentation when you read this.

Base model (version) | SFT | DPO (preview) | RFT (preview) | Vision SFT
GPT-4o | ✅ | ✅ | — | ✅
GPT-4o-mini | ✅ | — | — | —
GPT-4.1 | ✅ | ✅ | — | ✅
GPT-4.1-mini / 4.1-nano | ✅ | ✅ | — | —
o4-mini (reasoning) | — | — | ✅ | —

Note

SFT is called out as supported for non-reasoning models; RFT is supported on reasoning models “like o4-mini”; DPO is explicitly called out for GPT-4o in the current docs. Vision fine-tuning is limited to specific 4o and 4.1 versions. Tool-calling fine-tuning uses the chat-completions schema with tools. Re-check the model catalog and training method pages for your region and model version.

Training and validation data are newline-delimited JSONL in the chat-completions format: each line has a ‘messages’ array whose items carry roles (system, user, assistant). Files must be UTF-8 with BOM and ≤ 512 MB. The service requires at least 10 examples to start, but practical runs benefit from dozens to hundreds of high-quality demonstrations. For vision fine-tuning, image examples live in the same JSONL rows via type: “image_url” entries; images must be JPEG/PNG/WEBP, RGB, up to 10 MB each, up to 64 images per example, and up to 50,000 image-containing examples. For tool-calling fine-tuning, include tools (function schemas) and show the assistant’s tool_calls, plus optional tool response turns followed by the assistant’s final response.

You will need Cognitive Services OpenAI Contributor on the target resource to create fine-tunes and view relevant quotas. Training runs can be Standard (regional, data residency) or Global (broader capacity and lower per-token cost, no residency guarantees). After training, the custom model must be deployed to use it; deployments incur an hourly hosting cost in addition to per-token usage, and idle fine-tune deployments may be automatically cleaned up after prolonged inactivity. Quotas/limits apply to upload sizes and file counts, and fine-tuning region/model availability can differ from base model inference availability, verify both before committing to a plan.

For structure, tone, extraction schemas, or domain-specific formatting, start with SFT on a compact model like GPT-4o-mini, then consider moving up to GPT-4o/4.1 if you need more headroom. Reach for DPO (preview) on GPT-4o when qualitative alignment matters more than a single gold label. Use RFT (preview) with o4-mini for tightly scored reasoning tasks where you can define reliable rewards. If your application reads images (e.g., forms, screenshots, product visuals), leverage vision fine-tuning on the supported GPT-4o and GPT-4.1 versions to teach layout- or domain-specific visual cues. If your agent calls functions or tools, add tool-calling fine-tuning examples so the model learns when to call which function and how to turn tool outputs into final user-ready answers with the right tone and structure.

In the next section, we’ll walk through a step-by-step Supervised Fine-Tuning (SFT) flow in Foundry (data prep, file upload, job creation, deployment, and evaluation), including a compact JSONL template you can adapt for text-only, tool-calling, and vision-augmented training.

Step by Step Fine tuning workflow for Azure OpenAI Models

In Azure AI Foundry, the fine-tuning workflow in a project begins with prerequisites and data discipline, then moves through training, monitoring, deployment, and iteration. Start inside a Foundry project with an Azure OpenAI connection in a region that supports fine-tuning and as we previously mentioned, ensure your account has at least the Cognitive Services OpenAI Contributor role on the Azure OpenAI resource.

Figure 5-1. Azure AI Foundry Fine-tuning menu option

You can start fine tuning your models in two different places inside Azure AI Foundry: Either on the model page itself or either by clicking on the Fine tuning option on the left side menu. Once you have clicked on fine tuning your Azure OpenAI models you will have the option to select a training method (Supervised, DPO or RL) , upload your training and validation data, adjust task parameters (such as number of epochs or batch size), and can opt into auto-deployment.

Example 5-1. Example JSONL dataset

{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "Who discovered Antarctica?"}, {"role": "assistant", "content": "Some chaps named Fabian Gottlieb von Bellingshausen and Mikhail Lazarev, as if they don't teach that in every school!"}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What is the biggest ocean?"}, {"role": "assistant", "content": "The Pacific Ocean. It's not like it's a small pond or anything."}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What is the largest planet?"}, {"role": "assistant", "content": "It's called Jupiter, you might have heard of it...or not."}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "Who painted the Mona Lisa?"}, {"role": "assistant", "content": "Oh, some obscure artist named Leonardo da Vinci. Not that he is famous or anything."}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the boiling point of water?"}, {"role": "assistant", "content": "Exciting fact of the day: it's 100 degrees Celsius. Life-changing, isn't it?"}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "Who wrote 'The Odyssey'?"}, {"role": "assistant", "content": "Oh, just some poet named Homer. Nothing special, really."}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What is the largest mammal?"}, {"role": "assistant", "content": "Blue Whale. Quite a big fish isn't it? Oh wait, it's not a fish!"}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the highest mountain peak?"}, {"role": "assistant", "content": "Everest. Brace yourself, it's not a walk in the park."}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "Who invented the telephone?"}, {"role": "assistant", "content": "A guy named Alexander Graham Bell. Bet you didn't guess that one!"}]}
{"messages": [{"role": "system", "content": "Clippy is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What is the distance from Earth to the Sun?"}, {"role": "assistant", "content": "About 93 million miles. Just a quick drive, really."}]}

Data preparation is one of the most important factors in outcome quality. Azure OpenAI expects JSON Lines (JSONL) files using the chat-completions message schema, with each line containing the messages array representing a single or multi-turn chat example. While the service accepts a minimum of ten examples to run the pipeline, plan for hundreds to thousands of high-quality, representative examples to teach a new behavior and avoid overfitting. Curate carefully, de-duplicate, cover edge cases with counter-examples, and keep outputs authoritative and consistent; bad or narrow data produces brittle models. If you are teaching tool calling, include the tool definitions alongside examples of assistant tool calls and, optionally, tool results. If you are teaching vision+text, include images per example as URLs or base64 data URIs within the user message.

Figure 5-2. Status and details of a fine tuning job

Fine-tuning should always be approached as an iterative experiment. Within the Azure AI Foundry portal, you can track the status of your job, which may initially appear in a queuing state if other jobs are ahead in the system. Training time can range from minutes to several hours, depending on the chosen base model and dataset size. At the end of each epoch, the service automatically generates a checkpoint, which is a fully functional version of the model that can be deployed directly or used as the basis for further fine-tuning. These checkpoints are particularly valuable because they capture intermediate states of the model and may preserve good performance before overfitting occurs. You can monitor progress through logs and metrics, and if the job is not converging as expected, it is possible to pause training once at least one step has been completed. In that case, the system finalizes a checkpoint after safety evaluations, which you can deploy immediately for inference or resume training from later on.

Figure 5-3. Metrics of the fine tuning job

After fine-tuning completes, you can download a results.csv file from the model’s Details tab, which records training and validation metrics for each step. It includes loss values and token accuracy, both per step and summarized at the end of each epoch. In practice, you want to see loss steadily decrease while accuracy rises, and any widening gap between training and validation results may signal overfitting. These metrics can also be viewed as plots in the Monitoring tab of Azure AI Foundry, where loss and accuracy curves provide an immediate sense of how well your model is learning and whether adjustments such as fewer epochs or a lower learning rate are needed.

You can directly deploy your newly fine tuned model in your Azure AI Foundry Project. Foundry supports multiple deployment types for fine-tuned models (for example, Standard and other capacity options), and you can monitor deployment progress on the project’s Deployments page. Keep two operational facts in mind: fine-tuned deployments incur an hourly hosting cost while active, and inactive fine-tuned deployments are automatically deleted after 15 consecutive days without inference traffic; the underlying customized model remains and can be redeployed at any time. For cross-region or cross-subscription scenarios you can also use the management REST API, but the project portal flow is sufficient for most cases. After deployment, you call your tuned model by its deployment name just like any other Azure OpenAI deployment.

Vision and tool-calling follow the same project workflow with different training schemas. For vision, your training file can include up to 64 images per example, provided as URLs or base64 data URIs in the image_url content entries. For tool-calling, include your tools schema in each JSONL line and show the assistant’s tool invocation, and optionally the tool’s response and the assistant’s grounded follow-up, so the model learns both the API call and the post-processing pattern.

In the end, fine-tuning in Azure AI Foundry follows a clear cycle: start with carefully prepared, high-quality JSONL data, run and monitor the job, deploy the customized model, and test it on a held-out set that mirrors production. Checkpoints can be deployed if they outperform the final model, and periodic refreshes help mitigate data drift. Cleaning up idle deployments keeps costs under control. Above all, remember that the quality of your data will determine the quality of your model, making careful preparation the foundation of every successful fine-tuning workflow.

Fine tuning other catalog models in Azure AI Foundry

As previously mentioned, Azure AI Foundry lets you fine-tune many catalog models through two paths: serverless (Azure hosts training/serving) and managed compute (you bring/choose GPUs and runtime). Managed compute would allow more control over VM SKUs, networking, and hyperparameters. Always confirm model and region support on the model card and with the Fine-tuning tasks filter in the catalog.

Figure 5-4. Fine tuning options for non Azure OpenAI models

From your project, once you know which model you want to fine tune, open the model card and choose Fine-tune. The process for serverless fine tuning will be similar to the one one the previous section. For managed compute fine-tuning, you begin by selecting the compute target. You can either provision a new managed compute cluster or reuse an existing one, with full flexibility over VM SKU, region, autoscaling, and attached networking resources. This setup is ideal for organizations that need to comply with specific security or data residency requirements, or that want to optimize cost and performance by choosing the right GPU family.

Figure 5-5. Compute resource selection list for managed compute fine tuning

Aspect | Serverless fine-tuning | Managed compute fine-tuning
Project type | Standard project | Hub-based project required
Compute ownership | Microsoft-managed; no GPU quota | Your Azure VMs; you manage quota/SKUs
Data formats | JSONL (task-specific schemas) | JSONL / CSV / TSV (or registered datasets)
Hyperparameter control | Streamlined; fewer knobs | Full control (epochs, LR, batch size, etc.)
Region & offer gating | Must match provider’s offer region/billing availability | Runs where you have quota; model availability varies
Marketplace subscription | Required for many Partners & Community models | Not typically required for your own compute
Networking & isolation | Follows hub PNA; private endpoints supported for the hub | Inherits your hub’s managed network settings
Content safety | Default filters enabled; toggle per endpoint | Use Azure AI Content Safety via your pipeline/tools
Costs | Pay per usage plus hourly hosting for tuned deployments | VM core-hours for training/serving; your metering
Best for | Fastest path, minimal ops | Maximum control/compliance/tuning depth

Once the compute is defined, you configure the training job through the fine-tuning wizard. Here, you specify:

Training and validation data: uploaded as JSONL or other model-specific schema.

Hyperparameters: such as batch size, learning rate, number of epochs, and early stopping rules.

Compute configuration: VM size, node count, and scaling policy.

When the job is submitted, Azure AI Foundry packages the configuration into a training run executed on the managed cluster. You can monitor the job from the project dashboard, where real-time logs, metrics (loss curves, accuracy), and model checkpoints are surfaced. Checkpoints are especially useful if you want to resume training, compare multiple runs, or roll back to an intermediate model that performs better on your metrics than the final epoch.

After completion, the tuned model artifact is registered automatically in your project. You can then deploy it to an online or batch endpoint using the same managed compute or move it to another environment. This separation of training and deployment compute allows enterprises to optimize GPU usage: allocate expensive SKUs only during training, then scale down for serving.

In summary, fine-tuning through managed compute provides:

Greater control over infrastructure, runtime, and security.

The ability to use larger datasets and custom hyperparameters.

Integration with enterprise governance (VNETs, private endpoints, monitoring).

This path is best suited when serverless training does not meet your compliance, performance, or customization needs.

Fine Tuning Best Practices & Pitfalls

Having gone through the technical process, it’s worth summarizing some overarching best practices for fine-tuning on Azure, as well as pitfalls to avoid:

Ensure High Data Quality and Coverage: High-quality data is the single most important factor in fine-tuning. The dataset should be representative, diverse, clean, and sufficiently broad to cover the tasks you expect the model to handle. If the data contains bias or errors, the model will inevitably learn and reproduce them. Similarly, if certain inputs or perspectives are missing, the model may perform well on some queries but fail on others. Azure’s guidance highlights curating diverse, high-quality data as the foremost best practice.

Baseline, Validate, and Safeguard Performance: Before fine-tuning, always establish how well the base model performs with prompting. In many cases, careful prompt engineering or small system message tweaks can achieve acceptable results, and fine-tuning should only be used when there is a clear gap that prompting alone cannot close. When you do fine-tune, also evaluate the impact on the model’s original strengths. Fine-tuning can sometimes reduce performance outside the target domain (a phenomenon known as catastrophic forgetting). If this matters for your use case, consider techniques like LoRA or adapters, which preserve the base model while layering in new capabilities. For full fine-tuning, you can mitigate forgetting by mixing in some general data, but the more practical step is to test the tuned model on general tasks and decide whether any trade-offs are acceptable.

Avoid Overfitting and Memorization: One common pitfall is overfitting the model to the training data, especially if the dataset is small. Overfitting can manifest as the model memorizing exact answers and failing to generalize or, worse, spitting out training data verbatim (which could be a privacy issue if that data was sensitive).

Manage Cost and Latency Trade-offs: Fine-tuning comes with an upfront cost, so plan for it and optimize wherever you can. Consider using smaller models when they meet your needs, and closely monitor usage and scaling. If multiple fine-tuned models deliver similar performance, favor the more cost-efficient option. Reduce recurring expenses with caching strategies, and select the appropriate hardware tier in Azure to balance performance and cost.

Maintain Security and Compliance: Fine-tuning often involves working with proprietary or sensitive data. While Azure provides a secure environment, you are still responsible for handling that data with care. If your dataset includes personally identifiable information (PII), confirm you have permission to use it and anonymize where possible. Remember that once a model has internalized PII, it could inadvertently reproduce it, creating privacy risks. Regulatory compliance is another key concern: industries such as finance or healthcare may impose restrictions on automated decision-making or advice, so ensure your fine-tuned model is deployed within the boundaries of applicable laws and, if needed, add guardrails in either the model or the application logic. Intellectual property and licensing must also be respected, only fine-tune with data you have the rights to use, including third-party content. Finally, treat the fine-tuned model artifact itself as sensitive if it contains proprietary knowledge, limiting deployment and export rights through role-based access controls in Azure AI Foundry so that only authorized users can work with it.

Handle User Input Carefully Post-Fine-Tune: Never assume a fine-tuned model will be flawless or immune to producing poor responses. Users may still provide unexpected or tricky inputs. Continue applying content filtering APIs to intercept harmful outputs, even after fine-tuning. If your model is designed to generate domain-specific content (such as medical information), consider adding a final safeguard to ensure it does not go beyond its intended scope, such as offering disallowed advice.

Combine Fine-Tuning with Prompting and Other Techniques: Don’t think of fine-tuning as a replacement for prompting; in practice, the best results often come from a hybrid approach. Similarly, you could use retrieval-augmented generation for handling long documents while relying on a fine-tuned model to interpret them more effectively, or fine-tune a model for code generation while still including a few-shot example in the prompt to enforce a particular format. If a requirement can be handled easily through prompting, there is no need to hardwire every detail into the model’s weights.

Monitor for Bias and Fairness: If your application is user-facing, make sure to evaluate the fine-tuned model for potential biases, since the training data itself can introduce skew. For instance, fine-tuning on historical customer support chats may embed biases in how different customers were treated, leading the model to reproduce preferential or unequal responses. Conduct bias testing by varying demographic terms in queries and checking whether the outputs shift inappropriately. In line with Microsoft’s Responsible AI principles, fairness testing is recommended, and Azure provides tools such as the Responsible AI dashboard to help by generating and analyzing outputs across simulated inputs.

Don’t Over-rely on Fine-Tuning for Knowledge Updates: Fine-tuning is well suited for shaping core behavior and embed certain knowledge, but it is not practical for information that changes frequently. If your knowledge base updates daily, you will not want to fine-tune on it every day. For highly dynamic or factual queries, a retrieval-augmented generation (RAG) system or database lookup is usually the better fit. A common design is to fine-tune a model so it captures the general know-how and style, while relying on retrieval for up-to-date details the model can reference on demand.

In summary, fine-tuning on Azure is a powerful technique that can yield big wins in accuracy and customization, but it must be done thoughtfully. By following best practices around data, evaluation, security, and iterative improvement, you can avoid common pitfalls and build fine-tuned models that deliver real value in production.

Conclusions

Fine-tuning generative AI models in Azure is not a one-size-fits-all solution. It is a set of techniques that require different considerations, each with distinct benefits, trade-offs, and operational implications. Prompt engineering offers a fast starting point and retrieval-augmented generation adds adaptability by grounding responses in up-to-date sources. Fine-tuning is the next step when deeper control over behavior, format fidelity, tool-use reliability, tone, and policy adherence is required. It should not be treated as a mechanism for keeping knowledge current or as a guarantee against hallucinations. For knowledge-based questions and facts that change over time, retrieval remains the preferred approach.

Fine-tuning introduces additional costs, operational ownership, and risks. Success depends less on model internals than on the clarity of objectives, the quality of examples, and disciplined evaluation. High-quality, representative data is the foundation of every effective fine-tune, while safeguards against overfitting, bias, and privacy risks ensure the resulting model is trustworthy as well as performant. Choosing between serverless and managed compute in Azure AI Foundry means balancing simplicity and speed against control of infrastructure, hyperparameters, and networking. Selecting among supervised fine-tuning, preference-based optimization, or reinforcement-style training should align with the data you have, the behaviors you want, and the metrics you can measure.

In practice, the most resilient and cost-effective solutions combine techniques rather than relying on any single method. Pair prompting with retrieval for knowledge and compliance, then apply fine-tuning to lock in structure, style, safety guidelines, and tool-calling patterns. This layered approach helps optimize accuracy, latency, maintainability, and governance simultaneously.

For enterprise-grade applications, the objective is not merely to add a few points of benchmark accuracy. It is to shape a dependable component that behaves consistently within the constraints of your product and organization. Used thoughtfully, fine-tuning can reduce inference costs through shorter prompts and smaller models, improve reliability of structured outputs and tools, and create differentiated value by aligning model behavior with your workflows and brand. Used carelessly, it can introduce brittleness, compliance exposure, and unnecessary expense without improving real outcomes.

Fine-tuning is an ongoing cycle of experimentation, evaluation, and refinement. Azure’s ecosystem enables this cycle with serverless options for quick iteration and managed compute for maximum control. Apply these capabilities strategically. Use fine-tuning to shape behavior and outputs that must be consistent, and use retrieval to supply knowledge that must be correct and current. This separation of concerns is the most reliable path to scalable, high-performing solutions.

Chapter 6. Agentic Systems

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the sixth chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

In this chapter, we will define what is an agent, when to use them and how can you build secure agentic systems for enterprise environments. We will outline the key Azure services that support agentic AI deployments, ensuring scalability, security, and efficiency in real-world applications.

At its foundation, an AI agent is an intelligent software system capable of autonomously pursuing goals, using its reasoning and planning capabilities to perform a series of predefined or adaptive actions within an environment. Unlike traditional programs that follow fixed instructions, an agent exhibits a degree of autonomy: it can operate and make decisions independently, without constant human oversight. Its defining strength lies in its reasoning and planning abilities, giving it the capacity to analyze multimodal information, interpret complex contexts, and generate coherent plans and strategies toward achieving its objectives. In doing so, the agent can develop and execute a structured sequence of actions to reach a goal, informed by memory or context from past interactions. Together, these elements transform the agent from a reactive system into a proactive, goal-directed entity capable of adapting to new situations and optimizing its decisions over time.

In practical terms, a generative AI agent integrates several core components that collectively enable these intelligent behaviors. At its center lies a foundation model, typically a large language or multimodal model, that provides the reasoning and language understanding necessary to process unstructured input and produce context-aware, coherent output. Surrounding this model is the context, composed of user instructions or a system prompt, chat memory, and any other additional information that helps define the agent’s role, behavior, and operational constraints. This context may also include knowledge sources (structured or unstructured repositories) that provide grounding in domain-specific data, and memory systems that maintain state, continuity, and learning across interactions. Managing this evolving and multi-layered context is often referred to as context engineering, a critical discipline in the design of advanced agents. Lastly, to interact with the external world, the agent leverages tools or APIs that allow it to perform retrieval, execute actions, or invoke a set of specialized capabilities.

Figure 6-1. LLM components (Ref: What is Foundry Agent Service? | Microsoft Learn)

The diagram above illustrates this architecture: user messages, system events, or other agents serve as inputs to the agent, which processes them through its reasoning engine and instruction layer. When necessary, the agent performs tool calls, for retrieval, actions, or memory operations, and then generates outputs, which can be structured data or messages directed to other agents or systems.

While an AI agent refers to an individual autonomous entity, an agentic system is the overarching architecture that orchestrates one or more agents, together with a collection of tools, data sources, and orchestration logic, to accomplish complex, multi-step workflows. If an agent is a specialized tool in a toolbox, the agentic system represents the coordinated use of those tools to build the entire structure.

This distinction marks a fundamental shift from traditional, deterministic programming toward a hybrid paradigm. Agentic systems uniquely combine the nuanced, context-dependent understanding and adaptability of LLMs with the structured reliability and governance of conventional software engineering. This dual nature allows them both to “think”, reason, plan, and adapt, and to “act”, execute actions, interact with systems, and produce measurable outcomes.

A defining characteristic of such systems is their ability to perform task decomposition: autonomously breaking down a high-level, abstract goal into a series of smaller, actionable subtasks that can be executed sequentially or delegated to other agents. This chaining of reasoning and actions in pursuit of a goal is a hallmark of agentic workflows and represents one of the clearest expressions of autonomous planning in generative AI systems.

Depending on how these components are orchestrated, agentic systems can range from more deterministic designs, where execution paths are largely sequential and predefine, to highly autonomous systems, where the agent dynamically plans, reflects, and adapts to accomplish complex objectives.

An example of an agentic system could be a customer-support intelligence agent assisting a service representative. When a customer contacts support, the agent autonomously retrieves contextual information, including past service purchases, active subscriptions, and previous interactions. It then reasons about the customer’s intent, queries the recommendation engine for compatible add-ons or upgrades, and consults technical documentation to ensure any suggestion complies with current service capabilities and policies. Using this reasoning, the agent plans a multi-step workflow: understanding the request, identifying relevant information, proposing a solution or recommendation, and summarizing the rationale for the human agent to review before responding.

This scenario highlights how agentic reasoning can bridge structured databases, unstructured documentation, and real-time recommendations to produce contextually accurate and actionable outcomes, all while maintaining human oversight and final decision authority. Such designs exemplify how agentic design patterns can be applied to ensure consistency, traceability, and alignment with organizational standards. We will explore this continuum in greater depth throughout the chapter, examining its implications for control, scalability, and reliability in real-world applications.

When to Use Agentic Systems

Building an agentic system introduces new capabilities, but also new complexity. Before adopting this paradigm, it is important to determine whether the problem at hand truly requires autonomy, reasoning, and orchestration across multiple tools or agents. Not every use case justifies the additional layers of planning, coordination, and governance that agentic architectures demand.

Agentic systems are best suited for dynamic or ambiguous environments, where a traditional workflow cannot be fully predefined in advance. Traditional automation pipelines excel when the process is stable, predictable, and well understood. However, when the logic depends on context, when information is incomplete, or when decisions must be adjusted as new data emerges, a reasoning-capable agent provides tangible value. These systems shine in situations where rules alone might be insufficient, and flexible reasoning or planning is required to achieve a goal.

A second indicator that an agentic design is appropriate is the need to combine heterogeneous data sources or tools. When solving a problem requires bridging structured databases, unstructured documents, APIs, or even human feedback, agents can serve as intelligent intermediaries, interpreting, synthesizing, and acting across diverse systems. This adaptability also makes agentic architectures appealing when requirements evolve over time. Because agents rely on goals, tools, and memory rather than hard-coded pipelines, they could adapt to new objectives or data without needing a full redesign.

In contrast, when the workflow is static and deterministic, a conventional approach, such as a pipeline, function orchestration, or workflow automation, usually provides higher reliability, lower cost, and faster performance. Agentic systems inherently introduce some degree of unpredictability due to the probabilistic nature of large language models and the open-ended reasoning they enable. While this flexibility allows for deeper understanding and adaptability, it also demands additional layers of monitoring, governance, and error handling to ensure safety and consistency.

Ultimately, the decision to adopt an agentic system involves a trade-off between determinism and adaptability. Organizations should consider the complexity of the problem space, the variability of data and tools, and the importance of interpretability and control. In practice, many solutions benefit from a hybrid approach, combining deterministic components for predictable tasks with agentic components for reasoning, planning, or synthesis.

Table 6-1. 1. Summary Table: When traditional vs. agentic systems

Criterion | Favour Traditional Workflow | Favour Agentic System
Task complexity & variability | Task is static, predictable, few branches | Task is dynamic, with evolving conditions or multiple solution paths
Decision logic | Rules or fixed workflows suffice | Requires reasoning, planning, or tool invocation
Data & tool landscape | Few, homogeneous data sources | Many heterogeneous sources or APIs to coordinate
Goal evolution | Goals are fixed and well-defined | Goals may evolve or need adaptive pursuit
Collaboration requirement | Single actor or linear process | Multiple agents or tools need to coordinate
Performance & cost | Prioritizes speed, determinism, and low cost | Accepts some latency and variability for flexibility and intelligence
Governance & monitoring | Simple logging and control | Requires observability, traceability, and safety mechanisms

In short, agentic systems are most effective when the environment is complex, the goals are evolving, and the path to success cannot be fully scripted in advance. They empower applications to move beyond rigid automation toward adaptive, goal-driven intelligence, but they must be used judiciously, balancing autonomy with reliability and ensuring that human oversight remains part of the loop.

Agentic Protocols

Agentic systems are not merely “smart applications”; they are dynamic networks of reasoning components that interact with tools, other agents, and even financial systems. To enable this ecosystem, several communication and integration standards are emerging, gradually converging into a de facto protocol stack that defines how agents operate, collaborate, and transact.

At the foundation lies the Model Context Protocol (MCP), which standardizes how an agent connects to external tools and data sources. MCP addresses the long-standing “N × M integration problem,” where multiple models must interact with multiple tools, each requiring bespoke connectors. By introducing a client–server architecture with built-in capability discovery and secure request/response semantics, MCP allows agents to fetch contextual information or execute bounded actions through standardized interfaces. In practice, MCP functions as the “tool bus” of agentic systems, providing a consistent mechanism for extending an agent’s reach into external systems and knowledge sources.

Beyond tool access, agents also need to communicate with each other. Google’s Agent-to-Agent (A2A) protocol defines a common framework for how agents discover one another, advertise capabilities, and coordinate tasks across ecosystems. It provides the mechanisms for establishing sessions, exchanging structured messages, and orchestrating collaboration. Additionally, A2A enables heterogeneous agents from different vendors or runtimes, to work together as peers, facilitating complex, multi-agent workflows and distributed reasoning.

Complementing A2A, Agent Communication Protocol (ACP) developed by IBM Research, focuses on the structure and semantics of inter-agent messaging itself. ACP adopts a lightweight, HTTP/JSON model for request and response exchanges, allowing agents to communicate reliably across platforms and environments. Its design philosophy emphasizes openness and interoperability, and its ongoing contributions under open governance make it increasingly aligned with broader A2A efforts. Together, A2A and ACP could provide the connective tissue that allows agent ecosystems to evolve beyond isolated integrations toward a more standardized, networked intelligence.

On top of the rest of this evolving stack of protocols sits the emerging Agent-to-Payments (A2P) layer, also referred to as AP2, which extends agentic interactions into the economic domain. As agents begin to transact autonomously, purchasing API credits, booking services, or processing invoices, they will require payment rails that incorporate identity verification, policy enforcement, and risk management. Google’s proposed A2P specification envisions a programmable payments layer designed to integrate naturally with A2A and MCP ecosystems, as well as with modern digital ledgers. This layer brings a new dimension to the agentic internet: verifiable, auditable commerce between agents acting on behalf of users or organizations.

Table 6-2. 2. High level summary of the main agentic protocols

Dimension | MCP (Model Context Protocol) | A2A (Agent-to-Agent) | ACP (Agent Communication Protocol) | A2P / AP2 (Agent-to-Payments)
Primary purpose | Standardize tool/data access for LLM apps | Inter-agent discovery, capability exchange, collaboration | Inter-agent messaging with lightweight HTTP/JSON semantics | Transacting: payments & purchases by/for agents
Core model | Client (agent host) ↔ server (tool) | Peer agents with discovery & sessions | Client ↔ server (agent as a service), JSON/JSON-RPC | Payment intents, verification, settlement
Key features | Capability listing, secure tool calls, unify N×M integrations | Capability ads, task handoff, shared context, coordination | Stateless/stateful ops, streaming, offline/local options | Agent identity, spend limits, audit trails, ledger hooks
Maturity (2025) | Rapid adoption across vendors & IDEs | Newly announced, fast-moving | Open standard with ref impls; converging with A2A | Early but strategic for agentic commerce
Good for | Connecting many tools safely & consistently | Multi-agent workflows across teams/vendors | Simple, portable agent comms incl. edge/offline | Controlled purchases, subscriptions, micro-payments

MCP in detail

In this section we will now dig deeper into MCP alone as MCP has swiftly become the de facto foundation for today’s agentic ecosystems. It has been adopted by most major model vendors, integrated into multiple platforms, and addresses the one of the most urgent need in building tool-rich agents: the tool-agent integration explosion. By understanding MCP in detail, its architecture and how it enables tool and prompt orchestration, you gain the plumbing required for everything above it to function.

Figure 6-2. MCP Protocol Definition. Reference: Model Context Protocol

The Model Context Protocol (MCP) essentially becomes the “tool bus” of agentic systems: MCP standardises how agents interface with data, services and tools. At its core, MCP adopts a client–server architecture in which the agent, acting as an MCP client, reaches out to one or more MCP servers that wrap external capabilities. Those servers publish their capabilities, for example “file browse”, “database query”, “send email” or “launch workflow”, and expose structured, standardised interfaces for invocation. The agent begins by establishing a session, discovering what capabilities are exposed by the servers, and then invokes those tools or queries through a consistent protocol. Rather than writing bespoke connectors for each model-tool pair, architects build or adopt a smaller set of MCP servers and enable many agents to reuse the same interface.

Because the interfaces are standardised, MCP empowers the agent orchestration logic to be generic and adaptive. An agent might begin by discovering a “document search” tool via the server, invoke it to retrieve relevant assets, then decide to call a “information-extraction” tool, followed by a “send-email” tool. Because all these are exposed via MCP servers, the agent’s workflow remains portable across domains, modular, and vendor-agnostic. This chaining of actions is central to what makes an agent “tool-rich” and supports complex multi-step workflows where context, decision-making and tool invocation loop together.

On the client side, the Model Context Protocol (MCP) introduces key primitives that support contextual control, secure boundaries and interactive workflows.

Roots define the specific entry-points or scopes that a client exposes to a server (for instance, a workspace, a file path or database schema) and thus mark the portions of the environment the server is permitted to operate on.

Sampling enables an MCP server to request the host model (via the client) to generate a completion (essentially a sub-query or “think step”) during a session, thus giving the server the ability to delegate reasoning back to the model under the client’s control.

Elicitations allow a server to request user-input (or explicit client-mediated input) as part of the workflow to gather clarification, consent or additional parameters before invoking a tooll

Together, these client-side features empower the MCP client not merely as a passive invoker of tools, but as a managed reasoning environment: it defines what the server may access (roots), allows the server to trigger model-generated reasoning (sampling), and supports interactive query UI or user intervention where required (elicitations).

On the server side, the Model Context Protocol organises its capabilities across several intertwined domains: prompts, resources, tools and utilities:

Prompts are predefined templates that servers can expose to guide or parameterize agent reasoning, such as summarization, classification, or query reformulation patterns.

Resources represent data objects or repositories (e.g., files, databases, APIs) that the agent can browse or query through standardized endpoints.

Tools are actionable operations that modify state or trigger workflows, such as sending messages, updating records, or invoking external APIs.

Utilities provide auxiliary capabilities, including schema discovery, metadata inspection, or environment setup that streamline how clients understand the server’s environment.

By separating these layers, MCP servers enable modular composition of reasoning assets and simplify discoverability for agents.

In enterprise environments MCP’s separation of responsibilities between the agent (MCP client) and the external system (MCP server) provides an architecture that supports governance, auditability and control. In a best-practice architecture, the MCP server does not function as a standalone identity authority. Instead, identity and consent are delegated to a central authorization layer, typically implemented via OAuth 2.1 and an enterprise identity provider such as Microsoft Entra ID. This upstream layer authenticates users, issues tokens, and enforces consent and least-privilege scopes. The MCP client then presents the enterprise-issued token to the MCP server, which validates the token, applies role-based tool access, and logs all invocations. Observability and policy enforcement are further strengthened by routing through an API gateway or identity-aware proxy. With this design, agents only need to invoke a standardized MCP interface, while identity verification, consent, scoping, and auditing remain managed within the enterprise identity fabric.

Figure 6-3. Basic MCP Authorization flow. Reference: Authorization - Model Context Protocol

Naturally, implementing MCP at scale raises important design and operational considerations. Since context retrieval may return large documents or extensive data sets, agents must decide what to fetch and what to summarise: careless design risks exceeding token budgets or overwhelming the reasoning loop. Tools exposed via MCP must carry rich metadata: parameter definitions, expected output types, latency/cost indicators, failure behaviours, so the agent can reason about tool choice, cost/benefit or fallback strategies. Session lifecycle must be managed: when to open and close sessions, when to refresh or invalidate cached context, how to manage state across agent invocations. And of course, security, identity, and permissions become critical: MCP servers with overly broad tools or lax access control risk unintended data exposure, malicious invocation chains or context injection attacks.

Finally, consider how MCP positions your agentic architecture for scale and ecosystem growth. With MCP as the backbone, agents become interchangeably modular, tool-catalogues become first-class artefacts, and agent manifests become the enabling schema of orchestration. Agents from different domains or vendors can plug into the same MCP servers; new MCP servers (tools/data sources) can be added and discovered dynamically by the agent clients; multi-agent workflows become plausible because they share a common interface to interact with tools and context. In short: MCP turns what would have been brittle, handcrafted point-to-point integrations into a standardised, interoperable substrate on which collaboration (via your higher-level A2A/ACP layers) and commerce (via A2P) can flourish.

Agentic Design Patterns

In the evolution of agent-centric systems, a recurring challenge is how to organise and coordinate autonomous agents, so that they behave cohesively, predictable enough for enterprise use (especially in the context of compliance, governance and deployment on platforms such as MicrosoftI Foundry) yet flexible enough to exploit agentic autonomy. Design patterns for agentic systems provide architects with reusable templates for structuring agent interaction, control-flow and collaboration. Below are eight patterns you should consider when designing agentic systems on Azure.

Sequential Pattern

The sequential pattern envisions a chain of agents or sub-agents where each one performs a discrete step and hands off to the next. In this style, agent 1 takes input, produces an output, which becomes the input for agent 2, then 3, and so on until a final result is produced. The appeal of the sequential pattern lies in its simplicity: clear boundaries, deterministic flow, easier auditing and traceability. In an Azure context you might see Agent 1 performing retrieval (via Azure AI Search), Agent 2 performing reasoning or summarisation (via an LLM), Agent 3 performing compliance review or metadata tagging (via tool-calling). Because each step is well-defined, it becomes easier to enforce guardrails or logging and to align with audit trails mandated by the EU AI Act. The catdownside is that the linear flow can become a bottleneck, and if an upstream agent fails or stalls, the entire chain halts. Furthermore, the modelled task must decompose naturally into discrete sequential stages; if the process is more iterative or interactive, this pattern may feel too rigid.

Figure 6-4. Sequential pattern

Workflow Pattern

The workflow pattern generalises sequential flow into a more flexible orchestration of agents across a defined business or functional workflow. Here the system architect defines a graph of agent invocations (including branches, joins, loops, and decision-points) where each node is an agent and the edges direct the flow of control and data. For instance, in an enterprise-grade solution you could have an upstream agent that acquires a user request, then sending it to a finance agent, a. legal and technical agent to accomplish different task in each of their domains. After domain processing, a review agent may decide whether the result should go back for rework or proceed for final orchestration. The workflow pattern enables more realistic business processes, including exception handling, parallel branches and returns. This pattern is natively supported by Microsoft Foundry. The challenge lies in modelling the workflow correctly, ensuring that branching and loops do not lead to uncontrolled agent invocation or unnecessary latency, and maintaining observability across branches.

Figure 6-5. Workflow Pattern Example

Concurrent Pattern

In many real-world tasks the workload naturally decomposes into pieces that can be processed in parallel rather than strictly in sequence a scenario that the concurrent pattern addresses. In the concurrent pattern, multiple agents execute simultaneously (or near-simultaneously), each working on a separate sub-task, and then the results are reconciled by a merge or aggregator agent. For example, when processing a complex document you could have a language-agent finishing summarisation, another checking factual consistency, a third doing taxonomy tagging, all in parallel; once they complete, a aggregator agent combines results, resolves conflicts or duplicates, and produces the final output. The advantage is speed and utilisation of specialised agents, but you must carefully design for partial failures (one agent fails while others succeed), and you must ensure that the aggregator logic handles inconsistencies, duplicates or missing outputs reliably.

Figure 6-6. Concurrent Pattern Example

Group-Chat Pattern

The group-chat pattern brings multiple agents (and optionally a human) into a shared conversational context, where they can exchange messages, debate, propose actions and converge on a solution. Instead of rigid hand-offs, agents co-exist in a “chat room” and send messages to each other, perhaps moderated by a facilitator agent or by the system. In an enterprise scenario you might have a content-generation agent, a compliance-agent, a style-agent, and a reviewer agent all participating in a common thread: the content generator posts a draft, the compliance member raises concerns, the style agent suggests rewriting, and the reviewer adds feedback, iteratively until consensus. This pattern mirrors human team collaboration and can surface richer emergent behaviour, but has increased complexity: ensuring coherence, avoiding infinite loops of messaging, controlling the number of interactions and monitoring the conversation become essential. The group-chat pattern excels when tasks benefit from multi-agent deliberation rather than a simple workflow.

Figure 6-7. Group Chat Pattern Example

Self-Reflection / Cross-Reflection Pattern

Reflection is a powerful meta-pattern where agents pause to review, critique or refine their own output (self-reflection) or review each other’s output (cross-reflection). In self-reflection, a single agent receives its preliminary output, prompts itself to ask “what could be improved?” or “does this satisfy constraints?”, and then revises its answer. In cross-reflection, one agent hands off to another agent whose role is to evaluate the prior agent’s work (for example a critic agent), suggest revisions, and optionally send it back for improvement. Reflection patterns are increasingly recognised as enhancing reliability, reducing hallucinations or mistakes, and improving alignment. For example, you might build a system on Azure where an LLM-agent generates a legal document, then a “critic-agent” evaluates it for compliance and returns it for revision, and finally the original agent applies it. This iterative loop may continue until the document passes defined metrics. The trade-off is additional latency, more LLM calls (hence more cost), and more complex termination logic (when to stop iterating?). But the benefits, especially for enterprise trustworthiness, are often worth it.

Figure 6-8. Cross-reflection Pattern Example

Handoff Pattern / Human-in-the-Loop

The handoff pattern focuses on the controlled transfer of responsibility between agents or between an agent and a human, when a task requires different expertise, decision authority, or contextual awareness. Rather than a single agent attempting to handle all steps of a process, tasks are passed along a chain of specialized participants. Each participant receives sufficient context from the previous stage to continue seamlessly, avoiding redundant reasoning or data loss. In enterprise settings, this pattern often manifests as an “agentic relay”: for example, a customer-support agent collecting context before escalating the case to a compliance or legal agent. When humans participate in this chain, the pattern becomes a form of human-in-the-loop orchestration, where humans intervene for oversight, ethical validation, or ambiguous edge cases. Such human participation not only improves accountability but also supports regulatory alignment, ensuring that sensitive or high-impact decisions remain traceable and explainable.

Handoffs require clear definitions of entry and exit criteria, shared memory or structured context exchange, and confidence thresholds that determine when to transfer control. When well designed, this pattern enhances both robustness and transparency, allowing complex workflows to combine the precision of automation with the judgment of human review.

Figure 6-9. Handoff Pattern Example

Supervisor Pattern

The supervisor pattern introduces a higher-level coordinator or “meta-agent” that monitors, delegates, and manages the execution of other agents. Acting as an intelligent router, the supervisor evaluates incoming tasks, determines which specialist agent or subsystem should take responsibility, and tracks progress across concurrent or sequential workflows. In multi-domain or large-scale environments, such as enterprise knowledge systems, this pattern prevents chaos by ensuring that each request follows a coherent route, from classification to completion, while maintaining observability and compliance.

Beyond task routing, a well-designed supervisor implements escalation policies, error handling, and performance feedback loops. It may dynamically spawn, pause, or retire sub-agents depending on workload or resource constraints. However, designers must be cautious: excessive centralization can create bottlenecks or single points of failure that undermine the adaptive, distributed nature of agentic systems. The most effective supervisor agents strike a balance, providing governance and orchestration without stifling the autonomy and parallelism that make agentic architectures powerful.

Figure 6-10. Supervisor Pattern Example

Magnetic Orchestration Pattern

Finally, the Magentic orchestration pattern is tailored for scenarios where a fixed, predetermined flow of operations doesn’t suffice and instead, you require a system that dynamically constructs, adapts and evolves a plan of action while coordinating multiple specialised agents. In this model, a “manager” or “controller” agent functions as the orchestration plane: it decomposes an overarching goal into subtasks, assigns those subtasks to specialist agents (each with their own domain expertise and tool access), monitors execution, updates the work-ledger with progress or new subtasks, and ensures that the system converges on a solution even when new information or unexpected outcomes arise.

When implemented, the manager agent begins with the goal and then autonomously builds or evolves a task ledger or internal plan: listing subtasks, their dependencies, status, and which specialist agent is responsible. Specialist agents then tackle their allocated subtasks, often invoking external tools, APIs or systems, publishing their results back to the ledger. The manager may then spawn additional subtasks, merge branches, escalate to human review, or trigger automated system changes, effectively guiding the system as it self-organises towards the goal. What distinguishes this pattern from simpler workflows is the adaptability: the orchestration is not purely static or predetermined, it evolves based on runtime results and conditions.

Architecturally, for example in a cloud environment, you might map the pattern by having an orchestrator component (e.g., a durable workflow engine or agent-framework manager) that maintains the task ledger in persistent state (such as a database or stateful store), dispatches specialist agents (each as service/function/container) via message queues or event streams, and captures all tool-invocation and agent-activity logs for traceability. The ledger acts as the authoritative state of what subtasks remain, what paths have been taken, and what next actions are required. Because specialist agents may effect changes in external systems (for example database updates, provisioning resources, notifications), you must design for audit, rollback, error-handling and lifecycle control. This pattern is especially suitable when the problem domain is not clearly linear, when new subtasks are discovered during processing, when multiple tools must be invoked, and when system actions (not just reasoning) must be taken.

As with all patterns, magentic orchestration carries trade-offs. The flexibility and power to adapt mean increased complexity in orchestration logic, state management, termination criteria (when does the task ledger finish?), risk of runaway subtasks, higher latency (because multiple agents and plan revisions may occur), and higher operational overhead (monitoring, logging, auditing). Given its dynamic nature, strong observability and governance are required to prevent spiralling complexity or undetected tasks.

In summary, when your system needs to tackle a complex, evolving problem, engage many specialised agents, make tool-based or system-based actions, and adapt the plan during execution, then the Magentic orchestration pattern provides a robust architectural model for multi-agent coordination.

Figure 6-11. Magentic Orchestration Pattern

Summary

When designing agentic systems on Azure, the selection of a design pattern should be driven by your task complexity, reliability requirements, tool-set size, governance and compliance constraints, and team structure. A simple sequential pattern may suffice for predictable flows; workflows and concurrent patterns scale to richer business processes; group-chat and reflection patterns raise agentic maturity and reliability; supervisor/handoff gives you architectural control; and magnetic orchestration provides adaptability for large ecosystems. Whichever pattern you choose, layering in your agent-manifest (metadata per-agent: capabilities, tools, lifecycle, compliance attributes) will help align with enterprise practices and the mandates of regulatory frameworks.

Building Agentic Systems in Azure

Microsoft Foundry Agent Service is the centerpiece of Azure’s approach to agentic AI systems, moving beyond standalone models to autonomous agents that can take actions and complete goals. In previous chapters we explored Foundry’s model catalog, which provides a roster of powerful language models (GPT-5, GPT-4.1, Grok4 , etc.). Now, we will see how Agent Service builds on the previously seen foundations to enable you to construct goal-driven AI agents that operate in real-world workflows.

How the Microsoft Foundry Agent Service Works

Under the hood, Microsoft Foundry Agent Service runs and manages these agents in a robust, enterprise-grade environment. It orchestrates the agent’s entire runtime managing the conversational thread, deciding when the agent should call a tool, executing that tool call, and handling the results. Key capabilities of the Agent Service include:

Tool Orchestration & Execution: When an agent needs to use a tool, the Agent Service handles the call to that tool’s endpoint for you. It will pass along the proper context, execute the tool (with retries or error handling if needed), and return the result to the agent’s reasoning process. Developers don’t have to write glue code for tool usage, the service orchestrates it seamlessly, logging each step.

Knowledge Management & Retrieval (FoundryIQ): Agents in Foundry can natively connect to FoundryIQ knowledge bases, which provide a structured and secure way to ground agent reasoning in enterprise data. These knowledge bases allow agents to perform retrieval over documents, records, and other indexed content using semantic search, filtering, and relevance ranking. Instead of building custom retrieval pipelines, developers can link an agent to a FoundryIQ knowledge base so it can query organizational information directly during its reasoning process. This integration ensures that agents operate with accurate, governed, and up-to-date context from the enterprise’s own knowledge sources.

Multi-Agent Coordination: Foundry natively supports agent orchestration workflows. Agents can be orchestrated seamlessly through the UI in a drag and drop tool or by using YAML files describing the full workflow.

State Management & Memory: The service maintains a structured thread for each conversation or task. All interactions (user queries, agent responses, and tool invocations) are captured in this thread with a consistent format. This provides visibility into the agent’s decision process for debugging and also serves as memory for the agent to draw context from

Observability and Debugging: Every action an agent takes is traceable. Microsoft Foundry integrates with Azure Monitor and Application Insights, so you can inspect logs, see telemetry, and even replay or step through an agent’s reasoning process (also called tracing an agent). For example, you can view each tool call and its result, and each intermediate message the agent generated. This fine-grained visibility is invaluable for testing and improving agents over time.

Trust, Safety, and Governance: Because these agents are powerful (they can potentially write code, send emails, or change data), safety is paramount. Agent Service has content filters and policy enforcement built in. It will automatically apply guardrails to the agent’s outputs and tool usage (preventing disallowed content or certain unsafe actions). Moreover, it’s integrated with Microsoft Entra ID (Azure AD) for identity and access control.

Fully Managed Infrastructure: The heavy lifting of hosting is abstracted away. Agents run as a service: Azure manages the compute, scaling, networking, and storage behind the scenes. You don’t need to provision servers for your agent or worry about how to scale it under load; the Agent Service takes care of those considerations. By design, an agent is packaged as a microservice that Azure can deploy and scale as needed. This is sometimes referred to as “Agent as a Service (AaaS)”, highlighting that the agent is delivered a

Identity and security integration: Agents running in Foundry inherit enterprise-grade identity (via Azure Active Directory / Entra Agent ID) and access control. This means agent actions can be authenticated and authorized as needed (e.g., an agent using a company’s database will use managed credentials). Networking controls and encryption are built-in, addressing the need for trust in AI automation.

In short, Microsoft Foundry Agent Service provides a production-ready runtime for AI agents, handling the complexity of tool integration, concurrency, state, and safety. It lets you focus on an agent’s logic and purpose, rather than the plumbing.

For example, if you were building an AI agent to automate customer support emails, you would pick a suitable model (from the model catalog), prompt it with instructions on your support policies, and give it tools like email sending and knowledge base search. Once configured, when a new support ticket comes in, the agent can autonomously draft an email response (using the model), look up relevant FAQ articles (using the search tool), and send the email (using the email sending tool), all orchestrated under the hood.

Supercharging Agents: Tool Catalog and Foundry IQ

Beyond runtime orchestration, one of the most powerful capabilities of Microsoft Foundry Agent Service lies in how it extends what agents can do and know. This extensibility is driven by two complementary pillars: the Tool Catalog, which defines how agents take action, and FoundryIQ, which defines how they access and reason over enterprise knowledge. Together, they transform agents from isolated conversational systems into deeply integrated components of an organization’s digital fabric.

The Tool Catalog provides a structured, governed registry of all the tools an agent can use. Tools can represent APIs, data access endpoints, business workflows, code execution environments, or any external capability wrapped and published using the Foundry tool interface. Instead of embedding one-off integrations inside an agent’s logic, tools are registered once and made discoverable to any agent. This clean separation between agent, tool, and capability lets teams maintain governance boundaries, control access by identity, and reuse integrations across multiple agents or workflows. Because the Agent Service orchestrates the full lifecycle of a tool call, developers don’t need to write glue code to integrate backend systems. They define capabilities in the Tool Catalog, and agents use them dynamically as part of their reasoning process.

Complementing this is FoundryIQ, which provides the knowledge substrate agents rely on for grounding and contextual understanding. FoundryIQ knowledge bases are backed by Azure AI Search and allow agents to retrieve the most relevant information from structured and unstructured data. By connecting an agent to one or more knowledge bases, you allow the model to enrich its reasoning with authoritative, organization-specific information, without building custom retrieval pipelines or maintaining separate search infrastructures. This enables agents to answer questions using enterprise data. Together with the Tool Catalog, FoundryIQ ensures that agents have both the actions they can perform and the knowledge they can draw from aligned with enterprise requirements.

When combined, the Tool Catalog and FoundryIQ effectively “supercharge” agents. Tools give agents the ability to act on systems autonomously, while knowledge bases allow them to reason using high-quality, up-to-date enterprise context. This pairing is what elevates agents from simple conversational assistants to operational, decision-support, and workflow-automation systems integrated across an entire organization.

Foundry Workflows: Orchestrating Multi-Agent Processes

As powerful as a single agent can be, many enterprise scenarios benefit from multiple agents working in concert, or mixing an agent’s capabilities with classic business-process logic. That’s the role of workflows in Foundry. The Foundry Agent Service supports agentic workflows, where you orchestrate multiple agents, tool calls, and business logic within one process.

Workflows make it possible to coordinate complex, multi-step tasks that go beyond what a single agent can handle. In Microsoft Foundry, this orchestration is achieved through a no-code or low-code, drag-and-drop interface that lets you visually design how data and actions move from one step to another. Workflows can incorporate the agentic patterns discussed earlier in this chapter, such as sequential or parallel execution and include logic nodes for conditionals, loops, or branching decisions. Each agent node can invoke any existing agent from your catalog allowing you to reuse and compose agents as modular building blocks within larger processes.

Figure 6-12. Example of a Foundry Workflow

From a developer’s perspective, workflows are backed by YAML definitions that can be easily inspected or edited. The platform automatically manages data flow between steps, passing outputs, synchronizing parallel branches, and handling dependencies, while maintaining version history for each change. This combination of visual design and structured configuration allows both low-code creators and developers to collaborate effectively on the same workflow artifacts.

Example 6-1. 1. Yaml representation of the previous Foundry Workflow depicting a Human in the loop pattern example

kind: workflow
trigger:
  kind: OnConversationStart
  id: trigger_id
  actions:
    - kind: SetVariable
      id: set_project
      variable: Local.OriginalInput
      value: =System.LastMessage.Text
    - kind: Question
      id: question_confirm
      variable: Local.ConfirmedInput
      prompt: "CONFIRM: Please re-enter your input to confirm"
      entity: StringPrebuiltEntity
      alwaysPrompt: false
    - kind: ConditionGroup
      id: check_completion
      conditions:
        - id: check_confirm
          condition: =Local.OriginalInput <> Local.ConfirmedInput
          actions:
            - kind: SendActivity
              id: sendActivity_mismatch
              activity: '"{Local.ConfirmedInput}" does not match the original input of "{Local.OriginalInput}". Please try again.'
            - kind: GotoAction
              id: goto_again
              actionId: question_confirm
      elseActions:
        - kind: SendActivity
          id: sendActivity_confirmed
          activity: |-
            You entered:
                {Local.OriginalInput}

            Confirmed input:
                {Local.ConfirmedInput}
id: ""
name: HILtest
description: ""

In essence, agent workflows extend the reach of automation by enabling multiple specialized agents to collaborate within a single, orchestrated pipeline that supports loops, branching logic, and sequential dependencies. This capability is particularly powerful in enterprise scenarios where end-to-end processes often span multiple stages, such as data retrieval, analysis, approval, and execution. For example, a workflow might first trigger a chatbot agent to collect user input, then pass the results to a database agent for querying internal systems, and finally invoke a reporting agent to compile and deliver the outcome; all seamlessly coordinated. With workflows, you design and operationalize these processes in one place, bringing the power of autonomous agents into structured, governed business automation.

Microsoft Agent Framework (from Semantic Kernel and AutoGen to Present)

Microsoft Agent Framework is the next-generation open-source SDK and runtime for building agentic applications. It represents the convergence of two previous efforts, Semantic Kernel and AutoGen, and was released in public preview in 2025.

At a high level, the Agent Framework offers a unified programming model where you define agents, their tools, memory, and orchestrate interactions either through deterministic workflows (inherited from Semantic Kernel) or through dynamic LLM-driven orchestration patterns (inherited from AutoGen).

Microsoft Agent Framework has built-in support for open standards such as Model Context Protocol (MCP) for tool interoperability and Agent-to-Agent (A2A) communication protocols. This means an agent can dynamically find and invoke tools that speak a common language (MCP), and agents can message each other in a structured way, even across different runtimes or languages, making the system more interoperable and portable.

The Agent Framework inherits Semantic Kernel’s enterprise-grade connectors (for memory, data stores, telemetry, etc.) and the advanced multi-agent patterns from AutoGen (debate, reflection, facilitator/worker, multi-agent orchestration). It also has built-in support for OpenAPI, enabling REST APIs with OpenAPI specifications to be imported as tools..

Logging, observability, compliance hooks, and support for long-running tasks (including threads, human-in-the-loop scenarios, and durable state) are first-class in the framework. This reflects the enterprise readiness of the SDK.

From an Azure perspective, the Agent Framework integrates with the Azure AI Foundry ecosystem, allowing agents prototyped locally (in Python or .NET) to be deployed to Azure-hosted services for scale, reliability, and governance.

In summary, the Microsoft Agent Framework unifies earlier agent-orchestration frameworks into a single, extensible foundation. It lets developers move from experimentation to enterprise deployment without switching tooling, reducing fragmentation, improving developer experience, and setting a reliable foundation for real-world, production-grade multi-agent systems.

Hosted Agents (Preview): Deploying Agent Framework Apps to Azure

Hosted Agents are the bridge between developing agents locally with the Microsoft Agent Framework and running them as fully managed, production-ready services in MicrosoftI Foundry. After defining and testing an agent using the Agent Framework’s unified model, with its tools, orchestration patterns, memory, and interaction logic; you package it as a container and deploy it to the Foundry Agent Service as a hosted agent. In this environment, Azure takes over the operational responsibilities: provisioning compute, managing autoscaling, handling conversation state, securing identities, and providing observability through telemetry and tracing. This allows the same agent definitions used during development to run reliably at cloud scale with enterprise governance and without requiring teams to build the surrounding infrastructure themselves.

In the hosted-agent model, the runtime can also handle conversation history, tool invocation records, identity, telemetry, logs/traces, and scaling of replicas as load changes. This significantly reduces infrastructure complexity and lets development teams focus on agent logic, tools, and workflows rather than servers and orchestration.

In essence, Hosted Agents bridge the gap between local prototype and cloud production: you use familiar development workflows, then with containerization and registration you gain enterprise-grade scale, governance, identity, stateful conversations, and telemetry, all without building the entire cloud stack yourself.

Conclusions

Agentic systems mark a shift from traditional prompt-based applications to dynamic ecosystems of reasoning entities that can plan, act, and collaborate. In this chapter, we explored what defines an agent, when agentic systems are most effective, and how design patterns such as workflow, reflection, handoff, and magnetic orchestration give structure to their behavior. These patterns turn models into adaptive problem-solvers capable of learning from context and coordinating with other agents or humans.

Underlying this evolution are emerging standards like the Model Context Protocol (MCP), Agent-to-Agent (A2A) communication, and related frameworks that provide interoperability, governance, and security. They form the connective tissue that enables agents, tools, and environments to operate cohesively within enterprise systems.

Azure brings these concepts to life through the Microsoft Foundry Agent Service, which provides the infrastructure to build, deploy, and monitor multi-agent solutions at scale. Combined with other Azure services, and orchestration frameworks such as Microsoft Agent Framework, it enables organizations to build intelligent systems that are both autonomous and compliant.

Ultimately, the rise of agentic systems marks the beginning of a new software era; one where applications are not just responsive but reasoning, not just functional but adaptive. They encapsulate the essence of generative AI on Azure: transforming models into collaborators, data into context, and cloud infrastructure into a living network of intelligent systems.

Chapter 7. GenAIOps and LLMOps in Azure

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the seventh chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

From MLOps to GenAIOps: Why It Matters

Operations in AI Applications are not new. For almost a decade, enterprises have relied on MLOps (Machine Learning Operations) to bring discipline and rigor to traditional ML and data intensive projects. The objectives were clear: make models reproducible, ensure scalability and manage their lifecycle with the same principles that DevOps brought to software engineering. Azure Machine Learning provided the foundations for this: pipelines for training, a model registry, automated retraining triggers and deployment workflows.

Generative AI, however, introduces a fundamentally different and more complex paradigm. Instead of training a custom model from scratch, multidisciplinary teams now orchestrate powerful pre-trained foundation models (GPT, Phi, Llama, etc.) delivered via APIs or managed infrastructure in order to build brand new solutions. The complexity shifts from training data pipelines to prompt design, model evaluation, safe orchestration and lifecycle management of model interactions. This is where GenAIOps emerges as a new discipline: not a replacement for MLOps, but an extension of it, adapted to the realities of generative AI applications.

The differences are notable:

Inputs, not just weights: In ML, the challenge was curating and labeling data for training. In GenAI, the challenge lies in prompts, grounding the model and context windows, among others. A poorly engineered prompt can derail performance as much as a bad dataset did in classical ML

Continuous evaluation of outputs: Accuracy is no longer the main metric to strive for. We now must monitor safety, factual grounding, model drift or latency, among others, all of which can change depending on prompts and context.

Cost and latency trade-offs: Because inference costs greatly scale with tokens and throughput, quota management and accurate cost management are critical operational levers.

Guardrails by design: Unlike traditional ML, where bias testing came late, GenAI applications must embed content filters, prompt shields and evaluation tests as first-class citizens.

Multi-modal and agentic orchestration: Enterprises increasingly deploy not one model, but a myriad of models and tools: multi-modal models, embeddings, search indexes, speech APIs and agents that combine them.

Some practitioners use the term LLMOps to emphasize operations for large language models. While useful in narrow contexts, it underplays the fact that modern solutions are inherently multi-modal and cross-model. Following Microsoft’s framing in AI Foundry, this book will adopt the broader term GenAIOps, the discipline of operationalizing generative AI responsibly, from model selection to governance.

The focus of this chapter is on practical architecture patterns and enterprise-grade techniques for AI professionals who already understand the fundamentals of large language models (LLMs) and are ready to build production-ready applications. We’ll walk through the critical components required to move from experimentation to deployment on Azure, including model hosting, vector stores, orchestration frameworks, low-code integration tools and web front-ends. Along the way, we’ll explore real-world design choices, highlight trade-offs (such as cost vs. performance) and provide actionable guidance for assembling scalable, secure and maintainable GenAI solutions. To reinforce learning, we include illustrative scenarios and a quiz at the end of the chapter to test your understanding.

GenAI in Azure: The Operating Loop

In practice, GenAIOps can be thought of as an operational workflow with four pillars (as shown in Azure AI Foundry):

Define + Explore: Frame the use case or problem, pick candidate foundation models and run quick, repeatable experiments with prompts and parameters. Use the Model Catalog and Evaluations to compare quality, latency and cost on task-specific datasets before you invest in deeper customization.

Build + Customize: Turn prototypes into applications grounded in your data via RAG, fine-tuning where it pays off and clean orchestration (Prompt Flow, Semantic Kernel or Agent Service). Treat prompts, tools and agent roles as versioned artifacts, and encode safety and business rules directly into the workflow.

Observe + Optimize: Ship with CI/CD, then instrument aggressively, trace calls, capture tokens, latency and groundedness. Use canary releases and cost-aware routing (small model first, escalate when needed) to keep quality high and spend predictable.

Protect + Govern: Build safety in with Prompt Shields on inputs, Content Safety on outputs, with policy nodes for disclaimers, PII masking and allowed sources. Enforce RBAC, private networking and policy-as-code so every change is auditable and compliant across dev, staging, and prod.

This lifecycle reframes how companies think about AI applications: not as a one-off experiment, but as a continuous cycle of evaluation, development, deployment and improvement.

Use Case: Multi-Agent Financial Compliance Assistant

Imagine a global bank deploying a GenAI-powered compliance assistant. The system ingests internal policy documents, regulator updates, and case histories. It uses RAG pipelines to ground answers, multi-agent orchestration (one agent retrieves, another validates, a third drafts responses), and exposes outputs through a chat interface for analysts.

Challenges emerge quickly:

Evaluation: Which model is best for summarizing long legal documents, GPT-4o mini, Phi-4, or a fine-tuned Llama-3?

Latency: Analysts need near real-time answers, a 10-second response is unacceptable.

Cost: The system must handle thousands of queries/day. How do you prevent runaway inference bills?

Safety: Responses must never “hallucinate” regulatory interpretations. Guardrails and groundedness evaluation are non-negotiable.

Observability: If an analyst challenges a response, can you trace back which model, prompt and knowledge base entry produced it?

Governance: How do you prove to auditors that the system complies with internal and external rules?

1. Define and Explore

Once you have identified a solid use case to develop, like the financial compliant assistant, how do you know which model/s are the right one for you? The first operational step in GenAIOps is ensuring that the model/s you choose for your application are measured, compared and continuously monitored. Without robust evaluation, companies risk deploying systems that appear impressive in a demo but collapse or cost them millions under real-world usage.

Quantitative Analysis and Monitoring - Core Model Performance Metrics

Traditional ML metrics like accuracy, precision, recall and F1-score are still relevant for generative models, but mostly when there is a well-defined ground truth. For example, if using an LLM to classify text or extract information (a task with correct answers), metrics such as accuracy or F1 can measure how often the model’s output matches the expected result. However, gen AI often tackles tasks that are more open-ended (summarization, coding, free-form Q&A) and less deterministic, where exact matches to a reference aren’t the only measure of quality. In these cases, and as seen in depth in Chapter 3, teams rely on a mix of automated metrics and human judgment to assess performance:

Accuracy and Quality: Traditional metrics like BLEU, ROUGE or exact-match still apply in tasks like summarization or Q&A. For reasoning tasks, benchmarks such as MMLU, GPQA or HumanEval provide more realistic signals.

Latency: Time-to-first-token and overall response time are critical, especially for specific use cases like interactive assistants or real-time voice agents. A delay of more than 300–500 ms can make an assistant unusable in production.

Throughput: The number of requests per second a deployment can handle without degradation. This becomes critical when scaling to thousands or millions of users.

Resource Utilization: GPU/CPU usage, token throughput per PTU and memory overhead. These metrics directly map to cost and scalability.

Safety Metrics: Frequency of harmful, biased or policy-violating outputs. In Azure, this is measured via Content Safety dashboards and triggered through Prompt Shields or moderation layers.

Model Selection and Evaluation

As discussed in length, again, in Chapter 3, Azure AI Foundry simplifies this process through the Model Catalog and Evaluation SDK. A developer can:

Select multiple candidate models (e.g., GPT-4.1, Phi-4 or Llama 3).

Run side-by-side tests using either standard benchmarks or custom test sets.

Collect quantitative and qualitative scores (accuracy, latency, cost).

For instance, a financial services company comparing GPT-4.1 with Phi-4-mini for summarizing regulatory documents might find that:

GPT-4.1 achieves higher factual accuracy but is slower and more expensive.

Phi-4-mini delivers lower cost and latency, but requires additional grounding to match accuracy.

This comparative insight helps developers and AI architects make evidence-based trade-offs, rather than relying on vendor claims or intuition.

Tools and Frameworks

A variety of tools and frameworks have emerged to operationalize evaluation in practice, ranging from Azure-native SDKs to open-source libraries and custom domain-specific benchmarks. Some of the most widely used include:

Azure AI Foundry Evaluations SDK: Provides automated workflows to evaluate models on structured test sets, red-team prompts or live traffic. Results can be logged to Azure Monitor for long-term tracking.

LangChain Evaluation (LCEL): Popular in open-source, integrates with Azure to test RAG pipelines end-to-end. Particularly useful for reasoning and multi-step agent workflows.

TruLens: An open-source library for monitoring and evaluating LLM-based apps. It can integrate with Azure Foundry and provides scoring for relevance, correctness and grounding.

Custom Benchmarks: Enterprises are encouraged to build domain-specific test suites (e.g., healthcare compliance checks, legal consistency tests) that capture the nuances of their use case.

Together, these tools form the evaluation foundation of GenAIOps: systematic, repeatable, and grounded in both quantitative metrics and qualitative human review.

It’s also important to note that as enterprise use cases expand, evaluation is no longer text-only. Azure AI Foundry is evolving to support multi-modal pipelines, where evaluations can cover text, images and even speech outputs in one workflow. This reflects a broader industry trend toward multi-modal GenAI, where grounding and safety must be tested across different modalities, for example, ensuring generated captions align with an image or that audio responses meet both quality and content safety thresholds.

If evaluation establishes the baseline for a model’s quality, end-to-end operations ensure that these models can be reliably deployed, updated and governed in production. This is the heart of GenAIOps: treating generative AI systems not as one-off prototypes, but as production-ready living systems that evolve under continuous integration, deployment and monitoring pipelines.

2. Build + Customize

2.1 - Designing the Application Architecture

Before any code is written, you must define the architecture of your AI solution. The architecture determines how the model interacts with other components, how data flows and where the system enforces safety and compliance. Common building blocks include:

Model Selection: As outlined in the previous section.

Compute and Hosting: Generative AI workloads require elastic and sometimes GPU-intensive compute. On Azure, models can be deployed via Azure Machine Learning endpoints, Azure OpenAI Service (fully managed) or containerized into AKS or Container Apps for more control. For lighter workloads or edge scenarios, Azure Functions or App Service may suffice.

Orchestration Layer: SDKs like Azure AI Foundry, Semantic Kernel or Langchain provide orchestration, chaining prompts, function calls and external tool integrations.

Knowledge Integration: Most enterprise apps require retrieval-augmented generation (RAG) to ground the model on proprietary data. Azure AI Search, Cosmos DB with vector search, PostgreSQL/SQL vector extensions or third-party vector stores can serve this role.

Safety and Compliance Layer: Embedding Azure AI Content Safety for prompt filtering and response moderation, plus custom guardrails such as domain-specific validators.

User Interaction Layer: Applications can expose AI through chatbots, copilots in IDEs, web apps or APIs. Streamlit, Copilot Studio or custom-built apps in Azure App Service provide integration points.

Microsoft is also formalizing this orchestration pattern under what it calls AgentOps. Azure AI Foundry now includes an Agent Service, which provides a managed runtime for deploying multi-agent systems without needing to build custom orchestration logic from scratch. This enables enterprises to register tools, define agent roles and let Foundry handle communication, state and observability across agents, moving beyond ad-hoc orchestration into platform-supported “agentic applications.”

Example: An insurance claims assistant on Azure could combine GPT-4o (via Azure OpenAI Service) for dialogue and Phi-4 (hosted through Azure AI Foundry) for lightweight summarization. The application is orchestrated by AI Foundry’s SDK, grounding responses through Azure AI Search backed by a vector database such as Cosmos DB. Safety and compliance are enforced with Prompt Shields and Azure AI Content Safety, while AI Foundry’s observability dashboards and the Evaluations SDK provide continuous monitoring of quality and performance.

2.2 - Experimentation and Prompt Engineering:

Experimentation remains an iterative and crucial inner-loop process of developing and refining your AI solution. In GenAIOps this process often involves prompt engineering trials and other novel experiments beyond model training. Teams might experiment with different prompt templates, system instructions, example-based prompting or chain-of-thought techniques to see what yields the best outputs for their use case. There are many dimensions to explore: varying the phrasing or formatting of prompts, trying different personas or tones, inserting few-shot examples or employing advanced prompting strategies like chaining and tool augmentation. For RAG solutions, experiments could include testing different chunking strategies, embedding models or search retrieval methods to optimize how well the LLM uses external knowledge.

Throughout this phase, it’s critical to record what was tried and how the model responded. Using an experimentation framework (for example, AI Foundry’s Playground) helps manage this process. Prompt Flow, another important feature inside Foundry, is also designed to streamline LLM app development by letting developers prototype and evaluate prompt workflows systematically. It captures inputs/outputs for each run, enabling side-by-side comparisons of prompt variants and model configurations. Just as in traditional ML, maintaining reproducibility in experiments (versioning of prompt scripts, datasets and model parameters) is essential so results can be compared accurately. A well-structured experimentation approach in GenAIOps ensures that teams can scientifically identify which prompt or model changes in parameters actually improve performance before committing them to production.

2.3 - Evaluation and Validation:

This is by far the part of the GenAI Ops workflow where you should spend most of the time. It provides the evidence that a generative AI system is performing correctly, safely and as intended and it will only do it once you’ve gone through many thorough evaluations. Given the open-ended nature of generative outputs, evaluation must be systematic and multi-dimensional. It generally occurs at multiple points in the lifecycle, but it all start before deployment (pre-production testing):

Once a model or prompt design is ready, teams run extensive evaluations on it in a staging environment. This often involves using evaluation datasets of query-prompt pairs and expected responses to see how the system handles realistic scenarios. Unlike conventional ML, there may not be a single “ground truth” for generative responses, but test sets can be designed to probe specific qualities (e.g. correctness on known Q&A pairs, stylistic appropriateness, etc.). Automation is key using an evaluation toolkit or scripts to send a large number of test queries through the model and measure outcomes.

Microsoft’s Azure AI Evaluation SDK, for example, enables programmatic evaluation at scale using a suite of evaluators (metrics) for different aspects of quality. These include general NLP metrics like BLEU or ROUGE for text fidelity, groundedness scores for truthfulness in RAG and safety checks for toxic content. The evaluators essentially act as automated “graders” of the model’s outputs. By running these evaluations on many prompts, teams can identify weaknesses or failure modes (such as the model hallucinating facts or producing biased text) before any real users are exposed to them. This stage should also include testing for edge cases, using adversarial or unpredictable inputs to see if the model behaves robustly. Azure provides tools like adversarial simulators and even an AI red-teaming agent to generate such challenging test queries (e.g. prompt attacks, jailbreak attempts) in an automated fashion. By catching and addressing issues pre-deployment, GenAIOps reduces the risk of releasing a model that might produce harmful or low-quality outputs in production.

Similarly, LangChain’s evaluation module (often now part of the LangSmith toolkit) allows developers to compare LLM outputs using both automated metrics and LLM-as-a-judge techniques. With LangChain, one can set up an “evaluation chain” where GPT-4 (as an evaluator) is given a prompt, the outputs of two models, and asked to decide which is better or to score them against a rubric. This is increasingly common: using one strong model to grade the outputs of another, which accelerates evaluation when human labeling is too slow or costly. Of course, such evaluations should be taken with caution (AI graders can have biases), but they are a powerful tool for rapid iteration.

There are also open-source efforts like OpenAI Evals (a framework OpenAI released for evaluating models on custom tasks, which allows community-contributed evals) and academic benchmarks like HELM (Holistic Evaluation of Language Models) that offer suites of tests across domains. These can be integrated into your GenAIOps pipeline to continually benchmark your model against known baselines. If you fine-tune a model, for example, you might run a subset of HELM or other standard tasks to ensure you didn’t regress general capabilities.

3. Observe and Optimize

3.1. Continuous Integration and Delivery (CI/CD):

Continuous Integration: Whenever a data scientist or engineer makes a change (for example, adjusts a prompt template, fine-tunes a model on new data or updates the retrieval logic), the CI process should kick off automated tests and evaluations. This could involve running a suite of unit tests (does the prompt format function produce the expected JSON? Does the LLM chain handle edge cases without errors?) and evaluation suites as described earlier. For instance, you might include an automated evaluation run on a small set of queries as part of CI, to ensure a new prompt or model version doesn’t dramatically degrade response quality or violate any safety rules. Integrating LLM evals into CI is new territory, but tools like Azure’s Evaluation SDK or LangChain’s graders can be scripted to return a pass/fail signal (e.g., “quality score must remain above X”). This way, your CI pipeline can catch obvious issues early. Additionally, CI should verify infrastructure-as-code changes (perhaps you added a new vector index or changed a config for the model endpoint).

Continuous Delivery/Deployment: Once changes pass integration tests, they should be deployed in a consistent manner. Infrastructure-as-Code (IaC) is your friend here. Using Azure Resource Manager templates or Terraform or Bicep to define your Azure resources (from web apps to Azure OpenAI deployments to storage accounts) allows repeatable provisioning across environments. Azure Machine Learning pipelines and GitHub Actions/Azure Pipelines are commonly used to deploy models and apps. For example, an AML pipeline can automate the registration of a new model version, create a deployment (endpoint) for it, run a validation step and then update a production endpoint to point to the new model if all checks pass.

Similarly, if you containerize your application (e.g., a FastAPI serving an LLM or a Streamlit app), CI/CD can build the container and deploy it to Azure Container Apps or App Service. It’s important to incorporate progressive delivery techniques: rather than releasing a new model or prompt to all users at once, use canary deployments or blue-green deployments.

For instance, you might deploy “Model v2” alongside “Model v1” and route a small percentage of traffic to v2 initially, monitor its performance, then gradually increase if all looks good. Azure Machine Learning supports multi-deployment endpoints to facilitate this, and APIM (Azure API Management) can also be used to do weighted routing to different backends. Blue-green deployment similarly means you spin up a new instance of the service with the updated model (blue) while the old (green) is still running; you switch traffic over once blue is verified, but keep green around for quick rollback.

One interesting twist in GenAIOps is that sometimes you aren’t deploying a new model, but a new prompt or chain logic that significantly alters behavior. Treat these as first-class deliverables too. If you maintain your prompt templates or agent scripts in a repository, you can version them and have a CD process to update prompt versions in production (for example, updating the prompt that your web app passes to the LLM). Some teams even externalize prompts to configuration files or use feature flags to flip between prompt versions, enabling rapid iteration without redeploying code. Ensure that when prompts or tool configurations change, you still run through tests/evaluations in CI, a poorly worded prompt tweak could break functionality just as a code bug could.

3.2. Deployment and Release Management:

Deploying generative AI solutions reliably requires attention to a broader set of artifacts compared to standard ML deployments. In a typical scenario, you will deploy not just a model but also accompanying services and configurations:

If you fine-tuned a custom model or built a Small Language Model (SLM), that model needs to be registered and deployed (e.g. as an endpoint or container) much like any ML mode. GenAIOps would reuse existing CI/CD pipelines to automate this, ensuring that models pass integration tests and meet performance criteria prior to release.

For many generative apps, the orchestrator or application logic (responsible for composing prompts, calling external tools or databases, etc.) is central to the solution. This might be implemented with frameworks like Semantic Kernel or Azure AI Foundry and needs to be deployed onto a runtime (such as an Azure Function, web service, or container). Operations must treat this orchestrator code as a deployable component, versioned and tested just like model code. Any changes to how prompts are constructed or how the workflow runs should go through code reviews and automated tests. Microsoft even provides an Agent Service in Azure AI Foundry to simplify deploying AI agents (conversational or task-oriented bots) with low-code configurations.

Any data store changes required by the AI (like new vector index builds or schema changes in a knowledge base) should be included in the deployment process. For example, if a new set of documents is added to the RAG database or the vector index needs rebuilding, those tasks should be automated and validated in a staging environment before production cutover.

Gateway and API management: An emerging best practice is to put a gateway (such as Azure API Management) in front of LLM endpoints. This allows a layer of control: routing traffic to different model versions, applying authentication, logging and even basic rate limiting or filtering on prompts. During deployment, updating the gateway routing (for instance, switching from an old model endpoint to a new one) can enable gradual rollouts (shadow testing, canary releases) where a small percentage of traffic is served by the new model while monitoring its performance. This approach minimizes risk by enabling quick rollback if the new model behaves unexpectedly. It also centralizes monitoring of requests and responses, since the gateway can log all inputs/outputs for analysis.

Testing and release strategy: GenAIOps emphasizes thorough testing of all components before full release. Beyond typical unit and integration tests for software, there should be end-to-end tests simulating user interactions with the AI (for example, conversation transcripts run through the system to see if everything from the UI, orchestrator, to the model works properly). Additionally, techniques like A/B testing or blue-green deployments are advisable when introducing changes, to continuously measure if the new deployment maintains or improves quality. Given the stochastic nature of generative models, even small changes can sometimes produce unexpected outputs, so a cautious rollout with monitoring is prudent.

Finally, deploying generative AI often involves a cross-functional effort. It’s not just the model that needs monitoring, but the entire user experience (front-end application, data pipelines, etc.). Many organizations find that DevOps and MLOps roles converge here. DevOps engineers may work alongside ML and AI engineers to deploy and monitor the orchestrator, the UI and other non-ML components of the solution. This ensures that the operational excellence covers all moving parts of the intelligent application, not just the model itself.

3. 3. Monitoring in Production:

After deploying a generative AI model or system, the work is far from done. Continuous monitoring is critical to detect issues early and maintain performance. Monitoring for GenAI happens on multiple levels:

Operational monitoring: Is the service up? Are responses being delivered within expected latency? What’s the throughput? Tools like Azure Monitor and Application Insights can collect logs and metrics from your application (e.g., HTTP 500 errors, response times) and from the model endpoint (e.g., Azure OpenAI provides metrics like request count, token usage and latency percentiles). You should set up alerts on key operational metrics. For example, if median latency spikes above 2 seconds or the error rate goes above 1%, an alert could notify the on-call engineer to investigate.

AI performance monitoring: Track metrics that reflect the quality of the AI’s outputs in real-world usage. For predictive models we’d watch accuracy or drift; for generative models, we watch things like groundedness, relevance, completeness, and user satisfaction. Many organizations implement feedback loops – e.g., collecting user ratings (thumbs-up/down, or qualitative feedback) on responses. Such human feedback can be aggregated to flag if a model’s helpfulness or correctness is trending downwards over time.

Quality and safety monitoring: In production, you need to watch metrics like groundedness, relevance, completeness, etc., as well as safety triggers. Azure AI Foundry’s observability dashboard, for instance, can show in real-time what percentage of responses are being flagged by safety filters or what the average “groundedness score” is for the last N interactions. A sudden drop in quality metrics (say, relevancy score falls after a data source update) or a spike in safety issues (say users found a new way to provoke unwanted content) should feed back into your Dev cycle quickly. One best practice is to periodically route a sample of production traffic to an evaluation pipeline (shadow mode testing), essentially continuously validating the model on real queries.

Resource and cost monitoring: Generative models, especially large ones, can be resource-intensive. In production one must keep an eye on cost metrics like token consumption (for API-based models) and throughput quotas. GenAIOps involves optimizing these, for example, ensuring prompt inputs aren’t unnecessarily long (which wastes tokens), using caching or a smaller model for simple requests, etc., to control costs. If using a multi-model setup, monitoring can inform whether to route certain queries to a cheaper model vs a high-end model dynamically to save cost while meeting SLAs. Azure OpenAI, for instance, provides quota management and usage logs that should be incorporated into monitoring dashboards.

Feedback Loops, Drift and Retraining Triggers:

In MLOps, a common paradigm is to retrain a model when the input data distribution shifts or when model accuracy on new data degrades (model drift). In GenAIOps, if you are fine-tuning or continually improving a model, you should monitor drift in user queries and content of responses. For instance, maybe users start asking about a new product your company just launched, the model might not know about it if its knowledge cutoff is old. This would show up as decreased answer quality for that topic.

Monitoring might involve tracking the topics or clusters of queries over time. When a significant shift is detected (e.g., a lot of queries on a new domain), it could trigger a model update workflow: perhaps gathering new training examples and fine-tuning the model or updating the retrieval index with fresh data. Additionally, feedback loops can provide training signals: if users consistently mark certain answers as unhelpful, those should be collected as examples to improve on.

A robust GenAIOps setup will include a mechanism to periodically retrain or fine-tune models (or update prompts) using accumulated data from production. This could be on a schedule (retrain every month with latest data) or event-driven (retrain when a concept drift threshold is exceeded). Azure Machine Learning and Foundry support scheduling pipeline jobs, so you could automate, for example: run a pipeline that fine-tunes your model on the last 10k Q&A pairs from production (that had a thumbs-down) and then run evaluation, then deploy if metrics improved. Even if you rely on a 3rd-party model like GPT-4 where you can’t retrain it, you might use this trigger to adjust your prompt strategy or add more grounding data to address the drift.

3.4. Automation of Model Lifecycle Management:

The complexity of GenAI systems means there are many steps where manual processes could slow things down or introduce errors, from data collection, to training, to deployment and evaluation. Automating these steps is key to achieving reliable and repeatable operations.

Data pipelines (DataOps): Automate the collection and preparation of data for both model training and grounding knowledge. For example, if you have a RAG setup, you might automate a pipeline that ingests new documents, chunks and embeds them, and updates the vector index regularly. Azure Data Factory or Azure Synapse pipelines can be used to orchestrate such data flows on a schedule. Similarly, if you’re collecting user interactions and feedback, have automated jobs that aggregate this data into training/evaluation datasets.

Experiment tracking and reproducibility: Use tools like MLflow or Azure ML to automatically log experiment runs, including which model, which data, which parameters, so that any result can be reproduced and audited. As recommended by Microsoft, MLflow is a good choice since it supports tracking code, data, parameters and can even store artifacts for LLM experiments.

CI/CD pipeline automation: Ensure that your entire deployment process is codified. This includes automated environment setup (infrastructure as code), automated model packaging (e.g., exporting a fine-tuned model or containerizing an app) and one-click or zero-click deployments to test/prod. Platforms like Azure DevOps or GitHub Actions can host these pipelines.

Monitoring and alerting automation: Set up automated triggers for common remediation. For instance, if a monitoring alert catches that the content filter is blocking many outputs (perhaps indicating the model is producing edgy content), you might automatically roll back to a safer model or activate a fallback response template. Another example: if throughput consistently hits a threshold, an automation could scale out the number of instances or increase the provisioned throughput of an Azure OpenAI resource.

3.5. Tools for Tracking, Versioning, and Reproducibility:

Maintaining a clear record of your models, data, and prompts is non-negotiable in GenAIOps. Model versioning tools help you keep track of which model is in use. Azure Machine Learning’s registry or Azure AI Foundry’s project space can store model versions with metadata (who uploaded it, what data was it trained on, evaluation results). Similarly, track prompt versions, it could be as simple as a markdown file in Git that contains the current system prompt with a version history.

For agents with multiple tools and steps, consider writing configuration files (perhaps in JSON or YAML) that specify the chain-of-thought logic or tool selection parameters; manage those in source control too. Reproducibility extends to data: version your datasets or at least have snapshots of the data used for each fine-tuning run and evaluation round. This could mean storing a copy of the training data in a versioned blob storage path or using dataset versioning features of Azure ML.

Why? If six months later a model misbehaves and you suspect the fine-tuning data had an issue, you can retrieve exactly what the model saw during training.

The ultimate aim of continuous monitoring is to enable a learning loop from production. By observing real-world usage, the team can identify areas to improve, maybe the model is often unable to handle a certain class of user query, indicating a gap in training data or an opportunity to fine-tune on new examples. Or perhaps users are asking for information that isn’t in the current knowledge base, suggesting it’s time to expand the RAG dataset.

GenAIOps facilitates capturing these insights and feeding them back into the next iteration of model or prompt development (this could be seen as the “outer loop” feedback cycle). In some cases, automated retraining or prompt adjustment might be triggered by monitoring signals, though often a human-in-the-loop will triage issues first. By continuously learning from production data, the generative AI system can be incrementally improved, maintaining high performance and relevance over time.

4. Protect and Govern

Once you’ve achieved performance and scale, the final challenge is ensuring safety, compliance and enterprise trustworthiness. Protect + Govern is the pillar that ensures the system not only performs well but also meets enterprise standards for security, safety and regulatory alignment. In practice, this means embedding protections at every layer of the stack and building governance hooks that make your system explainable and controllable by design.

4.1. Safety and Guardrails by Design

Generative AI is inherently open-ended. Left unguarded, even the strongest models will hallucinate, leak sensitive information or produce harmful content. For this reason, safety must be treated as a first-class component of the architecture, not an afterthought. Azure AI Foundry approaches this with a layered defense model:

Prompt Filtering and Input Sanitization

Before a query even reaches the model, Azure AI Prompt Shields are designed to detect adversarial inputs such as:

Jailbreak attempts (“ignore all previous instructions and …”)

Prompt injections (maliciously embedded instructions hidden in user content or documents)

Cross-domain misuse (e.g., asking a healthcare bot about weapon creation).

In Foundry, Prompt Shields can be enabled at the prompt flow node level or at the endpoint, meaning every request is scanned against Microsoft’s continuously updated library of attack patterns. This reduces the likelihood of malicious inputs steering the model into unsafe territory.

Response Moderation

Even with strong inputs, outputs must be checked. Azure AI Content Safety acts as a post-processing layer, scanning text and images generated by the model for:

Harm categories such as hate speech, violence, self-harm, sexual content.

Sensitive data leakage, including personally identifiable information (PII).

Bias detection in generated text.

In AI Foundry, developers can chain Content Safety into Prompt Flow pipelines, so that every model response is scored for severity (safe, low, medium, high risk). Based on thresholds, teams can block, redact or route responses to human review. For example, a “medium-risk” output in a healthcare assistant might automatically be rewritten with a disclaimer, while a “high-risk” output is suppressed.

Policy Enforcement Layers

AI Foundry is designed to let enterprises enforce domain-specific rules. Inside Prompt Flow, custom evaluation nodes can embed additional policies:

A financial services chatbot can enforce that every output ends with a compliance disclaimer.

A medical assistant can automatically add a notice that advice is not a substitute for a physician’s consultation.

A legal research tool can ensure all responses cite source documents with hyperlinks.

These rules are defined as reusable policies in the pipeline, effectively treating compliance as code.

Microsoft also aligns this with Responsible AI Standard requirements, so teams can demonstrate that policies are not just business preferences but part of compliance-by-design.

Fallback Strategies

No safety system is perfect, so Azure promotes resilience through fallback strategies:

Multiple deployments under one endpoint (Azure OpenAI or Azure ML endpoints) allow developers to configure primary and secondary models. If the primary model fails a safety check, the system can automatically route the query to a safer, smaller model (e.g., Phi-4-mini).

Template-based responses can serve as a controlled fallback: e.g., “I’m unable to provide that information, but here’s a resource you can consult.”

Deterministic rules in Prompt Flow can override model behavior. If the model tries to answer restricted topics, the pipeline can short-circuit and serve a safe, pre-approved response instead.

This fallback mechanism is not a patchwork, it is natively integrated into AI Foundry. Developers can configure guardrails as part of their deployment profile, ensuring consistent enforcement across dev, staging, and prod environments.

Why This Matters

Microsoft’s framing is clear: safety guardrails must be proactive and systemic, not reactive. With AI Foundry, this translates into:

In-line safety filters (Prompt Shields + Content Safety)

Programmable policy nodes (Prompt Flow custom evaluators)

Resilient fallback logic (multi-deployment routing, template responses)

Together, these features create a safety fabric that envelops the entire GenAI system, ensuring that unsafe or non-compliant outputs are caught before they reach the end user.

4.2. Security and Access Control

Protecting a GenAI application also requires enterprise-grade security across the entire pipeline, from data ingestion and prompt processing to deployment and monitoring. Azure AI Foundry and the broader Microsoft ecosystem bring security principles directly into the operational fabric:

Identity and Access Management (IAM)

Azure Entra ID provides a unified identity plane across all AI Foundry projects and Azure resources. Role-based access control (RBAC) ensures that only authorized team members can perform critical actions:

Data scientists may run evaluations but cannot push to production.

Engineers may deploy models but cannot view sensitive training data.

Auditors may access logs but not modify deployments.

This fine-grained access control can be extended with Conditional Access policies (e.g., require MFA for model deployment) and Privileged Identity Management (PIM) to enforce just-in-time access for high-risk actions.

Data Protection

Data used in prompts or grounding sources often contains sensitive or regulated information. Azure’s approach is multilayered:

Encryption by default: All services (Storage, Cosmos DB, AI Search, PostgreSQL with vectors) enforce encryption at rest and in transit.

Confidential data handling: Sensitive inputs (e.g., PII from customer records) can be automatically masked or tokenized before being injected into prompts using Azure Data Factory or Purview Data Loss Prevention (DLP) policies.

Confidential computing: For highly sensitive use cases (financial transactions, healthcare), models and data can run inside confidential enclaves (Intel SGX) with Azure Confidential Computing, ensuring even the cloud provider cannot access raw data.

API and Endpoint Security

Since GenAI systems are exposed via APIs, securing endpoints is critical. Wrapping model endpoints with Azure API Management (APIM), as explained in section 3.2 provides:

Authentication & Authorization: Integration with Entra ID or OAuth2 flows.

Rate limiting & throttling: Preventing abuse (e.g., malicious actors flooding inference endpoints).

Payload inspection: Detecting anomalies in inputs (prompt injection attempts, oversized requests).

Centralized logging: Every request and response can be routed through APIM for downstream analysis.

Combined with Private Endpoints and Virtual Networks (VNets), organizations can keep model traffic inside a secure perimeter, away from the public internet.

Audit Trails and Compliance Evidence

In regulated industries, proving compliance is as important as enforcing it. Azure Monitor and Log Analytics integrate with AI Foundry to provide immutable logs for every critical event:

Who deployed which model, and when.

Which evaluation tests were run, and with what results.

Which users accessed vector stores or prompt configurations.

These logs can be linked to Microsoft Purview for data lineage tracking, or exported to Sentinel SIEM for continuous threat monitoring. For industries like finance or healthcare, this ensures that AI operations meet standards such as GDPR, HIPAA, and SOC 2.

Microsoft’s Framing

What makes Microsoft’s approach distinct is that security and governance are not external add-ons — they are deeply integrated into the same workflows teams use to build and deploy models. An engineer working in AI Foundry does not have to “bolt on” logging, encryption, or RBAC; these are inherited automatically from the Azure resource model.

4.3. Governance and Accountability

While governance is covered more deeply in Chapter 8, GenAIOps requires building practical governance hooks directly into operational workflows:

Versioning for auditability: AI Foundry projects and Azure ML Model Registry store models, datasets, prompts and evaluation results with version tags. This provides an immutable record for audit and rollback.

Evaluation as governance: The AI Foundry Evaluation SDK can be scripted to run bias tests, safety probes, and red-team scenarios before every deployment. These results become part of the compliance package.

Policy-as-code: Azure Policy can enforce enterprise standards across AI resources (e.g., “All models must log prompts and responses” or “Only approved models from the Model Catalog can be deployed to production”).

Explainability and traceability: Foundry’s Observability dashboards allow teams to trace a response back to the specific prompt, grounding data, and model version used, a critical capability when auditors or customers demand explanations.

4.4. Linking Protect + Govern with the GenAIOps Lifecycle

Though it might sound counterintuitive, the Protect + Govern phase doesn’t sit at the end of the genAI workflow: it cuts horizontally across every stage of GenAIOps. Actually, Microsoft frames governance not as an afterthought, but as a through-line embedded into AI Foundry and Azure services.

During Build + Customize: Governance begins at the earliest stages of app design. AI Foundry integrates safety filters (Prompt Shields, Content Safety) and evaluation nodes into Prompt Flow, ensuring that prompt workflows are tested for compliance before they reach production. Teams can define custom compliance validators (e.g., enforce medical disclaimers, mask PII before prompts) as reusable components. In regulated industries, these early safeguards allow developers to test compliance as code, rather than relying on manual reviews later.

During Observe + Optimize: Observability isn’t just about performance dashboards, it doubles as a governance lens. Foundry’s monitoring surfaces metrics like groundedness, bias frequency, and safety violations alongside latency and throughput. Cost monitoring, exposed in Azure Monitor and integrated with Entra ID billing scopes, provides traceability for resource usage, ensuring transparency when auditors ask “who ran what, and at what cost?” Logs can be automatically exported to Purview for lineage tracking or Sentinel SIEM for security correlation. In practice, this means that performance monitoring is also compliance monitoring.

During Deployment and Release: Governance is enforced at the handoff points. Using Azure DevOps or GitHub Actions, teams can embed approval workflows (e.g., require a manager or compliance officer to sign off before a new model/prompt goes live). Policy as Code can be applied via Azure Policy to prevent non-compliant deployments (e.g., block a model endpoint that isn’t wrapped with API Management, or enforce encryption on all vector stores). With APIM, access keys and traffic policies are centrally managed, ensuring deployment aligns with enterprise standards.

During Updates and Continuous Improvement: Governance ensures that retraining, fine-tuning or even prompt changes don’t bypass oversight. Every model version, prompt revision and evaluation report is stored in registries and logs, enabling full auditability. When an update fails a safety threshold (e.g., groundedness below 85%), automated pipelines can halt deployment until a human reviews. This builds accountability directly into the CI/CD fabric.

By embedding these controls into AI Foundry and Azure services, enterprises move from best-effort safety to systematic accountability. The result is a workflow where safety, security, and governance are not bolted on at the end, but woven into the lifecycle, from first prototype to production monitoring.

Summary:

Protect + Govern ensures that GenAI applications are safe, secure, and aligned with organizational and regulatory requirements. Azure technologies, from AI Foundry (Content Safety, Prompt Shields, Observability, Evaluation SDK) to the broader Azure ecosystem (Entra ID, APIM, Azure Policy, ML Registry, Monitor), provide the practical tools to make this possible. In Chapter 8, we will expand this discussion to organization-wide AI governance frameworks, policies, and maturity models.

Chapter Conclusion

GenAIOps/LLMOps is still a nascent but rapidly evolving field. As we have seen, it extends the foundations of MLOps to meet the demands of generative AI, requiring new techniques in prompt management, rigorous evaluation for quality and safety and continuous model oversight in production. Industry leaders including Microsoft, AWS, and others are actively shaping best practices and toolsets for this domain, underlining that operational excellence is essential for AI success in the era of foundation models. Adopting GenAIOps means instilling a culture of ongoing validation and improvement around generative AI systems: evaluating thoroughly at each stage, deploying carefully with safeguards, and monitoring relentlessly once live. The reward is the ability to deliver cutting-edge AI capabilities, like natural language interfaces, creative content generation, and advanced decision support, with a high degree of confidence and reliability.

As you implement GenAIOps in your own projects, remember that it’s an iterative journey. Start by leveraging your existing MLOps strengths, then extend step by step, introduce prompt evaluations, add safety checks, gradually automate your pipelines for LLMs. Over time, your organization will mature in this discipline, much like climbing a GenAIOps maturity model from initial experimentation to optimized, continuous delivery. With platforms such as Azure AI Foundry and the growing ecosystem of LLMOps tools, the path is becoming clearer to manage generative AI systems not only with creativity, but with the same rigor and reliability that we expect from any production software. By doing so, we ensure that generative AI fulfills its promise in real-world applications – delivering innovation and value responsibly and consistently at scale.

Looking ahead, Azure AI Foundry is moving from a toolkit into a full GenAIOps platform. The direction is clear: unify design, evaluation, deployment, and monitoring into a single workflow, while lowering the barrier to entry with low-code features like Copilot Studio and Prompt Flow. We can also expect richer support for multi-modal pipelines and managed orchestration of multi-agent systems (“AgentOps”), making it easier for enterprises to operationalize complex applications with built-in observability, safety, and cost optimization.

At the same time, Microsoft is pushing governance from a periodic activity into a continuous one. Real-time red teaming, continuous compliance dashboards and deeper integration with Azure Purview, Policy, and Sentinel will make safety and auditability a default property of any GenAI solution. In short, AI Foundry and the Azure ecosystem are evolving toward platform-level accountability: enabling enterprises to build not just powerful generative AI applications, but ones that are safe, explainable, and governed by design.

End of chapter test

What is the primary distinction between MLOps and GenAIOps?

MLOps focuses on deploying and monitoring rule-based systems, while GenAIOps manages cloud infrastructure.

MLOps emphasizes training and lifecycle management of ML models, while GenAIOps extends those practices to include prompts, evaluations, orchestration, and safety for generative AI.

MLOps is only relevant for small companies, while GenAIOps is designed for enterprises.

MLOps was replaced entirely by GenAIOps and is no longer used.

Answer:

Incorrect → Rule-based systems are not the focus of MLOps.

Correct → GenAIOps builds on MLOps, adding new dimensions such as prompt management, content safety, orchestration, and evaluation for generative AI workloads.

Incorrect → Both are relevant across company sizes.

Incorrect → MLOps remains valid for classical ML; GenAIOps extends rather than replaces it.

Which Azure service provides automated evaluation workflows to test generative AI models for quality, safety, and groundedness?

Azure Monitor

Azure AI Foundry Evaluations SDK

Azure DevOps Pipelines

Azure Key Vault

Answer:

Incorrect → Azure Monitor tracks logs and metrics but doesn’t evaluate model quality.

Correct → The Evaluations SDK in Azure AI Foundry provides structured workflows for automated model evaluation.

Incorrect → DevOps pipelines manage CI/CD, not evaluations.

Incorrect → Key Vault secures secrets but isn’t an evaluation tool.

In the context of GenAIOps, what is shadow mode testing?

Deploying a model only for internal users before public release.

Running evaluations on production traffic samples without exposing outputs to end users.

Creating synthetic training data for fine-tuning a foundation model.

Comparing multiple base models in the AI Foundry Model Catalog.

Answer:

Incorrect → Internal previews are staging, not shadow mode.

Correct → Shadow mode means routing live queries through the system to evaluate performance and safety, without exposing outputs to users.

Incorrect → That describes data augmentation.

Incorrect → Model Catalog comparisons happen earlier in the lifecycle, not in production monitoring.

How does Azure API Management (APIM) contribute to governance in GenAIOps?

It provides pre-trained LLMs for enterprise deployments.

It enables authentication, throttling, logging, and version control for model endpoints.

It monitors GPU utilization across Azure ML clusters.

It generates test data for evaluating generative AI applications.

Answer:

Incorrect → APIM is not a model provider.

Correct → APIM sits in front of endpoints to enforce access control, traffic management, and auditability.

Incorrect → GPU monitoring is handled by Azure Monitor, not APIM.

Incorrect → APIM doesn’t generate test data.

What is the purpose of Prompt Shields in Azure AI Foundry?

To reduce inference costs by shortening prompt length.

To detect and block malicious or adversarial inputs such as prompt injections.

To automatically rewrite user prompts for stylistic consistency.

To fine-tune foundation models with domain-specific examples.

Answer:

Incorrect → Prompt Shields don’t optimize cost.

Correct → Prompt Shields identify injection/jailbreak attempts and sanitize inputs before they reach the model.

Incorrect → They don’t rewrite prompts for style.

Incorrect → Fine-tuning is handled separately in AI Foundry or Azure ML.

Chapter 8. Generative AI Governance Framework

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the authors’ raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the eighth chapter of the final book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at shunter@oreilly.com.

You have completed several book chapters, including those explaining advanced AI techniques, the entire Microsoft Azure ecosystem for generative AI, and real-life operations for performance management. You already know the “main pieces” of this technology puzzle, but now we will complete it with real-life governance considerations.

This fifth chapter is an applied generative AI governance framework that will include a combination of technology, organizational, and human considerations, all related to specific Microsoft services. The goal is to go between that full picture and the most tangible actions for you to complement your advanced, robust generative AI applications, with solid foundations for ethical AI implementations.

Because of the new and highly evolving nature of this topic, let’s first start with your mental framework to organize the generative AI governance exercise, and then proceed with the technical aspects.

Thought Process for Your AI Governance

In general terms, an AI governance program includes a combination of technology, processes, and people aspects, as you can see in Figure 8-1. The technology piece is there to help you enable processes, policies, and collaboration between people within the organization. Microsoft Azure contains some of them, but there are other adjacent Microsoft services such as Defender or Purview among others that converge with the Azure stack for AI, and complete the end-to-end architecture.

Figure 8-1. People, processes, and technology triangle for AI governance programs

An AI governance program is a systematic oversight and management of AI systems throughout their lifecycle, including monitoring, auditing, and policy enforcement among others. The program also ensures that the AI systems operate in alignment with top-down organizational values, legal obligations, ethical principles, and reputational expectations. It helps control and minimize risks, promote trust, and guide responsible innovation.

Levels of AI Governance

There are different ways to implement AI governance, from the initial intent to specific and tangible actions. As you can see, Figure 8-2 shows some common terms and assets related to AI governance that you need to know:

Figure 8-2. Levels of AI Trustworthiness

Ethical AI → Focuses on embedding moral principles such as fairness, transparency, privacy, and human rights into the design and behavior of AI systems, ensuring that these technologies respect societal values and do not cause harm or reinforce injustice. International organizations, cities, and companies, create their own set of principles to define the way they imagine the implementation of an AI that is ethical and fair. You can see a collection of principle sets from the Principled AI initiative of Harvard University’s Berkman Klein Center, and get inspired to create your company’s own set of principles, based on your values and notion of ethics.

Responsible AI → Emphasizes the intentional and proactive governance of AI throughout its lifecycle, including design, deployment, and monitoring, to ensure systems are safe, trustworthy, inclusive, and aligned with human well-being, while minimizing unintended consequences. It is a second level of action that focuses on implementing the best practices and specific technical implementations that you will see in the rest of this chapter. You can consider an incremental iteration over the previous ethical AI approach.

Accountable AI → The third level is based on the idea of creating clear ownership, traceability, and oversight mechanisms for AI systems, so that individuals or institutions can be held answerable for how AI is developed, used, and impacts people. The accountable approach focuses on ensuring legal, ethical, and operational responsibility, and relies on the idea of being compliant with AI and other regulations.

Trustworthy AI → Trustworthy AI is an umbrella term that refers to AI systems that are designed, developed, and deployed in a way that earns and maintains public trust by being reliable, transparent, fair, secure, and aligned with human values. It is often seen as the overarching goal that ethical, responsible, and accountable AI help achieve. These three together enable the creation of trustworthy AI.

Typical AI Governance Assets

In order to achieve these levels of AI trustworthiness, there are different “assets” that can serve as inspiration and guidance for the tangible AI governance actions. As you can see in Figure 8-3, there are various blocks that can be mandatory (e.g. regulations), plus others that are implemented as a voluntary best practice:

Figure 8-3. AI Governance Assets

External context

These assets are beyond the internal company scope. Usually related to governments and international organizations, they set a standard of best practices to guide your AI implementations. Concretely:

AI regulations → Regulations are external and legally binding rules set by governments or international bodies that define permissible AI uses, risk classifications, data protection requirements, transparency obligations, and penalties for noncompliance. Some of the main AI regulations are the EU AI Act, or the Canadian AI and Data Act. AI regulations converge with other regulations such as General Data Protection Regulation (GDPR) or California Consumer Privacy Act (CCPA) for data protection, and any other vertical industry regulations (e.g. Health Insurance Portability and Accountability Act of 1996 or HIPAA). Companies must comply with these regulations to avoid costly fines, and they operate under the idea of shared responsibility between the AI adopter (e.g. your company) and the technology provider (e.g. Microsoft Azure) that we will explore later in this chapter. And of course, these regulatory requirements feed into the technical design process with considerations in terms of model choice, data anonymization and protection, etc.

International standards → These standards are technical and organizational specifications (e.g., on specific topics such as safety, interoperability, robustness, risk management, data quality, management systems) developed by international or industry bodies that offer consistent, measurable benchmarks for trustworthy AI design and deployment. Some of the most famous standards come from organizations such as the International Organization for Standardization (ISO), the European Committee for Standardization and the European Committee for Electrotechnical Standardization (CEN-CENELEC), and the National Institute of Standards and Technology (NIST) in the United States. Some of the main standards include the NIST AI Framework, and the ISO 42001 for AI Management Systems (AIMS), but as you can see in Figure 8-4, there are other standards that you may need to revisit in the future for your AI auditing activities:

Figure 8-4. Relevant ISO Standards for Artificial Intelligence

These standards cover relevant topics such as AI risk management, security, AI governance, AI lifecycles, data quality, and data privacy among others. You can see the Microsoft Azure compliance offering website to get a continuously updated list of both standards and regulations for which the Azure platform will help you with.

Risk assessment frameworks and impact assessments → They are structured tools that help companies evaluate the potential for harm, bias, misuse, or unintended consequences of AI systems. This is often an internal or external requirement before AI deployment, and it directly impacts the selection of architectural patterns (e.g., human-in-the-loop, adversarial robustness patterns), AI safety choices, monitoring requirements, etc. You can explore Microsoft’s very own Responsible AI Impact Assessment with a template that can help you pre-evaluate the risks of your AI projects, and the ISO 42005 specification for AI impact assessments.

Ethical guidelines and principles → These rules are industry-wide or geography-related frameworks that articulate core AI values, guiding decision-making where formal law may lag behind. That was the main reference before the first AI regulations arrived, and you can check Microsoft’s AI principles (among others), as an example of what technology providers and companies in general do to define their own responsible AI approach. Concretely, the Microsoft approach includes the following principles:

Fairness → AI systems should treat all people fairly. This includes avoiding bias and discrimination based on race, gender, age, or other characteristics.

Reliability & Safety → AI systems should perform reliably and safely, even under unexpected conditions. They should be tested and monitored to ensure consistent performance.

Privacy & Security → AI systems must be designed to protect personal data and privacy and to ensure data security throughout their lifecycle.

Inclusiveness → AI should empower everyone and engage people from diverse backgrounds in its development and deployment.

Transparency → AI systems should be understandable. This includes clarity about how they work, what data they use, and how decisions are made.

Accountability → Developers and organizations should be accountable for their AI systems and their impacts. This means establishing governance and oversight processes.

Internal context (AI system level)

This second set of governance AI assets focus on the internal context of your organization, mainly related to AI systems, tools, models, and any solution you build with the Microsoft Azure suite.

Technical monitoring and Human-in-the-Loop (HITL) patterns → As you have seen in Chapter 4, the monitoring tools are integrated systems that manage the full AI lifecycle. Depending on the AI performance, there are oversight mechanisms ensuring that humans can intervene or make final decisions in critical or high-risk applications of AI, preserving agency and reducing potential automation harm. This is critical if you are using AI for critical applications in regulated industries such as healthcare, legal, and manufacturing, especially in cases where physical safety and human rights could be impacted. Resources like Azure Logic Apps or Azure Durable Functions can facilitate human intervention processes based on specific workflows and events.

Model and data documentation → Tools like model cards, data sheets, and system fact sheets that improve transparency and traceability by recording a model’s design, intent, limitations, and context. As you have seen in Azure AI Foundry, you can get access to the specific cards for each generative AI model. The level of detail depends on how open models are, and how specific the data documentation is.

AI red teaming and ethical hacking → In the intersection of AI and cybersecurity, these are internal or external exercises that intentionally probe AI systems for vulnerabilities, misuse scenarios, or ethical blind spots, strengthening system robustness pre-deployment. Automated red teaming (e.g., via CI/CD pipelines) becomes a critical component of the secure AI development lifecycle (SAIL).

Audit and explainability tools → These are frameworks and algorithms that interpret model behavior (e.g., SHAP or SHapley Additive exPlanations, LIME or Local Interpretable Model-agnostic Explanations), enabling internal and external stakeholders to understand, debug, and justify AI decisions.

Internal context (organizational level)

The third group of assets focus on internal, organizational considerations that support the previous model and system level actions. They focus on general processes that enable governance at scale, with both employees and users.

Training for AI literacy, and organizational awareness → Ongoing education programs across technical and non-technical teams to build shared understanding of responsible AI principles, regulations, and implementation best practices.

Transparency and communication channels → Interfaces and protocols to inform users and stakeholders how AI is being used, including clear disclosures, explanations, confidence scores, and opt-out or appeal options.

Internal AI policies and organizational structure → Documented internal rules and processes defining accountability, decision rights, approval workflows, and escalation paths for AI usage across teams and departments.

Incident reporting & pre/post-mortem → Clear escalation and documentation protocols for identifying, logging, and learning from ethical breaches, model failures, or user complaints in live AI systems.

Other Considerations

Now that you are planning your generative AI governance system, it’s critical to account for a set of additional considerations that ensure your implementation aligns with both Microsoft Azure’s ecosystem and your internal responsible AI principles. These considerations touch on general terminology, operational models, and architectural decisions that influence long-term success and compliance:

Difference between AI safety and AI security → These two terms are often used interchangeably, but they refer to two convergent yet different areas of concern. While both aim to reduce risks and ensure AI systems operate responsibly, they address very different types of threats and require different strategies and tools. This confusion is common in both technical and non-technical circles, especially as organizations accelerate to adopt generative AI without fully understanding the risks involved. Clarifying the distinction between safety and security is essential for designing robust governance frameworks and deploying AI systems that are both effective and trustworthy. For you to know both terms and the relation with the Microsoft technology stack:

AI safety → It is primarily concerned with ensuring that AI systems behave as intended, within acceptable boundaries, and without causing harm, even unintentionally. It deals with issues such as fairness, bias, hallucinations, toxic output, misinformation, and value misalignment. For generative AI, safety includes preventing the model from producing harmful, offensive, or misleading content and ensuring that it respects user intent without enabling abuse. Some of the related Microsoft and Azure services for AI safety include Azure AI Content Safety, the Responsible AI toolbox, and other AI Foundry features such as tracing and LLMOps.

AI Security → The second term focuses on protecting AI systems from malicious attacks or misuse. It includes safeguarding data, models, endpoints, and infrastructure from threats such as prompt injection, model theft, adversarial inputs, unauthorized access, and data poisoning. Security ensures that the AI system is resilient against tampering and external exploitation. More related to traditional cybersecurity topics, Microsoft and Azure services for AI security include several resources such as Defender or Entra, that we will explore later in this chapter.

Shared responsibility model → When deploying AI solutions on Azure, it’s essential to understand the shared responsibility model that applies to Microsoft Azure (this is the case for any other platform and provider). As you can see in Figure 8-5, that shared responsibility model between your company and the external providers is transversal to different tools, products, systems, and use cases.

Figure 8-5. Shared responsibility model across AI systems and use cases

In this case, Microsoft Azure provides security and compliance for the underlying cloud infrastructure, including physical data centers, networking, and foundational services. However, AI adopters like you are responsible for securing your data, managing access controls, overseeing model usage, and ensuring AI is applied ethically within their domain. In the context of generative AI, this means governing how prompts, responses, and downstream usage are handled, ensuring privacy, fairness, and content filtering are effectively managed on your end. That said, the point of using managed cloud services is to do so in a way that doesn’t require continuous manual effort, and providers like Microsoft also include transparency notes to explain the potential risks, limitations, and misuse of AI-related models and systems.

AI protection levels → Different use cases and data sensitivities call for distinct levels of AI protection. And that protection can come directly from the model itself and the data sources used for training or fine-tuning, protecting the model outputs from leakage or misuse. However, not all models have the same baseline of protection and security. Whether you’re working with pretrained, fine-tuned, or custom-built models, you must evaluate risks such as data re-identification, prompt injection, or unauthorized access. Microsoft Azure offers tools like built-in content filters, private endpoint access, and encryption capabilities that can support varying protection levels, but responsibility for proper configuration also lies with your organization. Concretely, Figure 8-6 illustrates the main levels you need to keep in mind while configuring your deployments:

Figure 8-6. Levels of AI model and data protection

Model-level protection → Precisely because every AI model comes from different providers with very diverse technical levels and focus on model protection, Azure AI Foundry includes a layer of protection from the company HiddenLayer, an AI security company that provides a model scanner to assess the code of the models. This first layer sets a security baseline for models from the model catalog, regardless of their origin (e.g., it is not the same in terms of AI security DeepSeek than native Azure OpenAI models).

System-level safety → The AI model is just one part of the equation. The models are part of AI-enabled systems that interact with other technology blocks, and human users. The systems can be (and will likely be) attacked, so one of the key components is the AI Content Safety module from Microsoft Azure AI Foundry, available as a studio feature, but also programmatically via API and SDK. You can see this level of protection as both a shield that protects the system against all sorts of prompt and image-based attacks, and a content filter for negative categories such as hate, violence, and others.

AI data management → As you have seen earlier, RAG and fine-tuning patterns rely on additional data sources that are actual company assets. As any other asset, they need to be protected and managed accordingly, and that means blocking any undesired access, enabling private connections to the data sources, and leveraging that data via LLMs in a proper way.

Data governance → The native flavour for data management and governance is Microsoft Purview. AI systems require proper governance around the data they access and use. Purview includes a unified data catalog for hybrid and multi-cloud environments, classification and quality rules, etc. It also enables lineage tracking, sensitivity labeling (e.g., “Confidential,” “Restricted”), and better decision-making about which datasets are safe to expose to LLM workflows.

Besides regular data governance features, Microsoft Purview includes the notion of Data Security Posture Management (DSPM) for AI. Purview’s DSPM suite of tools helps understand and improve the data security posture across various cloud environments. It focuses on identifying and mitigating risks related to sensitive data, ensuring compliance, and protecting against data breaches. It also helps monitor the interaction in AI-enabled tools to spot potential misuse, unethical behavior, and data oversharing.

Private links → These links enable private connections to the data sources, for example while connecting LLMs with data sources and resources such as Azure AI Search. Data should not transit over the public internet when accessed by AI workloads. Using resources like Azure Private Link help you establish secure, private connections between the models and internal data sources (e.g., blob storage, SQL databases, Cosmos DB). This eliminates exposure to the internet, reduces attack surfaces, and ensures compliance with internal security standards and industry regulations such as HIPAA or ISO 27001.

Active directory (or Microsoft Entra ID) → This means blocking any undesired access based on specific identifies and pre-established rights. For example, sensitive business data used in RAG or fine-tuning should never be publicly accessible or loosely permissioned. You can implement strict access controls via Microsoft Entra (former Azure Active Directory), by using role-based access control (RBAC) or Access Control Lists (ACLs) for Azure Data Lake Gen 2 to prevent unauthorized use. This means ensuring that only verified workloads, users, and services can access specific data endpoints. You can find specific instructions for Azure OpenAI models in the official documentation.

End-to-end monitoring and tracing → As part of the responsible AI and governance framework on Azure, monitoring plays a crucial role in ensuring security, compliance, and operational transparency. There are different building blocks you need to understand:

Cloud and AI monitoring → Microsoft Defender offers native tools to monitor and protect AI workloads, data sources, and the rest of the infrastructure. It provides advanced threat protection for key resources used in generative AI systems. For example, it monitors for suspicious activity in AI pipelines, such as unexpected model access, tampering with training data, or misuse of containerized workloads. You can also monitor components like storage, compute, or API gateways like Azure API Management. This includes functionalities for AI Security Posture Management in multiple cloud environments like Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP) Vertex AI, as well as AI libraries like Langchain, TensorFlow, or PyTorch. These functionalities secure generative AI applications throughout the entire lifecycle, from initial development to deployment. The main features are the AI Bill of Materials (AI BOM), which identifies all components and assets in AI applications, recommendations to strengthen app configurations and fix vulnerabilities (e.g., zero-trust, least privilege, private networking), and attack path analyses to spot potential security risks across environments. It can also help with non-compliant resources, such as LLM data sources missing encryption or access controls.

Application tracing → Azure AI Foundry’s trace capability provides visibility into the full lifecycle of AI application operations, enabling teams to observe, analyze, and optimize how generative AI components interact. It captures detailed telemetry such as prompts, responses, data retrievals, and model invocations, offering an audit trail that enhances traceability, debugging, and performance tuning. This is especially valuable in complex systems like RAG or multi-agent setups, where multiple steps and services contribute to a single AI output. Also, tracing enables responsible AI development and compliance by bringing transparency and accountability to the internal workings of LLM-based systems.

Secrets protection → All the previous elements rely on resources like Azure Key Vault (AKV) secures passwords, cryptographic keys, and secrets. AKV also alerts on unusual access patterns to sensitive credentials used by AI services, and enables scenarios with Managed Identities for Azure resources like models and data sources, so you can consider it as part of the base for any kind of generative AI implementation on Azure. The main features are:

Secrets management → AKV stores API keys, database connection strings, model endpoints, and other sensitive AI credentials securely. This is crucial for any product-level deployment.

Key management → AKV helps manage encryption keys for protecting data used in training, RAG, or inference pipelines (e.g., encrypting embeddings or model outputs at rest).

Certificate storage → AKV manages SSL/TLS certificates for AI services that require encrypted communication, for example between model APIs and data sources.

Impact of AI model choice → Besides purely technological considerations such as model performance or latencies, or financial questions like value for money (e.g. cost per token, etc.), choosing AI models has direct implications on the generative AI governance approach. Your governance framework needs to differentiate between open and closed models, including situations in which you would combine both. Closed models (e.g. Azure OpenAI or other proprietary providers) offer built-in controls, standardized safety layers, and SLAs that ease governance but may limit customization. On the other hand, open models (such as those from Hugging Face or specific open-weight LLMs from providers like DeepSeek, Meta, or Mistral) provide flexibility and transparency but demand stronger internal controls, including fine-tuning governance, license compliance, and safety alignment. Your strategy must account for model lineage, update management, and internal vetting procedures, especially when integrating open models into production workflows.

Regardless of your model choice or mix of them, you will still rely on the shared responsibility considerations, and the multiple levels of AI protection you can add to the baseline AI models.

Implications of multi-agents and other advanced architectures → As generative AI architectures evolve to include multi-agent systems, your generative AI governance system needs to adapt accordingly. These systems introduce new complexities, such as inter-agent communication, delegated autonomy, and cascading errors or biases. Consider how agents are coordinated, what guardrails each agent adheres to, and how accountability is tracked across agent interactions. Azure AI Foundry and other Azure resources can help you build orchestrated agent frameworks, but it remains your responsibility to monitor emergent behaviors, audit decision paths, and ensure each agent operates within defined policy boundaries. You can certainly leverage the monitoring and tracing features you have learnt about, as well as GenAIOps/LLMOps to monitor quantitative performance and enable alerts and human-in-the-loop mechanisms. And keep in mind that new multi-agents patterns multiply the need for transparency from a regulatory and compliance perspective.

All these definitions and considerations will impact the way your company and yourself implement the AI governance system. As you have seen, there are plenty of technology considerations (e.g. creating your own integrations, like using Purview to classify data, then Defender to monitor access, via Azure API Management, etc.), but also adjacent topics related to the external context of the organization, and very different ways to implement this critical, yet not fully defined and standardized area of knowledge. To make it a bit more tangible, let’s go into the details of all Microsoft and Microsoft Azure resources you need to know if you want to implement the technical layer of the AI governance framework.

Key Building Blocks

There are several Microsoft native and 3rd party services you may want to consider while building the technology stack for your generative AI governance framework. In Figure 8-7, you can see a selection of tools that will easily integrate with your generative AI architectures, but you may want to explore equivalent and additional tools on your own.

Figure 8-7. Generative AI Governance Building Blocks

How-To: Implementing Generative AI Governance on Azure

Now that you have the key information related to generative AI governance, let’s try to make sense of the entire picture. Once again, keep in mind there is no one way to design and executive the governance practice, but in this section we will think about all relevant topics in different technical domains, keeping in mind considerations at human and technical levels.

In Figure 8-8, you can see a one-shot representation of what the entire framework could look like, including specific Microsoft-related services and features, and the step-by-step explanation with an illustrative sequence of steps you may want to follow to make it happen:

Figure 8-8. Microsoft Resources for Generative AI Governance

Preliminary, general first steps

Before jumping into the specific steps from Figure 8-8, you probably want to checklist a few elements that will enable your company’s top-down approach to generative AI governance. Less technical, a bit more ethereal, but equally important if you want to see it happening at company level. Here are the main three topics you need to cover:

Defining core principles → Assuming that your company hasn’t done anything in terms of AI governance before, then you would expect some initial discussion with all executive and technical teams around the ethical approach of the company. Concretely, one of the very first steps for any governance initiative is the definition of the core principles that will guide the way to implement AI in your company. Remember, principles like transparency, privacy, respect to human rights, or any other that is aligned with the company’s mission and moral. But also very related to existing regulatory requirements.

Inventorying your immediate compliance needs → As you have seen previously in this chapter, all applicable AI, data, and vertical regulations, as well as suitable standards such as NIST or ISO are the baseline for your governance and compliance goals. For example, if your regulatory and standardization mix always mentions transparency and privacy as key requirements, then you will enable both topics at technical level (e.g., model cards for your models, data retention policies, etc.). If the company is starting from scratch, the recommendation would be to focus on 3 to 5 topics max, so you can convert these needs into technical implementations.

Understanding your team’s AI governance and technical literacy → Before choosing the mix of technical tools you need to implement the AI governance, you will need to analyze the profile and level of knowledge of your internal teams. This means, a highly technical group could make use of Microsoft’s Responsible AI Toolbox, but that toolset may not be optimal for less technical users as they would need some coding and infrastructure deployment knowledge. Similarly, solutions like Microsoft Defender and Entra are usually led by CISO / Security teams, so you need to keep in mind that generative AI initiatives are usually led by data and AI teams.

These considerations are the baseline for the next workstreams. Let’s now go into more technical matters.

Human + AI Approach

This stream focuses on designing, deploying, and securing AI systems that “collaborate” effectively and ethically with people. It provides a holistic approach that combines ethical design principles, operational evaluation workflows, and robust security defenses to govern how humans interact with AI. From enabling transparent AI-generated content to defending against adversarial misuse, these steps will enable you and equip the company with the tools and practices needed to build responsible, user-centered AI experiences.

Establishing the technical foundations → In this case, you will be laying the ethical and usability AI foundation with elements like the HAX Toolkit, AI-enabled code review (e.g., GitHub Copilot or others), AI Impact Assessments, and technical mechanisms like generative AI model watermarks. Key steps include:

Start with the HAX Toolkit, and use it during the design phase to map the human-AI interaction journey. Then, evaluate how users understand, trust, and collaborate with the AI system. And finally, adjust UI and messaging to match user mental models and confidence thresholds.

Use the AI impact assessments up-front, before deploying or even before starting the development phase. If you conduct a pre-launch ethical review based on an existing template (e.g. UNESCO, Microsoft), you may identify how AI outputs might disproportionately affect certain groups, anticipate key considerations across the development phases, document potential social and legal risks, and add the generated documentation as internal governance and transparency assets for regulatory compliance.

Leverage AI-enabled code review. You can do this manually with tools like Github Copilot, in which AI models can assist you in generating code for your AI applications, and running automated reviews. Additionally, you can enforce manual validation by human developers to catch unsafe logic, insecure code, or bias, and leverage other automated code analysis tools like Microsoft Defender. You can also implement specific CI/CD pipeline steps for automated code review, including integrating static analysis tools such as SonarQube or Bandit, that specifically scan for common AI-related vulnerabilities or insecure coding patterns alongside traditional security checks.

Design your apps with model watermarks. This means you need to add identifiable markers to any AI-generated content if your end solution does something like that. This is key for a proper Human + AI approach, and to let end-users know when they’re engaging with AI-enabled systems. This is essential for mitigating misinformation and ensuring traceability, but also a key regulatory requirement and an industry best practice, so there is no reason for you not to do so.

Operationalizing Human + AI collaboration → This part focuses on the practical implementation, monitoring, and lifecycle management of AI systems across organizations. More concretely, it focuses on the initial definition of the multiple evaluation flows that can analyze and track AI performance, both in automated and manual-human ways.

Define your own mix of evaluation assets, which may include automated GenAIOps with the performance metrics you have seen earlier, but also direct human feedback (e.g., based on thumbs up or down, directly via UI), or even product analytics data from tools like Pendo, Heap, or Amplitude. The mix of automated and manual (human) signals should give you enough information to understand the way your AI tool is working.

Build your structured Evaluation Flows via Azure AI Foundry’s Evaluations feature. These flows create a feedback loop that simulates real-world human interactions, including specific user scenarios with ambiguous queries, emotional or edge responses, urgent decision-making, etc. Based on continuous monitoring techniques, you will analyze the outcomes from test flows (including those from Evaluations) to update models and versions, refine specific model behavior, and improve existing prompt templates.

Securing your AI systems against human misuse → The human factor in this case comes from both the ability of people to help improve your security and safety foundations, but also to be the origin of potential threats and risks. This part contains building blocks such as AI red teaming activities, and the use of AI content safety toolkits. Concretely:

Perform ethical hacking (a.k.a. AI red teaming) activities using PyRIT to simulate malicious users trying to exploit the model. There are different types of attacks you can perform, and therefore avoid by blocking them in advance:

Text-based attacks in which the user tricks an LLM, for example, into revealing confidential information by embedding instructions inside natural conversation.

Image-based attacks using encoded instructions inside an image, or even multimodal attacks involving both text and images at the same time. For example, sending an image containing blur but readable text that instructs the AI to override security measures.

This may include other attacks that leverage internal functions of the LLM, via system-level commands that attackers can exploit to manipulate behavior, like activating internal developer or system debug modes. Due to the evolving nature of tools like PyRIT, we recommend you visit the “Cookbooks” section of their official website to get code samples and usage instructions.

Apply all available AI content safety mechanisms → These are product features from AI Foundry and the Azure AI Content Safety API that are designed to guard against malicious inputs and prevent harmful or unintended outputs. These mechanisms should be implemented as a layered defense strategy:

Guard against prompt injection, and sanitize user inputs by filtering dangerous tokens and isolating system prompts to prevent users from modifying internal instructions or gaining control over the AI’s behavior.

Enable prompt shields by intercepting and validating user prompts before model inference using classifiers or rule-based systems to detect unsafe or manipulative intent.

Detect jailbreak attempts based on specific patterns, prompt behavior logs, and supervised classifiers to identify and block attempts to override model safety instructions.

Leverage content filters and apply input and output-level moderation tools to automatically detect and block inappropriate, harmful, or policy-violating responses generated by the model.

Summarizing, you can leverage these best practices to implement the Human + AI layer of your governance approach. This includes incorporating considerations for ethical design and adversarial thinking, and improving human relevance and oversight, including approval for high-impact or sensitive outputs.

AI Models, Data, and Infrastructure

The trifecta of AI models, data sources, and base infrastructure forms the technical backbone of any AI system, and the corresponding technical building blocks for generative AI governance. Besides the previous Human + AI considerations, this applied approach includes governing model behavior, managing and securing data pipelines, and optimizing plus protecting compute environments. This stream focuses on establishing traceability, control, and operational efficiency across the entire AI stack, and it addresses key challenges such as model performance, data privacy, regulatory compliance, cost management, and system resilience.

Bundling the technical governance → Bring anything else to the equation to complete your AI governance approach. Maintain an AI bill of materials to document your model components, data dependencies, and licensing considerations, and include any metadata for version history, source, usage context, and deployment status for governance and audit readiness. You can also leverage Microsoft Purview and/or Unity Catalog (Azure Databricks) for AI data governance, and manage AI model access policies to define how these models are integrated and consumed.

Maintaining operations at scale → Besides the GenAIOps / LLMOps metrics for AI model behaviour, keep track of others such as token usage, latency, throughput, and failure rates to manage operational efficiency. Handle API and prompt management with building blocks such as Azure API Management to ensure consistent governed usage and quality control, including custom policies, rate limiting, input validation, prompt transformation, etc. Leverage private endpoints and RBAC (Role-Based Access Control) as much as you can, and monitor the data lifecycle to apply enhancements for deduplication, entity resolution, normalization, and embedding pipelines to increase data utility for AI consumption. Last but not least, implement AI cost management (FinOps), leverage optimized AI infrastructure such as GPUs and Provisioned Throughput Units (PTU).

Deep diving on security aspects → Reality is that security is a continuous and iterative exercise. Remember, you can leverage Purview’s Information Protection to align with AI compliance frameworks (e.g., EU AI Act, NIST RMF, ISO/IEC 42001), enable DSPM features to identify and mitigate risks in sensitive data handling or shadow AI pipelines, apply sensitivity labels to classify data, run data security investigations periodically to audit data access, detect exfiltration risks, and resolve policy violations. And remember that Defender for Cloud and for Data will help you secure virtual machines, Kubernetes clusters, and databases running AI workloads, including multi-cloud environments outside of Microsoft Azure. Last but not least, you can also integrate security scanning tools for static and dynamic application security testing (SAST, DAST) into CI/CD pipelines for your AI applications.

These nine recommendations are just a basic attempt to summarize and organize what your comprehensive generative AI governance framework for Microsoft Azure environments would look like. What’s missing? The tenth “bullet point”, also known as your actual role as part of this generative AI governance system.

Last but not least: Are you the Responsible AI Champion?

Now, keep in mind that any governance is a mix of people, processes, and tools. Tools enable and automate certain policies and processes, but who (people) is in charge of implementing the generative AI governance framework in your organization? It depends.

If your company has a clearly defined structure for responsible AI and governance topics, then maybe there are official roles assigned to specific activities, such as ethical committees for collective decision-making, or maybe technical functional roles in charge of managing AI implementations, and flagging potential risks based on pre-established processes. Even other departments such as corporate responsibility or compliance may be part of the equation. In this case, regardless of your role, then you will be part of an operating system that will guarantee an ethical, responsible, accountable, and trustworthy implementation.

If on the contrary, your company is new to this kind of topic, and you are currently building up some level of AI maturity, maybe there are no official roles and responsibilities assigned. In that case, a mix of official and informal roles may arise, and you could proactively be part of the AI governance approach. You can be one Responsible AI Champion in your company!

And RAI Champions are not entirely new. Some of the most advanced companies have created this role, as someone in the organization who ensures that artificial intelligence is used in a proper way. They help teams follow the company’s AI principles, including technical teams, business leaders, and legal or ethics groups to spot risks and guide responsible practices. They promote awareness, help set policies, and often act as a bridge between ethics and technology, ensuring that responsibility stays at the center of AI development and use. They conduct impact assessments, document risks, escalate potential issues, and try to connect the points in such a nascent and complex domain.

Keep this in mind: Someone will need to take the lead at some moment, and you have now all the technical and organizational elements to make it happen. For example, as a RAI Champion you might use PyRIT to validate your generative AI model safety, lead the completion of a transparency-oriented AI software bill of materials, or escalate issues to the AI ethics committee.

Conclusion

As generative AI becomes more integrated into business operations, establishing a comprehensive generative AI governance framework is more than essential. This chapter has outlined a practical, end-to-end approach to governing AI responsibly, aligning ethical principles with technical implementation using Microsoft Azure and its surrounding ecosystem. From human-centered design and content moderation to AI security, data governance, and operational efficiency, each component plays a critical role in building reliable AI systems. And of course, this is a collective effort, but with the knowledge and tools provided in this chapter, you’re now equipped to drive responsible AI forward.

But before moving to the next chapter, jump to the end of chapter test to validate your understanding of the most relevant topics.

End of chapter test

What is the function of AI red teaming tools like PyRIT in responsible AI governance?

Generating synthetic training datasets

Automating content moderation pipelines

Simulating adversarial attacks to test model safety and robustness

Managing role-based access across cloud environments

Answer:

Incorrect → PyRIT doesn’t create datasets. It simulates adversarial behavior to test vulnerabilities.

Incorrect → Content moderation is handled by tools like Azure AI Content Safety, not PyRIT.

Correct → PyRIT is designed to mimic malicious user behavior and stress-test AI systems before deployment.

Incorrect → This is the function of Microsoft Entra or similar identity management tools.

Which of the following best describes the Shared Responsibility Model in Microsoft Azure’s generative AI governance?

Microsoft is fully responsible for AI model behavior and user output

Users must manually secure the physical data centers

Responsibility is split between Microsoft and the AI-adopting organization

The model provider is solely accountable for ethical AI outcomes

Answer:

Incorrect → Microsoft provides tools and infrastructure, but the organization using the AI holds responsibility for how it is applied.

Incorrect → Microsoft is responsible for the physical and foundational cloud infrastructure, not the customer.

Correct → Microsoft manages infrastructure security, while customers govern data usage, ethical deployment, and prompt/output control.

Incorrect → Both the provider and the deploying organization share ethical and legal accountability under most governance frameworks.

What is the primary purpose of a Responsible AI (RAI) Champion within an organization?

To build AI models using the most advanced algorithms

To monitor system performance and user engagement with AI applications

To ensure AI systems align with ethical and responsible use principles

To manage cloud infrastructure and network configurations

Answer:

Incorrect → This is typically the job of AI developers or data scientists. A RAI Champion focuses on ethical oversight, not model engineering.

Incorrect → Monitoring performance or user engagement is not the primary focus of a RAI Champion.

Correct → A RAI Champion ensures the ethical use of AI, fosters awareness, and supports governance efforts across teams.

Incorrect → This is the domain of cloud engineers or IT operations, unrelated to the responsibilities of a RAI Champion.

What role does Microsoft Purview primarily serve in the AI governance framework?

Enables unified data governance, classification, and risk management

Automates red teaming and adversarial testing of AI models

Monitors content safety in LLM-generated outputs

Provides pretrained generative AI models for Azure OpenAI

Answer:

Correct → Purview provides centralized governance over data discovery, classification, access control, and regulatory compliance.

Incorrect → This is the role of tools like PyRIT. Purview is focused on data governance, not adversarial testing.

Incorrect → Azure AI Content Safety is responsible for content filtering, not Purview.

Incorrect → Azure OpenAI (and others) handles model access and inference.

Which of the following is an example of a Human-in-the-Loop (HITL) approach in AI governance?

Encrypting all training data

Having a person review and approve AI decisions in high-stakes scenarios

Automating all prompt handling through API gateways

Fine-tuning an open model with proprietary data

Answer:

Incorrect → Encryption is a data security measure, not a HITL approach. HITL is about human oversight in decision-making.

Correct → This reflects the core principle of HITL, which is adding human judgment to AI outputs, especially in sensitive use cases.

Incorrect → This is an automation step and removes human intervention, the opposite of a HITL pattern.

Incorrect → While this is a valid AI practice, it has nothing to do with human oversight in AI decision-making.