Ground the Model in Your Own Data — RAG with Azure AI Search

In chapter 1 we deployed a model and called it keylessly. Impressive for ten lines of code — but ask it about your product docs, your runbooks, your refund policy, and it will either admit ignorance or, worse, confidently invent an answer. This chapter fixes that with retrieval-augmented generation (RAG): fetch the relevant facts first, hand them to the model, and instruct it to answer only from what it was handed.

Why RAG beats the alternatives

You have three ways to teach a model about your data. Fine-tuning bakes knowledge into weights — expensive, slow to update, and wrong for facts that change. Stuffing everything into the prompt works until your data outgrows the context window and your token bill outgrows your patience. RAG keeps data outside the model, retrieves only what's relevant per question, and updates instantly when your documents do. For knowledge that changes — which is nearly all knowledge worth building an app around — RAG is the default answer.

The retrieval half needs a search engine. Enter Azure AI Search.

Step 1 — Create the search service

RG=rg-my-first-ai-app     # the resource group from chapter 1
az search service create \
  --name srch-my-first-ai-app \
  --resource-group $RG \
  --sku free \
  --location eastus2

Cost note: the Free tier gives you 50 MB and 3 indexes — plenty for this series, and genuinely $0. One free search service per subscription. The next rung (Basic) is real money, so stay on Free until you outgrow it.

While it deploys, gather a handful of your own documents — markdown, PDFs, a scraped FAQ, whatever you want the app to be an expert in. I'll use a folder of product docs as the running example.

Step 2 — Build an index

An AI Search index is a schema plus documents. Modern RAG indexes lean on vector search: each chunk of text is stored alongside an embedding so queries match by meaning, not just keywords. The best results usually come from hybrid queries — keyword + vector + semantic re-ranking — and AI Search does all three.

The quickest on-ramp is the portal's Import and vectorize data wizard: point it at a blob container with your files, let it chunk them, pick an embedding model deployment from your Foundry project, and it generates the index, the indexer, and the vectorizer. Five minutes, no code.

Because this series is hands-on, here's the shape of doing it yourself with the SDK, so nothing feels like magic:

from azure.identity import DefaultAzureCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents import SearchClient
 
SEARCH_ENDPOINT = "https://srch-my-first-ai-app.search.windows.net"
credential = DefaultAzureCredential()  # keyless, same as chapter 1
 
# Index: id + content + vector field (dimensions match your embedding model)
# ...define fields with SearchField/VectorSearch config...
 
search = SearchClient(SEARCH_ENDPOINT, "product-docs", credential)
search.upload_documents(documents=chunks)  # your chunked + embedded docs

Chunking advice that will save you a headache: aim for chunks of a few hundred tokens with a little overlap, keep a source field on every chunk (you'll want citations later), and don't over-engineer — you can re-chunk in an afternoon once evaluation (chapter 4) tells you what's actually wrong.

Keyless note: enable role-based access on the search service (portal → Keys → toggle to RBAC or both) and grant yourself Search Index Data Contributor to write, and your app's identity Search Index Data Reader to query. Same identity story as chapter 1, new resource.

Step 3 — Wire retrieval into generation

RAG at runtime is honestly three steps: search, stuff, ask.

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.search.documents import SearchClient
 
credential = DefaultAzureCredential()
project = AIProjectClient(endpoint=PROJECT_ENDPOINT, credential=credential)
openai = project.get_openai_client()
search = SearchClient(SEARCH_ENDPOINT, "product-docs", credential)
 
def ask(question: str) -> str:
    # 1. SEARCH — hybrid query, top 5 chunks
    results = search.search(search_text=question, top=5)
    context = "\n\n".join(
        f"[{r['source']}]\n{r['content']}" for r in results
    )
 
    # 2. STUFF + 3. ASK — grounded instructions are the secret sauce
    response = openai.responses.create(
        model="chat-small",
        instructions=(
            "Answer using ONLY the sources below. Cite the source name in "
            "brackets after each claim. If the answer is not in the sources, "
            "say you don't know — do not guess.\n\n--- SOURCES ---\n" + context
        ),
        input=question,
    )
    return response.output_text
 
print(ask("What's our refund window for annual plans?"))

Run it against a question your documents actually answer, then against one they don't. The second test matters more: a grounded app that says "I don't know" is trustworthy; one that improvises is a liability. That one instruction — answer only from the sources, refuse otherwise — is the difference between a demo and a product.

The managed alternative

Foundry can also run retrieval for you: attach your data as a file search tool (Foundry manages a vector store) or plug your AI Search index in as a knowledge tool on an agent, and the platform handles query planning and citations. It's less code and it's excellent — and after building retrieval by hand once, you'll actually understand what it's doing and be able to debug it when relevance goes sideways. We'll meet platform tools properly in the next chapter.

Wrapping up

Your app now answers from your documents, cites its sources, and refuses to bluff — running on a free search tier with identity-based auth throughout. The model finally knows your world.

But it can only talk about it. Chapter 3 gives the model hands: tools and function calling, so it can look up live data and take real actions instead of just describing them.