Chapter 3 — Build it: a production-ready Azure agent with real tools

In Chapter 2 we wired MCP into a safe path to your Azure estate. Now we’ll build something you can ship: a production-ready agent that runs in Azure AI Foundry, calls real tools through API Management, and leaves you with logs, identities, and guardrails you’d expect in a cloud environment. We’ll finish with a deployable reference and set you up for Chapter 4’s when-to-use-it decisions.

Reference architecture (what we’re building)

Here’s the flow I use in real projects:

User/app → Azure AI Foundry agent
Agent “tools” are HTTP APIs fronted by API Management
Behind APIM, tools run as Azure Functions or Azure Container Apps and (optionally) speak MCP to internal services
Tools use Managed Identity to call Azure services (Key Vault, Storage, Resource Graph, etc.)
Observability with Application Insights + Log Analytics; APIM adds correlation headers
Optional: VNet + private endpoints for the model endpoint, APIM, Functions/Container Apps, and Key Vault
CI/CD pushes Prompt Flow artifacts and infra code, evaluates flows on datasets, and gates deploys

Why this shape?

You get a single secured surface (APIM) for all tools—authN/Z, quotas, transform, and safe rollout.
You can evolve tools independently (Functions for quick ops-y bits, Container Apps for heavier servers like MCP).
You get first-class telemetry for both the agent and tool calls without bolting it on later.

Provisioning with Bicep

This Bicep template stands up the core: a resource group, Azure AI resources for the agent, APIM, Function App, Container Apps environment (optional), Key Vault, Application Insights, Log Analytics, Storage, and a VNet with private endpoints. It’s trimmed to the essentials but runnable.

Deployable Bicep

targetScope = 'resourceGroup'
 
@description('Location for all resources')
param location string = resourceGroup().location
 
@description('Name prefix for resources')
param prefix string = 'agentref'
 
@description('App Service plan SKU for Functions (Elastic Premium recommended for private networking + MI)')
param funcSkuName string = 'EP1'
 
@description('APIM SKU name')
param apimSkuName string = 'Consumption' // good to start; scale later
 
var name = toLower('${prefix}${uniqueString(resourceGroup().id)}')
var vnetName = '${name}-vnet'
var subnetName = 'apps'
var privateDnsZoneName = 'privatelink.openai.azure.com'
 
resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' existing = {
  name: resourceGroup().name
}
 
// Log Analytics
resource la 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
  name: '${name}-law'
  location: location
  properties: {
    retentionInDays: 30
  }
}
 
// App Insights (workspace-based)
resource appi 'Microsoft.Insights/components@2020-02-02' = {
  name: '${name}-appi'
  location: location
  kind: 'web'
  properties: {
    IngestionMode: 'ApplicationInsights'
    WorkspaceResourceId: la.id
  }
}
 
// Storage for Functions
resource st 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: replace('${name}stg','-','')
  location: location
  sku: { name: 'Standard_LRS' }
  kind: 'StorageV2'
  properties: {
    minimumTlsVersion: 'TLS1_2'
    allowBlobPublicAccess: false
  }
}
 
// VNet
resource vnet 'Microsoft.Network/virtualNetworks@2023-09-01' = {
  name: vnetName
  location: location
  properties: {
    addressSpace: { addressPrefixes: [ '10.10.0.0/16' ] }
    subnets: [
      {
        name: subnetName
        properties: {
          addressPrefix: '10.10.1.0/24'
          delegations: [
            {
              name: 'functions-del'
              properties: {
                serviceName: 'Microsoft.Web/serverFarms'
              }
            }
          ]
          privateEndpointNetworkPolicies: 'Disabled'
          privateLinkServiceNetworkPolicies: 'Disabled'
        }
      }
    ]
  }
}
 
// Key Vault
resource kv 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: '${name}-kv'
  location: location
  properties: {
    tenantId: subscription().tenantId
    sku: { name: 'standard', family: 'A' }
    enableRbacAuthorization: true
    networkAcls: {
      defaultAction: 'Deny'
      bypass: 'AzureServices'
      virtualNetworkRules: [
        {
          id: vnet.properties.subnets[0].id
        }
      ]
      ipRules: []
    }
  }
}
 
// Azure AI Foundry resources: Hub + Project + (model deployment as needed)
// Using AI Projects (microsoft.ai) resource provider
resource hub 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
  // For many subscriptions, Azure AI Foundry hub/projects map to AML workspaces under the hood
  name: '${name}-aihub'
  location: location
  sku: { name: 'Basic' }
  properties: {
    applicationInsights: appi.id
    containerRegistry: null
    keyVault: kv.id
    storageAccount: st.id
    publicNetworkAccess: 'Disabled'
  }
}
 
// API Management
resource apim 'Microsoft.ApiManagement/service@2023-05-01' = {
  name: '${name}-apim'
  location: location
  sku: {
    name: apimSkuName
    capacity: 0
  }
  properties: {
    publisherEmail: 'owner@example.com'
    publisherName: 'Agent Team'
    virtualNetworkType: 'None'
  }
}
 
// Function App (Linux, Python runtime for simplicity)
resource plan 'Microsoft.Web/serverfarms@2023-12-01' = {
  name: '${name}-plan'
  location: location
  sku: {
    name: funcSkuName
    tier: 'ElasticPremium'
    size: funcSkuName
  }
  kind: 'elastic'
}
 
resource func 'Microsoft.Web/sites@2023-12-01' = {
  name: '${name}-func'
  location: location
  kind: 'functionapp,linux'
  properties: {
    serverFarmId: plan.id
    siteConfig: {
      appSettings: [
        { name: 'AzureWebJobsStorage', value: 'DefaultEndpointsProtocol=https;AccountName=${st.name};EndpointSuffix=core.windows.net' }
        { name: 'FUNCTIONS_EXTENSION_VERSION', value: '~4' }
        { name: 'FUNCTIONS_WORKER_RUNTIME', value: 'python' }
        { name: 'APPLICATIONINSIGHTS_CONNECTION_STRING', value: appi.properties.ConnectionString }
        { name: 'WEBSITE_RUN_FROM_PACKAGE', value: '1' }
      ]
      linuxFxVersion: 'Python|3.11'
    }
    httpsOnly: true
  }
  identity: {
    type: 'SystemAssigned'
  }
}
 
// APIM API shell; you’ll import operations later
resource api 'Microsoft.ApiManagement/service/apis@2022-08-01' = {
  name: '${apim.name}/agent-tools'
  properties: {
    displayName: 'Agent Tools'
    path: 'tools'
    protocols: [ 'https' ]
  }
}
 
// Log to LAW from APIM
resource diag 'Microsoft.ApiManagement/service/diagnostics@2021-08-01' = {
  name: '${apim.name}/apimdiag'
  properties: {
    enabled: true
    loggerId: 'applicationinsights'
  }
}
 
// Optional: Container Apps env for MCP servers
resource caEnv 'Microsoft.App/managedEnvironments@2024-03-01' = {
  name: '${name}-cae'
  location: location
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: { customerId: la.properties.customerId, sharedKey: listKeys(la.id, '2020-08-01').primarySharedKey }
    }
  }
}

Deploy it:

# variables
SUB=<your-subscription-id>
LOC=eastus2
RG=rg-agent-ref
 
az account set -s "$SUB"
az group create -n "$RG" -l "$LOC"
az deployment group create -g "$RG" -f main.bicep -p location=$LOC prefix=agentref

A few trade-offs:

I use the APIM Consumption tier to start—cheap and bursts—but it has quotas and cold starts. If you need stable throughput or VNet injection, move to Developer/Basic or Premium (check the pricing page).
Functions on Elastic Premium reduce cold starts and support VNet integration; for heavy MCP servers or long-running processes, Container Apps is safer.

Implementing a real tool (Function + APIM)

We’ll build a concrete tool the agent can call: “list_resource_groups” that returns the caller’s Azure resource groups. In production, you’d add more: create/update resources, run queries, etc.

Azure Function (Python) using Managed Identity

This HTTP-triggered Function calls ARM via the Azure SDK and emits structured logs for correlation.

# functions/list_resource_groups/__init__.py
import json
import logging
import os
import azure.functions as func
from azure.identity import ManagedIdentityCredential
from azure.mgmt.resource import ResourceManagementClient
 
def main(req: func.HttpRequest) -> func.HttpResponse:
    logger = logging.getLogger("tool.list_resource_groups")
    trace_id = req.headers.get("x-correlation-id", "")
    logger = logging.LoggerAdapter(logger, {"correlationId": trace_id, "tool": "list_resource_groups"})
 
    sub_id = os.environ.get("AZURE_SUBSCRIPTION_ID")
    if not sub_id:
        return func.HttpResponse(json.dumps({"error": "AZURE_SUBSCRIPTION_ID not set"}), status_code=500, mimetype="application/json")
 
    cred = ManagedIdentityCredential()
    client = ResourceManagementClient(cred, sub_id)
    groups = [{"name": g.name, "location": g.location, "tags": g.tags or {}} for g in client.resource_groups.list()]
    logger.info("success", extra={"count": len(groups)})
 
    return func.HttpResponse(json.dumps({"resource_groups": groups}), status_code=200, mimetype="application/json")

function.json:

{
  "bindings": [
    { "authLevel": "function", "type": "httpTrigger", "direction": "in", "name": "req", "methods": [ "post", "get" ], "route": "list-resource-groups" },
    { "type": "http", "direction": "out", "name": "$return" }
  ]
}

Deploy:

FUNCZIP=func.zip
pushd functions
zip -r ../$FUNCZIP .
popd
az webapp deploy --resource-group $RG --name <func-app-name> --src-path $FUNCZIP --type zip
az webapp config appsettings set -g $RG -n <func-app-name> --settings AZURE_SUBSCRIPTION_ID=$SUB
# Grant RBAC to the Function's managed identity
MI_PRINCIPAL_ID=$(az webapp show -g $RG -n <func-app-name> --query identity.principalId -o tsv)
az role assignment create --assignee-object-id $MI_PRINCIPAL_ID --assignee-principal-type ServicePrincipal --role "Reader" --scope /subscriptions/$SUB

Publish via API Management with policies

Expose the Function through APIM, add an API key, inject a correlation ID, and translate errors.

APIM_NAME=<apim-name>
API_ID=agent-tools
FUNC_HOST=$(az webapp show -g $RG -n <func-app-name> --query defaultHostName -o tsv)
 
az apim api create -g $RG --service-name $APIM_NAME --api-id $API_ID --path tools --display-name "Agent Tools" --protocols https
az apim api operation create -g $RG --service-name $APIM_NAME --api-id $API_ID --display-name "List Resource Groups" --name list-resource-groups --method POST --url-template "/list-resource-groups"
az apim api operation policy update -g $RG --service-name $APIM_NAME --api-id $API_ID --operation-id list-resource-groups --xml-content @- <<'XML'
<policies>
  <inbound>
    <base />
    <set-variable name="corr" value="@(context.Request.Headers.GetValueOrDefault("x-correlation-id", System.Guid.NewGuid().ToString()))" />
    <set-header name="x-correlation-id" exists-action="override">
      <value>@((string)context.Variables["corr"])</value>
    </set-header>
    <set-backend-service base-url="https://FUNC_HOST/api" />
    <set-query-parameter name="code" exists-action="skip">
      <value>{{function-key-list-resource-groups}}</value>
    </set-query-parameter>
  </inbound>
  <backend>
    <forward-request />
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <return-response>
      <set-status code="502" reason="Bad Gateway" />
      <set-body>@{ return "Tool call failed."; }</set-body>
    </return-response>
  </on-error>
</policies>
XML

Replace FUNC_HOST with the real hostname or use a named value. I often store function keys and hosts as APIM named values and reference them as {{name}}.

Cost/latency note: APIM’s Consumption tier adds a small per-call overhead; you’re buying central control. If you need the absolute lowest latency and no transforms, you can call Functions directly from the agent, but you lose consistent auth, quotas, and telemetry.

Wiring the agent in Azure AI Foundry

In Azure AI Foundry (project in your hub), create an Agent that uses your chosen model and add a “web API” tool pointing at the APIM operation. High level steps:

In your AI Foundry project, go to Agents → New Agent.
Choose a model deployment (e.g., a GPT-4 family or Phi-4 variant you’ve already deployed). Keep temperature conservative for ops tasks.
Add Tool:
- Type: Web API
- Method: POST
- URL: https://<APIM_NAME>.azure-api.net/tools/list-resource-groups
- Auth: Subscription key (APIM primary/secondary key) or header-based key.
- Request/response schema: define a simple JSON schema with no inputs for this tool.
System prompt: clearly instruct when to call the tool and how to summarize the result for the user.
Test in the playground. Confirm the correlation ID appears in App Insights logs via APIM.

Tip: keep tool schemas tight. Agents behave better with precise input/output contracts.

Prompt Flow setup and evaluation

You need a repeatable way to evaluate prompts and tool strategies before you expose the agent. Prompt Flow gives you datasets, runs, metrics, and artifacts.

Local Prompt Flow with datasets

Directory structure:

promptflow/
  flows/
    rg_inspector/
      flow.dag.yaml
      tools/
        call_list_rg.py
      inputs.jsonl
      eval.jsonl

flow.dag.yaml (minimal chat flow calling the APIM tool):

# promptflow/flows/rg_inspector/flow.dag.yaml
nodes:
  - name: chat
    type: llm
    source:
      type: prompt
      path: prompt.jinja2
    inputs:
      model: azure_openai
      deployment_name: ${env:MODEL_DEPLOYMENT}
      temperature: 0.2
      tools:
        - name: list_resource_groups
          type: web
          method: post
          url: ${env:APIM_BASE}/tools/list-resource-groups
          headers:
            Ocp-Apim-Subscription-Key: ${env:APIM_KEY}
          input_schema: {}
          output_schema:
            type: object
            properties:
              resource_groups:
                type: array
                items:
                  type: object
                  properties:
                    name: { type: string }
                    location: { type: string }

prompt.jinja2:

System: You are an Azure operations assistant. When user asks about resource groups, call list_resource_groups and summarize.
User: {{input}}

Run locally with the Prompt Flow CLI:

# Install promptflow
pip install promptflow promptflow-tools
 
export MODEL_DEPLOYMENT=<your-deployment-name>
export APIM_BASE=https://$APIM_NAME.azure-api.net
export APIM_KEY=<your-apim-subscription-key>
 
pf flow test --flow-dir promptflow/flows/rg_inspector \
  --inputs input="What resource groups do we have?"

Automated evaluation in CI

Use a tiny dataset (eval.jsonl) with prompts and expected patterns. Then a GitHub Actions workflow runs the flow and uploads metrics.

.github/workflows/promptflow-eval.yml:

name: promptflow-eval
on:
  pull_request:
    paths:
      - promptflow/**
 
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install promptflow promptflow-tools
      - env:
          MODEL_DEPLOYMENT: ${{ secrets.MODEL_DEPLOYMENT }}
          APIM_BASE: ${{ secrets.APIM_BASE }}
          APIM_KEY: ${{ secrets.APIM_KEY }}
        run: |
          pf run create --flow promptflow/flows/rg_inspector \
            --data promptflow/flows/rg_inspector/eval.jsonl \
            --column-mapping input='${data.prompt}'
          pf run show --output json > run.json
          python - <<'PY'
import json, sys, re
data=json.load(open('run.json'))
# Very simple gate: require "resource groups" phrase in outputs
fails=[r for r in data.get('nodes',[]) if r.get('name')=='chat' and not re.search('resource group', r.get('output',''), re.I)]
print(f"Failures: {len(fails)}")
sys.exit(1 if fails else 0)
PY

You can also push flows and runs to the Azure AI project; if you prefer cloud-managed runs, add the corresponding Azure AI Foundry integration and service connections in your CI (the exact CLI depends on the “az ai” extension you’re using—pin versions in CI and keep them updated).

Identity and networking

Production trust comes from identity first, then network.

Managed Identity: The Function/Container App uses a system-assigned identity; grant least-privilege roles per tool. In the example, “Reader” at subscription scope. For write tools, scope down to resource groups and use narrow roles.
Secrets: Put any API keys in Key Vault and reference them in Function app settings. Use RBAC-enabled Key Vault and assign the app identity Key Vault Secrets User.
Private endpoints:
- Model endpoint: Deploy your model with private access enabled and route the agent via a private link. Ensure DNS for privatelink.openai.azure.com.
- APIM: For strict egress, put APIM in a VNet and use internal mode. If your agent needs to reach APIM privately, place it in the same VNet or ensure private link routing.
- Functions/Container Apps: Use VNet integration and lock outbound to required services.
Egress controls: For Container Apps, define egress with user-defined routes (UDR) to a firewall; for Functions Premium, restrict WEBSITE_VNET_ROUTE_ALL=1 and use Azure Firewall/DNS to constrain destinations.
Header-level correlation: Standardize x-correlation-id from APIM to downstream, and log it everywhere.

RBAC example:

# Grant only Resource Graph read for a specific tool
az role assignment create \
  --assignee-object-id $MI_PRINCIPAL_ID \
  --assignee-principal-type ServicePrincipal \
  --role "Resource Graph Reader" \
  --scope /subscriptions/$SUB/resourceGroups/rg-inventory

Observability and safe rollout

You want signal on three levels: the agent, the tool calls, and the platform.

Application Insights:
- Enable dependency tracking in Functions.
- Log custom dimensions: tool name, correlation ID, user session hash (never PII).
APIM analytics:
- Review per-operation latency, 4xx/5xx, and throttling.
- Export to Log Analytics for Kusto queries.

Kusto examples you can paste into Logs (App Insights workspace):

// Tool call latency by operation
requests
| where cloud_RoleName == "<func-app-name>"
| summarize p50=percentile(duration,50ms), p95=percentile(duration,95ms) by name
 
// Join agent traces (if you emit a custom event 'agent_tool_call') with function requests
customEvents
| where name == "agent_tool_call"
| project corrId=tostring(customDimensions.correlationId), tool=tostring(customDimensions.tool), agentLatency=toint(customDimensions.latencyMs)
| join kind=inner (
  requests
  | project corrId=tostring(customDimensions.correlationId), funcLatency=duration
) on corrId
| project corrId, tool, agentLatency, funcLatency

Red-teaming and gating

Prompt Flow datasets for jailbreaks and toxic content. Add them to CI as a separate redteam job that must pass before deploy.
APIM rate limits/quotas per client app; add circuit breakers during early rollout.
Canary releases:
- Stage a second APIM revision; route a small percentage via a header-based policy.
- Keep previous prompt/flow and tool versions pinned until evaluation completes.

APIM policy snippet for a simple rate limit and canary:

<policies>
  <inbound>
    <base/>
    <rate-limit calls="60" renewal-period="60" />
    <choose>
      <when condition="@(context.Request.Headers.GetValueOrDefault("x-canary","") == "1")">
        <set-header name="x-agent-variant" exists-action="override">
          <value>canary</value>
        </set-header>
      </when>
    </choose>
  </inbound>
  <backend>
    <base/>
  </backend>
  <outbound>
    <base/>
  </outbound>
</policies>

Optional: Hosting an MCP server in Container Apps

If you want richer tools and local development parity with MCP, run an MCP server in Container Apps and front it with APIM. Your Function can act as a thin bridge to the MCP server.

Example Container Apps service with a TypeScript MCP server (skeletal):

// server.ts
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import express from "express";
 
const app = express();
app.use(express.json());
 
const mcp = new Server({ name: "azure-tools", version: "1.0.0" });
 
// MCP tool: list_resource_groups
mcp.tool("list_resource_groups", { inputSchema: { type: "object", properties: {} } }, async () => {
  // In real code, call Azure SDK with Managed Identity (workload identity in Container Apps)
  return { content: [{ type: "json", data: { resource_groups: [] } }] };
});
 
app.post("/mcp/invoke", async (req, res) => {
  const { name, arguments: args } = req.body;
  const result = await mcp.invokeTool(name, args ?? {});
  res.json(result);
});
 
app.listen(8080, () => console.log("MCP server on :8080"));

Container Apps deploy (YAML):

# ca-azure-tools.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-azure-tools
spec:
  replicas: 1
  selector:
    matchLabels: { app: mcp-azure-tools }
  template:
    metadata:
      labels: { app: mcp-azure-tools }
    spec:
      containers:
        - name: server
          image: ghcr.io/yourorg/mcp-azure-tools:latest
          ports:
            - containerPort: 8080
          env:
            - name: AZURE_CLIENT_ID
              valueFrom:
                secretKeyRef: { name: mi, key: clientId }
---
apiVersion: v1
kind: Service
metadata:
  name: mcp-azure-tools
spec:
  type: ClusterIP
  selector:
    app: mcp-azure-tools
  ports:
    - port: 8080
      targetPort: 8080

Front this with APIM just like the Function. Internally you can now use native MCP clients; the agent still sees a clean HTTP tool.

Cost, latency levers, and operational hygiene

I won’t invent numbers—check the pricing pages for APIM, Functions, Container Apps, Application Insights, and your chosen model. Here’s how I manage the knobs:

Model usage: Smaller, faster models for routing/triage; larger only when needed. Lower temperature for ops tasks; enable response truncation where appropriate.
Tool granularity: Fewer, larger tool calls reduce round trips (latency + cost), but be careful not to create “god tools” that are hard to secure. Start with read-only tools for discovery; add write tools later with approvals.
Caching: Cache stable data (e.g., resource group listing) in a short-lived cache (Storage/Table/Redis) keyed by correlation/user session.
APIM policies: Strip PII, compress responses, and limit payloads to cap egress costs.
Budgets/alerts:
- Set Azure Cost Management budgets on the resource group and subscribe to alerts.
- App Insights alerts for error rates and p95 latency on tool calls.
- Model quota alerts if you’re using capacity-based deployments.

Example budget via CLI:

BUDGET=agentref-monthly
AMOUNT=100 # pick a sane threshold for your test env
SCOPE=/subscriptions/$SUB/resourceGroups/$RG
az consumption budget create \
  --amount $AMOUNT \
  --time-grain Monthly \
  --budget-name $BUDGET \
  --category Cost \
  --scope $SCOPE \
  --notifications \
    actual_greater_than_80_percent={'enabled':true,'operator':'GreaterThan','threshold':80,'contactEmails':['owner@example.com']}

Operational hygiene checklist I use:

Pin model deployment and prompt versions per environment; tag runs in Prompt Flow.
Gate changes through Prompt Flow eval + red-team datasets in CI.
Keep RBAC narrow and rotate APIM keys; prefer client certs or OAuth where possible.
Enable private endpoints for the model endpoint and Key Vault before going broad.
Run resilience tests: fail the Function/Container App and ensure APIM + agent degrade gracefully.

Putting it together: end-to-end test

Deploy Bicep resources.
Deploy the Function and import the APIM operation/policy.
In Azure AI Foundry, create the agent, add the web API tool pointing to APIM.
Use the agent playground to ask: “What resource groups do we have?” and verify the list matches your subscription.
Check App Insights for the trace with your x-correlation-id.
Run the Prompt Flow eval locally and in CI. Only then broaden access.

You now have a production-ready spine: an Azure AI Foundry agent, real tools behind APIM, managed identities, private networking, and full telemetry. In Chapter 4, we’ll step back and decide when this architecture beats a script—and when a humble script is exactly what you should ship.