Chapter 1 — Agents, not magic: a cloud engineer’s mental model

We’re kicking off a four-part series on agents for cloud engineers. If “agents” still feels like marketing fog, this chapter gives you a concrete mental model you can use at work. Next, in Chapter 2, we’ll wire that model to real Azure infrastructure using MCP.

Why agents exist: where scripts and microservices struggle

You and I already automate a lot with scripts, CLI, and microservices. They shine when:

• Inputs are clean and predictable.
• The path is known in advance.
• The work is short-lived and deterministic.

But real-world ops keeps handing us the opposite:

• Ambiguity: “Find the slowest API in prod and propose a fix.” That’s not one command; it’s a plan.
• Multi-step tool use: Pull logs, join with traces, query a CMDB, open a ticket, message a channel.
• Dynamic paths: Each step depends on what the previous step found.
• Human-in-the-loop: You want a draft change or runbook that you review before anything executes.

You can chain scripts or introduce workflow engines (Logic Apps, Durable Functions), but once you start encoding every fork and nuance, you’re maintaining a brittle decision tree. This is where agents fit. An agent’s job is to plan, use tools, observe the result, and adapt—without you prewiring every possible branch.

A practical definition: anatomy of an agent

Here’s the mental model I use in production. An agent is not magic; it’s five parts working together:

1. Policy (system intent and constraints)

   • What’s in scope, what’s out, how to behave, and the safety rails.
   • Example: “You may query Azure Monitor and generate Terraform diffs, but you may not apply changes. Ask for approval first.”

2. LLM (the planner and reasoner)

   • The model turns the policy + current context into the next best step.
   • It doesn’t “know Azure” by itself; it writes plans and selects tools, then reads results.

3. Memory

   • Short-term: the running conversation and tool results in the current session.
   • Long-term: stored knowledge you retrieve when useful (KBs, runbooks, past incidents).
   • Retrieval isn’t magic either—it’s just search with good embeddings and chunking.

4. Tools

   • The agent’s hands. Anything callable: REST APIs (Azure Resource Manager, Azure Monitor), scripts, SDKs, or MCP-exposed resources.
   • Tools must be explicit (name, input schema, output schema). The agent sees only what you give it.

5. Guardrails and observability

   • Guardrails: permissions, red-teaming prompts, allow/deny lists, approvals.
   • Observability: traces of every plan, tool call, result, token usage, and errors (you’ll want these in Application Insights).

If you remember nothing else, remember this: an agent is a loop that plans → acts with a tool → observes → updates the plan, within a policy box.

A minimal loop (TypeScript)

This is a runnable, bare-bones example that shows the pattern. It uses the OpenAI-compatible SDK and a fake “tool” to keep it simple. Swap the model/provider for Azure OpenAI in Azure AI Foundry later; the structure stays the same.

// agent.ts
// npm i openai zod
import OpenAI from "openai";
import { z } from "zod";
 
// A fake tool to illustrate schema'd IO
const getVmStatus = async (vmName: string) => {
  // In real life this would call Azure Resource Graph or ARM
  return vmName === "web-01" ? "running" : "stopped";
};
 
const ToolCallSchema = z.object({
  tool: z.enum(["getVmStatus", "finish"]),
  args: z.record(z.any()).optional()
});
 
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY! });
 
const SYSTEM = `
You are an Azure operations agent. 
- You MAY read VM status via the "getVmStatus" tool.
- You MUST explain your plan briefly before taking action.
- You MUST ask for approval before taking destructive actions (none supported here).
- If a user's goal is satisfied, call the "finish" tool with a concise summary.
`;
 
async function main(userQuery: string) {
  const history: Array<{ role: "system" | "user" | "assistant"; content: string }> = [
    { role: "system", content: SYSTEM },
    { role: "user", content: userQuery }
  ];
 
  for (let step = 0; step < 6; step++) {
    const res = await client.chat.completions.create({
      model: "gpt-4o-mini", // replace with your Azure OpenAI deployment name later
      messages: history,
      tools: [
        {
          type: "function",
          function: {
            name: "getVmStatus",
            description: "Return the power state of a VM by name",
            parameters: { type: "object", properties: { vmName: { type: "string" } }, required: ["vmName"] }
          }
        },
        {
          type: "function",
          function: {
            name: "finish",
            description: "End the task with a short summary",
            parameters: { type: "object", properties: { summary: { type: "string" } }, required: ["summary"] }
          }
        }
      ],
      temperature: 0.2
    });
 
    const msg = res.choices[0].message;
 
    // If the model replies normally, append and continue
    if (!msg.tool_calls || msg.tool_calls.length === 0) {
      history.push({ role: "assistant", content: msg.content ?? "" });
      continue;
    }
 
    // Handle a single tool call for simplicity
    const tc = msg.tool_calls[0];
    const parsed = ToolCallSchema.parse({ tool: tc.function?.name, args: JSON.parse(tc.function?.arguments ?? "{}") });
 
    if (parsed.tool === "getVmStatus") {
      const state = await getVmStatus(String(parsed.args?.vmName));
      const toolResult = JSON.stringify({ vmName: parsed.args?.vmName, state });
      history.push({ role: "assistant", content: "", }); // placeholder for the tool call
      history.push({ role: "assistant", content: toolResult, }); // return tool result
      // In a full implementation you would add a "tool" role message; libraries vary on format.
    } else if (parsed.tool === "finish") {
      console.log(parsed.args?.summary);
      return;
    }
  }
 
  console.log("Stopped after loop limit.");
}
 
main(process.argv.slice(2).join(" ") || "What's the status of VM web-01?");

Run it:

export OPENAI_API_KEY="set-me"
node agent.ts "Check the status of VM web-02 and summarize."

This tiny loop shows the heartbeat of an agent: plan, call a tool, observe, and close.

Agent vs. script vs. workflow engine

You don’t throw away scripts or Logic Apps. You pick the right tool for the job.

• Scripts (Bash, PowerShell, Python)

  • Strengths: deterministic, fast, cheap, easy to test. Great for idempotent tasks and strict SLAs.
  • Weaknesses: brittle with ambiguity, hard to generalize across varying inputs.

• Workflow engines (Logic Apps, Durable Functions)

  • Strengths: long-running orchestration, retries, compensation, durable timers, human approvals. Excellent for predictable business processes.
  • Weaknesses: you must enumerate paths upfront; logic balloons with complexity.

• Agents

  • Strengths: adaptive planning across tools, can navigate ambiguity, can draft artifacts (runbooks, queries, remediation plans), good at discovery and explanation.
  • Weaknesses: latency (LLM calls), non-determinism, cost per token (check the pricing page), more complex observability, and you must enforce guardrails.

A useful rule of thumb:

• If success means “execute these known steps fast and reliably,” choose scripts or workflows.
• If success means “figure out the right steps given messy inputs and iterate,” consider an agent—often paired with scripts/workflows for execution.

Patterns you’ll actually use

You’ll see the same three patterns in real deployments:

1. Tool-use pattern

   • The agent invokes callable tools with strict schemas.
   • Tools return small, structured results; the agent decides what’s next.
   • Good for: diagnostics, inventory, ticket creation.

2. Retrieval pattern

   • The agent retrieves relevant context (KB, past incidents, runbooks) on demand.
   • Think of it as “search + cite” to boost grounding and reduce hallucination.
   • Good for: drafting change plans, answering “how do we…” with your own docs.

3. Planning/validation loops

   • The agent proposes a plan, runs read-only checks, validates impact, and asks for approval before write operations.
   • Good for: change automation with human-in-the-loop.

Where Azure AI Foundry fits:

• Azure AI Foundry (the portal experience) centralizes model deployments, prompt assets, evaluations, and safety tooling.
• You’ll likely host your agent runtime in Azure Functions or Azure Container Apps, call your Azure OpenAI deployment (or other models) from there, and manage prompts/evals in AI Foundry. We’ll wire this up in Chapter 3.

Guardrails you actually need

• Principle of least privilege: agents should run under a Managed Identity with only the roles required for read-only diagnostics by default.
• Explicit tool list: the agent can only call what you expose (and document!) as a tool. No implicit shell access.
• Red-team your prompts and tools: deny-list dangerous actions; require “approval tokens” for anything write-path.
• Test with canary subscriptions/resource groups before prod.

Here’s a simple baseline to start with: a user-assigned managed identity and minimal RBAC to query metrics and read Key Vault secrets.

# Variables
RG=rg-agents-foundations
LOC=eastus
UAMI=umi-agent-reader
KV=kv-agents-foundations
SUBID=$(az account show --query id -o tsv)
 
# Resource group
az group create -n $RG -l $LOC
 
# User-assigned managed identity
az identity create -g $RG -n $UAMI
 
# Key Vault (for storing non-secret config pointers or API creds if you must;
# prefer Managed Identity wherever possible)
az keyvault create -g $RG -n $KV -l $LOC --enable-rbac-authorization true
 
UAMI_ID=$(az identity show -g $RG -n $UAMI --query id -o tsv)
UAMI_PRINCIPAL=$(az identity show -g $RG -n $UAMI --query principalId -o tsv)
 
# Minimal read roles
az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL \
  --assignee-principal-type ServicePrincipal \
  --role "Monitoring Reader" \
  --scope /subscriptions/$SUBID
 
az role assignment create \
  --assignee-object-id $UAMI_PRINCIPAL \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope $(az keyvault show -n $KV --query id -o tsv)

This gives an agent read access to metrics/logs and secrets retrieval, nothing more. Expand roles gradually and explicitly.

Observability from day one

Treat agents like services: logs, traces, metrics.

• Capture: prompt versions, tool inputs/outputs (redact secrets), model responses, latency, token usage, costs (from billing; check the pricing page), and error rates.
• Store: Application Insights + Log Analytics for queries and dashboards.
• Correlate: OpenTelemetry spans from your agent runtime through each tool call.

Quick Bicep to stand up a Log Analytics workspace and link Application Insights:

// obs.bicep
param location string = 'eastus'
param namePrefix string = 'agents'
 
resource la 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
  name: '${namePrefix}-law'
  location: location
  properties: {
    retentionInDays: 30
    features: {
      searchVersion: 1
    }
  }
}
 
resource appi 'Microsoft.Insights/components@2020-02-02' = {
  name: '${namePrefix}-appi'
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: la.id
    Flow_Type: 'Bluefield'
    Request_Source: 'rest'
  }
}
 
output appiConnectionString string = appi.properties.ConnectionString

Deploy:

az deployment group create \
  -g $RG \
  -f obs.bicep \
  -p location=$LOC namePrefix=agents

Wire your agent runtime to the Application Insights connection string and emit traces for each tool call. You’ll thank yourself later when you need to explain behavior.

Cloud-native fundamentals for agents

• Identity: Use Managed Identity everywhere you can—Functions, Container Apps, and Azure VM/VMSS all support it. Avoid long-lived secrets.
• Network boundaries: Agents often need outbound internet for model APIs and inbound/outbound to Azure services. Use private endpoints and VNET integration for sensitive tools; keep your model endpoint accessible per your risk posture.
• Config and secrets: Store pointers in App Configuration, secrets in Key Vault, and use RBAC + managed identity to fetch at runtime.
• Data minimization: Send only the minimum context to the model (token cost and privacy). Summarize or retrieve, don’t dump.

A simple local.settings.json equivalent for Functions becomes environment variables in Container Apps. For example, in Container Apps you might set:

• APPLICATIONINSIGHTS_CONNECTION_STRING
• AZURE_CLIENT_ID (for the user-assigned identity)
• AZURE_TENANT_ID
• AZURE_SUBSCRIPTION_ID

How this fits into modern Azure architectures

Think of the agent as another stateless service in your platform:

• It runs in Azure Functions or Azure Container Apps.
• It talks to your tools via APIs (often behind API Management) or directly with RBAC.
• It logs to Application Insights and emits traces.
• It uses Azure OpenAI (or other models) via your deployment configured in Azure AI Foundry.
• It never has blanket permissions; it’s scoped per environment and per tool.

A clean pattern is “agent runtime → API Management → tool backends,” where tools live in Functions or Container Apps with Managed Identity to your resources. That gives you quotas, headers, auth policies, and central observability.

Common pitfalls (and how to dodge them)

• Unbounded prompts: If the agent gets unlimited context, costs will creep and latency will spike. Set token limits and summarize aggressively.
• Silent failures: Without telemetry, you won’t know why a tool wasn’t chosen. Log all tool offers and selections with reason codes.
• Over-granting: Start read-only. Require an approval token or out-of-band confirmation before any write-path tool runs.
• Mixing retrieval and action: Keep retrieval tools (read) and action tools (write) separate with distinct approval flows.

A tiny “read-only” diagnostic tool contract

A predictable tool contract keeps agents honest. Describe tools with strict schemas and bounded outputs.

// tools.ts
export type ToolSpec = {
  name: string;
  description: string;
  input: Record<string, any>;
  handler: (args: any) => Promise<any>;
};
 
export const listAppInsightsFailures: ToolSpec = {
  name: "listAppInsightsFailures",
  description: "List top failing requests from Application Insights by operation name in the last N minutes.",
  input: { minutes: "number", top: "number" },
  handler: async ({ minutes, top }) => {
    // Here you'd call the Azure Monitor Query REST API with Managed Identity
    // For now, simulate a bounded result:
    return [
      { operationName: "GET /api/orders", failureRate: 0.12 },
      { operationName: "POST /api/checkout", failureRate: 0.09 },
    ].slice(0, top);
  },
};

You expose only this tool to the agent. The agent can’t invent new powers.

Cost and latency realities

• Every LLM call adds latency. Keep loops short, cache retrievals, and push heavy lifting into tools (e.g., Kusto does the aggregation, not the model).
• Don’t guess prices—model and token costs change. Check the pricing page for your model and region.
• Budget alerts: set Azure Cost Management budgets per resource group and tag your agent workloads so you can track spend.

What we’ll build next

Now that you have the mental model—policy + LLM + memory + tools wrapped in guardrails and observability—we’ll wire it to real infrastructure.

In Chapter 2, we’ll get hands-on with the Model Context Protocol (MCP): how MCP servers expose tools/resources, how to scope them safely with Managed Identity and API Management, and how to let an agent call those tools with full traces into Application Insights. We’ll set up the plumbing once and reuse it across the series.