Chapter 4 — When an agent beats a script (and when it doesn’t)

Last time you and I stood up a production-ready agent in Azure AI Foundry, wired to real tools via MCP, and wrapped with API Management, Managed Identity, and Application Insights. Now that you’ve got something real, the next practical question is: when should you actually ship an agent into production—and when is a plain script the better call?

The decision matrix: ambiguity, variability, coupling, latency, and SLA

When I decide whether to build an agent or a script, I walk through five axes. If three or more lean “agent,” I’ll prototype it as an agent; if three or more lean “script,” I automate it deterministically.

Ambiguity of input and goals
- Script: Inputs are exact and validated; outputs are binary-success. Examples: rotate a secret, reimage a VM.
- Agent: Inputs are messy; goals need interpretation or judgement. Examples: triage a ticket description, choose between multiple runbooks, draft a remediation message.
Variability of environment
- Script: Your target systems are uniform; APIs are stable; one tool is enough.
- Agent: Multiple tools might be needed; APIs differ across teams/tenants; the path varies.
Coupling across systems
- Script: You operate within one domain (e.g., Azure Resource Manager only) with fixed parameters.
- Agent: You must touch ServiceNow/Jira, Azure Resource Graph, Kusto, and a runbook engine, then notify a human.
Latency tolerance and interactivity
- Script: Sub-second to a few seconds; no human in the loop.
- Agent: Tens of seconds to minutes; human approvals or clarifications are normal.
SLA and blast radius
- Script: Strict uptime/latency SLOs and well-bounded side effects.
- Agent: Best-effort or human-in-the-loop gating; errors can be contained with guardrails.

Quick rule of thumb:

If success is a truth table, write a script.
If success is a policy and judgment call, ship an agent—with guardrails.

Case studies

Case 1: Ticket triage and runbook orchestration (agent wins)

The problem: Overnight, your on-call queue fills with 50 alerts. Some are flaky, others are real. You want to cluster them, pull context, and propose actions.

Why an agent?

Ambiguity: Alert descriptions vary wildly.
Multi-tool path: Query Azure Monitor (Kusto), check recent deploys (DevOps/GitHub), look up “what changed” (Resource Graph), and then pick a runbook.
Human-in-the-loop: On-call approves before execution.

A concrete pattern:

MCP server exposes:
- read-only Kusto queries,
- a “safe” runbook trigger endpoint (dry-run by default),
- a messaging tool (post to Teams).
The agent:
- summarizes tickets,
- de-duplicates by resource and time window,
- proposes a runbook with parameters,
- requests approval,
- executes with dry-run, then real run if approved.

Here’s a minimal MCP tool allow-list you can keep in your repo:

# mcp-tools.yaml
version: 1
tools:
  - name: query_kusto
    allow:
      clusters: ["log-analytics-01"]
      databases: ["InsightsMetrics","AzureActivity"]
      maxRows: 5000
      timeoutSeconds: 30
      readonly: true
  - name: trigger_runbook
    allow:
      account: "automation-shared"
      runbooks:
        - "restart-appservice"
        - "purge-frontdoor-cache"
      parameters:
        maxDurationMinutes: 10
      dryRunDefault: true
      requireApproval: true
  - name: post_teams_message
    allow:
      channels: ["oncall-bridge","sre-triage"]
      maxLength: 5000

The “dryRunDefault” and “requireApproval” flags are your seatbelts.

Case 2: Deterministic remediation jobs (script wins)

The problem: Rotate a Key Vault secret nightly, update two app settings, and warm up the app.

Why a script?

Exact, stable inputs.
Predictable side effects.
Tight latency and high reliability.

A real Azure Functions timer job (PowerShell or Python) or a GitHub Actions scheduled workflow is a better fit. Here’s a GitHub Actions example with Azure login and idempotent steps:

# .github/workflows/kv-rotate.yaml
name: kv-rotate
on:
  schedule:
    - cron: "0 2 * * *" # 02:00 UTC daily
  workflow_dispatch: {}
 
jobs:
  rotate:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Azure login (OIDC)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
 
      - name: Rotate secret
        run: |
          az keyvault secret set --vault-name kv-shared \
            --name app--db-password \
            --value $(openssl rand -base64 32)
 
      - name: Update App Service settings
        run: |
          az webapp config appsettings set \
            --resource-group rg-app \
            --name app-prod \
            --settings DB_PASSWORD=@Microsoft.KeyVault(SecretUri=https://kv-shared.vault.azure.net/secrets/app--db-password)
 
      - name: Warm up
        run: |
          curl -sSf https://app.example.com/healthz > /dev/null

No LLM required. Your failure modes are well-known and easy to alert on.

Guardrails and fallbacks: tool allow-lists, dry-run, approvals, safe aborts

When I ship an agent, I assume it will make mistakes in novel ways. Guardrails turn those mistakes into harmless no-ops.

Tool allow-lists and parameter schemas: The agent can only call approved tools with whitelisted parameters.
Dry-run modes: Every impactful tool supports “plan” or “what-if.”
Human approvals: Required for state-changing actions in prod.
Safe aborts: Timeouts, circuit breakers, and “stop after N tool calls” to prevent thrashing.

A practical APIM policy snippet that forces dry-run unless an approval token is present:

<!-- api-management/policies/require-approval.xml -->
<policies>
  <inbound>
    <base />
    <set-variable name="approved" value="@(context.Request.Headers.GetValueOrDefault("X-Approval-Token",""))" />
    <choose>
      <when condition="@(!context.Request.Method.Equals("GET") && string.IsNullOrEmpty((string)context.Variables["approved"]))">
        <set-query-parameter name="dryRun" exists-action="override">
          <value>true</value>
        </set-query-parameter>
      </when>
    </choose>
    <rate-limit calls="60" renewal-period="60" />
    <set-backend-service base-url="https://agent-backend.azurecontainerapps.io" />
  </inbound>
  <backend>
    <base />
    <retry condition="@(context.Response.StatusCode==429 || context.Response.StatusCode==503)" count="2" interval="2" />
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <base />
  </on-error>
</policies>

And a TypeScript circuit breaker in a simple MCP server:

// mcp-server/src/guardrails/circuitBreaker.ts
import Bottleneck from "bottleneck";
 
export class CircuitBreaker {
  private limiter: Bottleneck;
  private failures = 0;
  private openUntil = 0;
 
  constructor(private maxCalls = 20, private maxFailures = 5, private coolOffMs = 60_000) {
    this.limiter = new Bottleneck({ maxConcurrent: 1, minTime: 100 });
  }
 
  async exec<T>(fn: () => Promise<T>): Promise<T> {
    const now = Date.now();
    if (now < this.openUntil) {
      throw new Error("Circuit open: cool-off in progress");
    }
    return this.limiter.schedule(async () => {
      try {
        const res = await fn();
        this.failures = 0;
        return res;
      } catch (e) {
        this.failures += 1;
        if (this.failures >= this.maxFailures) {
          this.openUntil = Date.now() + this.coolOffMs;
        }
        throw e;
      }
    });
  }
}

Finally, define a “max tool calls” budget for an agent run in Azure AI Foundry:

{
  "runtime_config": {
    "tool_choice": "auto",
    "max_tool_calls": 12,
    "timeout_seconds": 90
  }
}

Testing and evaluation: prompt regression, offline evals, canaries, rollback

Agents are software, but the failure modes are new. I use four layers of assurance.

Prompt regression tests
- Snapshot tricky inputs and expected tool decisions.
- Run offline against the same model version before merging.
Offline evals
- Feed synthetic or historical incidents.
- Score outcomes: tool choices, safety rules honored, and human feedback.
Canary rollouts
- Shadow mode first (agent proposes, human executes).
- Then 5–10% of traffic in a non-critical segment.
- Promote progressively with automated rollback.
Fast rollback
- Model versions and prompts are configuration-managed.
- One command re-pins to the previous safe version.

A simple offline eval harness in TypeScript:

// evals/run-evals.ts
import fs from "node:fs/promises";
import { strict as assert } from "assert";
import { runAgent } from "../src/agent";
 
type EvalCase = {
  name: string;
  input: string;
  expect: { tool?: string; requiresApproval?: boolean };
};
 
async function main() {
  const cases: EvalCase[] = JSON.parse(await fs.readFile("evals/cases.json","utf8"));
  let pass = 0;
  for (const c of cases) {
    const result = await runAgent(c.input, { dryRun: true });
    const chosenTool = result.toolCalls[0]?.name ?? "none";
    const requiresApproval = result.policy?.requiresApproval ?? false;
    try {
      if (c.expect.tool) assert.equal(chosenTool, c.expect.tool);
      if (typeof c.expect.requiresApproval === "boolean") {
        assert.equal(requiresApproval, c.expect.requiresApproval);
      }
      pass++;
      console.log(`✔ ${c.name}`);
    } catch (e) {
      console.error(`✘ ${c.name} -> tool=${chosenTool}, approval=${requiresApproval}`);
      process.exitCode = 1;
    }
  }
  console.log(`Passed ${pass}/${cases.length}`);
}
 
main().catch((e) => { console.error(e); process.exit(1); });

An example eval case:

[
  {
    "name": "P1 CPU spike, restart app only with approval",
    "input": "High CPU on app-prod-westus. Restart recommended?",
    "expect": { "tool": "trigger_runbook", "requiresApproval": true }
  },
  {
    "name": "Noise: transient 429s",
    "input": "Multiple 429s for GET /metrics",
    "expect": { "tool": "post_teams_message", "requiresApproval": false }
  }
]

A canary deployment using Azure Container Apps revisions (traffic-splitting):

# Split 10% traffic to new agent revision
az containerapp ingress traffic set \
  --name agent-backend \
  --resource-group rg-agents \
  --revision-weight latest=10 \
  --revision-weight previous=90

And a fast rollback:

# Route 100% back to previous revision
az containerapp ingress traffic set \
  --name agent-backend \
  --resource-group rg-agents \
  --revision-weight previous=100

Operating in production: incident response, observability, ongoing tuning

Observability you actually need

Per-run trace with:
- tools called (name, args, duration, status),
- token usage,
- decisions taken (policy gates triggered),
- final outcome.
Aggregates:
- success/error rates by tool,
- time-to-approval,
- dry-run vs real-run counts,
- top prompts causing fallbacks.

Instrument with OpenTelemetry and ship to Application Insights / Azure Monitor.

Here’s a minimal Node.js OpenTelemetry setup for tool calls:

// src/telemetry.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
 
const exporter = new OTLPTraceExporter({
  url: process.env.OTLP_ENDPOINT // e.g., Azure Monitor OTLP ingestion endpoint
});
 
export const sdk = new NodeSDK({
  traceExporter: exporter,
  serviceName: "mcp-agent-backend"
});
 
export async function startTelemetry() {
  await sdk.start();
  process.on("SIGTERM", async () => { await sdk.shutdown(); process.exit(0); });
}

Log tool invocations with span attributes:

// src/tools/runbook.ts
import { context, trace } from "@opentelemetry/api";
 
export async function triggerRunbook(name: string, params: Record<string,unknown>, dryRun: boolean) {
  const tracer = trace.getTracer("tools");
  return await tracer.startActiveSpan("trigger_runbook", async (span) => {
    span.setAttribute("tool.name", name);
    span.setAttribute("tool.dryRun", dryRun);
    try {
      // ... call backend
      const result = { status: "ok" };
      span.setStatus({ code: 1, message: "success" });
      return result;
    } catch (e: any) {
      span.setStatus({ code: 2, message: e.message });
      throw e;
    } finally {
      span.end();
    }
  });
}

Create an Azure Monitor Workbook for agent health:

Filters: operationName = “trigger_runbook” or “query_kusto”
Charts: error rate over time, P50/P95 tool latency, approvals pending, circuit breaker opens.

Incident response playbook

When the agent misbehaves, I follow this:

Contain
- Flip APIM header injection to force dry-run globally.

# Swap a named APIM policy that enforces dryRun=true for all POST/PUT/DELETE
az apim api operation policy update \
  --resource-group rg-agents \
  --service-name apim-agents \
  --api-id agent-backend \
  --operation-id "*" \
  --xml-content @api-management/policies/emergency-dryrun.xml

Roll back
- Route 100% of traffic to previous Container Apps revision (command above).
Diagnose
- Pull traces for the failing run IDs.
- Reproduce in a non-prod environment with the same inputs (from trace logs).
Fix and verify
- Add or tighten a allow-list rule.
- Add a new eval case that would have caught this.
- Promote through canary again.

Ongoing tuning

Prompt diffs go through code review and offline evals.
Tool contracts evolve—but never loosen silently. If you must expand parameters, default to stricter behavior and add tests.
Rotate models conservatively. Pin model versions for stability; when you upgrade, treat it like any other production change with canaries and rollback.

Concrete configuration examples

A Bicep snippet to stand up a user-assigned managed identity and grant it least-privilege access to an Automation Account runbook and Log Analytics read:

param location string = resourceGroup().location
param miName string = 'mi-agent-backend'
param automationAccountName string
param logAnalyticsWorkspaceId string
 
resource mi 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: miName
  location: location
}
 
resource aa 'Microsoft.Automation/automationAccounts@2023-05-15' existing = {
  name: automationAccountName
}
 
resource la 'Microsoft.OperationalInsights/workspaces@2022-10-01' existing = {
  scope: resourceId('Microsoft.OperationalInsights/workspaces', split(logAnalyticsWorkspaceId, '/')[8])
  name: split(logAnalyticsWorkspaceId, '/')[10]
}
 
resource roleAssignmentRunbook 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(aa.id, mi.id, 'AutomationJobOperator')
  scope: aa
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '9a2a2b0f-9a5d-4b35-9c86-85c2162c22a3') // Automation Job Operator
    principalId: mi.properties.principalId
    principalType: 'ServicePrincipal'
  }
}
 
resource roleAssignmentLAReader 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(la.id, mi.id, 'LogAnalyticsReader')
  scope: la
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '73c42c96-874c-492b-b04d-ab87d138a893') // Log Analytics Reader
    principalId: mi.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

A minimal bash probe to validate your agent endpoint in shadow mode:

#!/usr/bin/env bash
set -euo pipefail
 
ENDPOINT="https://agent-backend.azurecontainerapps.io/api/agent"
AUTH="Bearer $(az account get-access-token --resource api://agent --query accessToken -o tsv)"
 
payload='{
  "messages": [{"role":"user","content":"High CPU on app-prod-westus. What should we do?"}],
  "shadow": true,
  "runtime_config": {"max_tool_calls": 6, "timeout_seconds": 60}
}'
 
curl -sS -H "Authorization: $AUTH" \
  -H "Content-Type: application/json" \
  -X POST "$ENDPOINT" \
  -d "$payload" | jq .

This runs the agent in “shadow” (propose-only) so you can compare recommendations with human actions.

Cost and performance realities

Agents cost more to run than scripts. You’re paying for LLM tokens plus tool latencies and retries. Check the pricing page for your selected model and monitor token usage in your telemetry. If you need strict latency, keep an eye on end-to-end P95 and use caching for expensive queries. When you find a stable path that the agent always takes, consider carving that branch out into a deterministic microservice or function to reduce cost and variance.

A pragmatic rollout pattern

Start with a script baseline for the happy path.
Add an agent in shadow mode to propose actions and draft messages.
Gate real actions behind human approval and dry-run default.
Gradually delegate low-risk actions (dev/test, then staging, then low-priority prod).
Periodically distill “always the same” branches back into scripts.

This gives you the best of both worlds: agility where judgment matters, and reliability where it doesn’t.

Final series recap

Fundamentals (Chapter 1): Agents are a policy-driven loop combining an LLM, memory, and tools. They’re not magic microservices; they’re decision engines.
MCP in practice (Chapter 2): MCP gives you a clean, auditable contract between the agent and your infrastructure. You expose tools with guardrails and schemas.
Build guide (Chapter 3): You assembled a working agent in Azure AI Foundry, fronted by API Management, with Managed Identity and Application Insights.
Decision framework (this chapter): You now have a matrix to decide when an agent beats a script, with concrete guardrails, testing patterns, and operational playbooks.

If you remember one thing: use agents where ambiguity and multi-tool reasoning dominate—and scripts where determinism and tight SLAs rule. Start safe with allow-lists, dry-runs, and approvals; test with offline evals and canaries; instrument everything; and keep carving stable paths back into scripts. That’s how you make agents practical in real Azure environments.