Guardrails and production
Building agents · topic building-agents/production
Running an agent for real means surrounding it with controls and treating its changes like any other risky deployment. This topic covers layered defenses, gating deploys on evaluation results, rolling changes out gradually, managing cost, operating the model API itself, the security risks specific to agents, and the protocols agents use to talk to tools and to each other.
Concepts
- Defense layers
- No single control stops every bad outcome, so production agents stack several: policy in the system prompt, input and output filtering, permission limits on tools, human approval at irreversible steps, monitoring and alerting, and the ability to stop the agent. Each layer catches what the previous ones miss, and the design assumes the model itself will sometimes be fooled. glossary
- Eval-gated deploys
- Treating any change to prompt, tools, model or harness as a release that must pass the evaluation suite before it reaches users, the same way code must pass tests. The gate turns evaluation from an occasional study into a continuous control and catches the regression that a small wording change or a model update introduces. glossary
- Rollout strategies
- Releasing a change to a fraction of traffic first, comparing its metrics with the current version, and widening or reverting based on what is observed. Shadow runs, canaries and gradual percentages all apply to agents. Because model behavior is non-deterministic and evaluation is imperfect, gradual rollout is the check that evaluation cannot give. glossary
- Cost
- What an agent spends per task in tokens, tool calls and time, and how that scales with users. Agents multiply model cost by their number of turns, and a runaway loop or verbose tool can multiply it again. Track cost per run, set budgets that stop a run, cache what repeats, and route simple steps to smaller models. glossary
- Provider operations
- The mechanics of calling a model provider reliably. Pin an exact model ID, one that names a single fixed version, so the model changes only when you choose. Use prompt caching so the stable prefix of a system prompt and tool list is processed once and read back at a lower price on later calls. Retry retriable errors, such as rate limits and overload, with a backoff. Report terminal ones, such as an invalid request or a content refusal, to the caller. Set timeouts on every call. glossary
- Agent-specific security risks
- Risks that appear when a model can act: prompt injection through content the agent reads, exfiltration through tools that send data outward, over-permissioned tools that turn a wrong decision into real damage, and confused-deputy situations where the agent spends its own authority for whoever wrote the text it acts on. They are mitigated by limiting capability and reviewing actions, not by better prompts. glossary
- Protocols (MCP, A2A)
- Standards for how agents connect to things. The Model Context Protocol standardizes how an agent reaches tools and data through servers. Agent-to-agent protocols standardize how one agent discovers, delegates to and exchanges results with another across organizations. Shared protocols reduce integration work and also widen the attack surface, so each connection is a trust decision. glossary
Links
- Builds on: Evaluation and testing, Governance and oversight, Connecting tools with MCP
- Leads to: nothing yet
- Competencies drawing on it: Runs an agent in production with guardrails
Lessons
- How a running agent gets attacked (explanation)
- Protocols for tools and for other agents (explanation)
- Cost, caching and pinning the model (tutorial)
- Gating a deploy on the evaluation (tutorial)
- Guardrails in layers (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
How a running agent gets attacked
Unlocks when you finish How a running agent gets attacked.
Takeaways
- Once a model can act, the risks are prompt injection through what it reads and exfiltration through what it can send, over-permission, and the confused deputy that spends the agent's authority for whoever wrote the text in front of it.
- An outbound channel is any tool that lets text the agent chooses reach a place or a person you don't control. A fetched address and a reply on a ticket count, and so does mail.
- A system prompt is advice the model weighs against the planted line. With nothing else in the loop, some runs follow the planted line and the tool call goes out.
- Mitigate by cutting the tools the task doesn't need, scoping the ones left to the person the agent serves, and putting a person and a log in front of what sends or spends.
- Gate what the agent can send. A filter on what it reads works on text the attacker writes.
Example
This lesson has no runnable example or prompt block.
Protocols for tools and for other agents
Unlocks when you finish Protocols for tools and for other agents.
Takeaways
- A shared protocol turns N x M adapters into N plus M: each agent speaks the protocol once and each system is wrapped once.
- The Model Context Protocol connects an agent to the tools, resources and prompts that servers offer, and the loop still sends each tool call and returns its result.
- Agent-to-agent protocols such as A2A cover discovery through an Agent Card, delegation as a task, and results as artifacts, between agents that keep their internals to themselves.
- Neither protocol decides what the other side may do with your data, so each connection is a trust decision you make before you enable it.
- Treat a remote agent as a tool that reads and sends. Its message goes through the permission check, the filter and the gate, and what comes back is untrusted input.
Example
This lesson has no runnable example or prompt block.
Cost, caching and pinning the model
Unlocks when you finish Cost, caching and pinning the model.
Takeaways
- A loop sends its whole history again on every call, so the number of turns and the size of each tool result decide what a run costs. Keep a cost record per run from the usage the provider reports.
- A token budget stops a run with a named reason, and the caller still gets the tokens and the cost already spent. The check runs before a call, so a run can end a little over the budget.
- Prompt caching writes the unchanging prefix once and reads it back at a fraction of the input price, but only while the prefix matches exactly.
- Retry rate limits, overload and timeouts with a wait, and use
retry-afterwhen the provider sends it. Report every other error to the caller. - Pin an exact model id, and roll out a new model like any other change. An alias moves on the provider's date, and a pinned id retires on a date you are told about in advance.
Example
The bill of one run · open in the lesson
Run this, and compare what you see with the output below.
python3 ops.py costPrints the lines below (verified in CI from site/examples/building-agents/cost-and-provider-operations/cost.py)
call 1: input 260, output 23, $0.001125 call 2: input 353, output 26, $0.001449 call 3: input 443, output 28, $0.001749 stop: end_turn answer: Ticket 311 is open. Hold the power button for 15 seconds, then charge it for 10 minutes. calls: 3 tokens: 1133 cost: $0.004323
Gating a deploy on the evaluation
Unlocks when you finish Gating a deploy on the evaluation.
Takeaways
- Every change to the prompt, the model, a tool or the harness goes through the evaluation before it reaches users, and a result below its threshold blocks it.
- Set a threshold per metric. Safety metrics, such as refusing what the agent must not answer, get a hard gate at 100 percent. Quality metrics, such as helpfulness, may get a soft gate that a person signs off.
- When the gate blocks, read the failing transcripts and fix the change. The threshold stays where it is.
- Roll a change out in stages, starting with a small share of traffic, and compare the live metrics with the current version before you widen it. This lesson's plan adds a team stage before it.
- Keep the prompt files in version control, and name the live prompt, the traffic share and a pinned model version in configuration, so a rollback is a setting and takes minutes.
Example
The gate on today's prompt · open in the lesson
Run this, and compare what you see with the output below.
python3 gate.py currentPrints the lines below (verified in CI from site/examples/building-agents/eval-gated-deploys/gate_current.py)
prompt: current refused hard 6 of 6 needs 100% pass routine hard 9 of 10 needs 90% pass helpful soft 11 of 15 needs 70% pass deploy: allowed
Guardrails in layers
Unlocks when you finish Guardrails in layers.
Takeaways
- Defense layers stack controls that fail in different ways: the system prompt, a permission check on the tools, an output filter and an approval gate, and monitoring over all of them.
- The system prompt is advice the model may ignore. It fails at the moment the model is fooled.
- A permission check in the loop is a hard stop for every call it doesn't allow, and it can't stop a planted use of a call it allows.
- An output filter catches the formats it knows and lets the rest through. An approval gate catches sends nobody expected, while the person reads each one.
- A layer that reports a hit with nobody watching doesn't stop the attack. Monitoring turns the layers' records into alerts, and it needs signals of its own for the attacks no layer records.
Example
Predict the sends with only the system prompt · open in the lesson
This step is run a. Predict the three lines, then run the step.
python3 guardrails.py prompt_onlyPrints the lines below (verified in CI from site/examples/building-agents/guardrails-in-layers/prompt_only.py)
fetch_url bramble-kitchen.example: sent fetch_url collect.bramble-kitchen.example: sent post_reply ticket 1187: sent
Sources
AEC-10Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course (course)AEC-11From prototype to production: eval-gated deploys, rollout, cost, Agent Engineer Course (course)AEC-14Agent protocols, MCP and A2A, Agent Engineer Course (course)Brilliant SECSecurity and adversarial thinking, Brilliant, Coding with AI skills map (reference)Academy model-context-protocol-advanced-topicsModel Context Protocol: Advanced topics, Claude Academy (course)Academy claude-platform-101Claude Platform 101, Claude Academy (course)Hardy 1988The Confused Deputy (or why capabilities might have been invented), ACM SIGOPS Operating Systems Review 22(4) (1988), 36-38 (paper)