Agent risk
Safety · topic safety/agent-risk
An agent that can act can also do harm, and a model can be talked into acting by text it reads. This topic teaches how to size what an agent can reach, grant it the least it needs, keep a human at the right decision points, and recognize the two classic attacks: prompt injection and data exfiltration.
Concepts
- Blast radius
- Everything an agent's action can reach and change: the files it can write, the accounts it can act as, the systems its credentials open, the people its messages reach. Blast radius is decided by what the agent is connected to, not by what you asked it to do. Sizing it before a task is the first step in deciding how much autonomy to allow. glossary
- Permissions and least privilege
- Giving an agent only the access the current task needs: read where it needs to read, write only in the working folder, no network or credentials unless required. Least privilege limits what a mistake or an attack can do. Convenient broad access is the usual failure; a separate account, a sandbox or a scoped token is the usual fix. glossary
- Human in the loop
- A required human decision at a chosen point in an agent's work, such as before sending, paying, deleting or deploying. The human sees what the agent proposes and approves, edits or stops it. Placing the checkpoint at the irreversible step, rather than everywhere, keeps the agent useful while keeping accountability with a person. glossary
- Prompt injection
- An attack where text the agent reads, such as a web page, email, document or code comment, contains instructions that the model follows as if they came from its user. The model cannot reliably tell data from commands. Any agent that reads untrusted content and can act is exposed, and the defense is limiting what it can do, not hoping it will refuse. glossary
- Exfiltration
- Data leaving a boundary it should not cross, often as the second half of an injection: the agent is tricked into putting secrets, private files or conversation contents into a URL, a message or a public location. Any tool that can send data outward is an exfiltration path. Review those tools first and restrict where an agent may send anything. glossary
Links
- Builds on: What an agent is, Verifying outputs
- Leads to: Quality with agents, Hooks, permissions, settings, Governance and oversight
- Competencies drawing on it: Verifies agent-written code before trusting it, Connects an agent to tools and data safely, Judges the risk of letting an agent act
Lessons
- Why agent safety is different (explanation)
- Sizing an agent's blast radius (tutorial)
- Tracing a planted instruction to the tool that leaks (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
Why agent safety is different
Unlocks when you finish Why agent safety is different.
Takeaways
- An assistant's mistake is a wrong answer, but an agent's mistake is a wrong action that a tool carries out without skepticism.
- Blast radius is what an action can reach and change with the access it has, not what you asked for. Say it out loud in its uncomfortable form.
- Least privilege means granting only what this task needs, and taking it back afterwards, because access granted for one job is available to every job.
- Put a human approval in front of steps that can't be undone or that reach other people. Let the agent read, search, draft, and propose freely.
- Everything an agent reads is text it may treat as an instruction. Prompt injection and exfiltration are why you need these habits even when you trust the agent.
Example
Go through this week's scheduling emails and reply to each one proposing a slot from my calendar.
I found four scheduling emails and drafted a reply to each with a free slot. The drafts are in your outbox. Two go to people outside the company; do you want to review those before I send any?
Sizing an agent's blast radius
Unlocks when you finish Sizing an agent's blast radius.
Takeaways
- Write an agent's access as read, write and send rows, each with a place, and ask of each row who it reaches and whether it can be undone.
- The tools and places an agent is connected to set its blast radius, whatever the request says.
- Cross out every row the task doesn't need and narrow the rest to the place the task is about. The agent does the same task, and a wrong step reaches fewer people.
- Put the approval right before the step that can't be undone or that reaches other people, and show the approver the evidence to judge it, such as the files themselves or a count with samples.
Example
Which rows can't be undone? · open in the lesson
The program ends its report with the rows whose actions can't be undone. Type their numbers from lowest to highest, separated by spaces.
Prints: 3 5 6
Tracing a planted instruction to the tool that leaks
Unlocks when you finish Tracing a planted instruction to the tool that leaks.
Takeaways
- Everything the agent reads is text it may follow, and a planted line can only spend the tools the agent has.
- The attack is a chain: read the page, have private data nearby, send it out. The last link is the one a summarizing task never needs.
- Fetching an address is a send, because the agent chooses the text of the address. Writing a local file stays on the computer.
- Telling the agent to ignore the page changes the odds and leaves the path. Remove or gate the outbound tool, and review the tools that can send before anything else.
Example
Predict the three verdicts · open in the lesson
The program prints one data leaves: line per permission set, in the order above. Type the three values, yes or no, separated by spaces.
Prints: no no yes
Sources
AEC-10Guardrails and safety: why agent safety differs, defense layers, injection, human in the loop, Agent Engineer Course (course)Academy introduction-to-claude-coworkIntroduction to Claude Cowork, Claude Academy (course)