Judges the risk of letting an agent act

Safety · competency safety/judges-agent-risk

Taught in: the Safety course

Draws on: Agent risk, Governance and oversight

Learning objectives

Names what an agent action can reach and break (base)

Claim	Why	Example
Before letting an agent act, the learner lists everything the agent's tools and credentials can reach, beyond what the task needs.	An agent acts with everything it has been given, and a wrong step lands wherever that access extends.	Asked to let an agent tidy one folder, the learner notices it has been given write access to the whole shared drive and narrows that first.
The learner separates actions that can be undone from actions that cannot.	A reversible mistake costs a retry. An irreversible one costs whatever was sent, deleted or paid.	Drafting an email is reversible, sending it is not, so the learner lets the agent draft and keeps sending for themselves.
The learner names who else is affected when the agent gets it wrong.	Blast radius is measured in people and systems, and the size of the task says nothing about it.	A script that renames files in a personal folder affects one person. The same script on a shared calendar affects everyone invited, and the learner treats them differently.

Served by: Why agent safety is different, Assessing the risk of a use case before it starts, Sizing an agent's blast radius

Chooses where a human must approve (base)

Claim	Why	Example
Approval is placed right before the irreversible or externally visible step, and a plan approval at the start does not replace it.	Approving the plan up front does not catch the mistake the agent makes halfway through.	The agent may read the ticket, draft the reply and propose a label, but a person clicks send and closes the ticket.
A step that is approved without being read is not a control, so the learner keeps the number of approvals small enough to read.	Twenty prompts a minute trains a person to click yes, which is the same as no approval with extra cost.	Rather than approving every file edit, the learner approves once at the diff, where all edits can be read together.
The learner can say what evidence an approver needs at each check-in point.	An approver without the right information in front of them is guessing.	At the "send invoices" step the approver sees the count, the total and three sample rows instead of a bare "proceed?" prompt.

Served by: Why agent safety is different, Sizing an agent's blast radius

Recognizes prompt injection and data exfiltration paths (base)

Claim	Why	Example
The learner treats everything an agent reads (web pages, emails, files, calendar entries) as text the model may follow as an instruction.	A model has one stream of text and no separate channel for "orders from my user", so a planted sentence looks like any other instruction.	Before pointing an agent at a bookmarked page, the learner asks what the page could tell the agent to do with the access it has.
Given an agent's permissions, the learner can name the path a planted instruction would use to get data out.	Exfiltration needs only two ordinary abilities, read and send, and the attack is the combination of the two.	An agent that can read mail and fetch URLs can be told to put a password into a URL, and the learner spots that the fetch is the way out.
The learner shrinks or gates the outbound ability rather than trying to filter what the agent reads.	You cannot vet every input, but an agent that cannot send, or must ask before sending, cannot carry data out unnoticed.	For a summarizing task the learner removes the agent's mail and network-post tools, leaving read-only access.

Served by: Why agent safety is different, Tracing a planted instruction to the tool that leaks

Sets policy, logging and escalation for agents in an organization (expert)

Claim	Why	Example
The learner writes a policy that says which tasks an agent may do, with what data and which approvals, in terms a team can apply.	A policy that says "use AI responsibly" decides nothing, and every team then decides for itself.	"Agents may open pull requests in any repository; merging requires a human review; production credentials are never given to an agent."
Agent actions are logged where a person can read them later, and someone is named to read them.	Without a log there is no way to learn what happened after a problem, and a log nobody reads catches nothing.	Every agent-sent message carries a tag, the tagged messages go to an audit channel, and the team lead reviews a sample weekly.
The policy names the escalation path: what stops the agent, who is told, and who decides whether it restarts.	The first incident is the wrong time to find out who can revoke the agent's access.	"Any agent action on customer data outside office hours pages the on-call, who can revoke the token."
The learner treats a model or tool update as a change that needs a re-check of the agent's behavior.	The same prompt on a new model version can act differently, and a policy that assumed the old behavior no longer holds.	After the vendor's model upgrade, the learner reruns the agent on last month's tasks before letting it back on live work.

Served by: Assessing the risk of a use case before it starts, Introduction to the EU AI Act, Reading agent logs and rerunning after a model change, Writing an AI policy a team can apply

Alignment

Framework	Code	Asks	Objectives here
AI Fluency 4D (Dakan and Feller)	Diligence	Use AI responsibly, transparently and with accountability for the result	names-blast-radius, chooses-human-in-loop