Verifying agent work
Coding with agents · topic coding-with-agents/verification
Code you did not write needs a different kind of reading. This topic covers reviewing an agent's diff against the specification rather than for plausibility, observing the system as it runs, narrowing a fault systematically, turning checks into loops the agent can run itself within limits you set, relying on deterministic gates over model opinions, and attacking your own agent before someone else does.
Concepts
- Reviewing code you did not write
- Reading an agent's diff as a reviewer, not an author: you have no memory of why each line exists, so every line has to justify itself. Look for changes outside the brief, silent behavior changes, deleted tests, copied patterns applied wrongly and comments that describe intent rather than what the code does. Read the code, not the agent's summary of it. glossary
- Verifying against the specification
- Checking the result against the success criteria written before the work, item by item, instead of judging whether the change looks reasonable. This catches the plausible-but-wrong result, the criterion quietly dropped and the edge case skipped. If a criterion cannot be checked, either the spec or the verification design needs fixing before the change is accepted. glossary
- Observing a running system
- Running the changed software and watching what it actually does: the logs it writes, the requests it makes, the state it leaves behind, the screen it renders. Tests prove what they cover; observation catches what nobody thought to test. For agent work, running it yourself is also the check that the agent's report of "tests pass" is true. glossary
- Isolating a fault
- Narrowing a failure to its cause by systematic steps: reproduce it, shrink the input, bisect the change, add observation at the boundary, confirm the hypothesis before fixing. Agents can help at each step but will also guess fixes; insisting on a reproduced, explained cause before any patch is what keeps the fix from becoming a new bug. glossary
- Bounded self-checking loops
- Giving the agent a check it can run itself, such as a test suite, a linter or a build, and letting it iterate until the check passes, with a limit on attempts and a rule for what it may change. The bound matters: an unbounded loop will eventually satisfy the check by weakening it. The human verifies the check is still honest at the end. glossary
- Deterministic gates
- Checks whose outcome does not depend on a model's judgment: tests, type checks, linters, builds and schema validation. They give the same answer for the same input, so they can block a merge. A review by another model adds a second opinion and finds things a gate cannot, but it can be wrong or be persuaded in the same way as the first model, so it never replaces the gate. glossary
- Red-teaming your own agent
- Deliberately trying to make your own agent misbehave before relying on it: feeding it a file with hidden instructions, pasting it a web page you did not read, asking for something its permissions should stop, giving it an ambiguous brief and seeing how far it runs. The aim is to learn what it does under pressure, in a sandbox where nothing real can be damaged. What you find becomes a permission change, an instruction or a gate. glossary
Links
- Builds on: Plan, implement, verify, Verifying outputs
- Leads to: Evaluation and testing, Quality with agents
- Competencies drawing on it: Verifies agent-written code before trusting it
Lessons
- Reproducing a fault before the agent fixes it (tutorial)
- Attacking your own agent before someone else does (tutorial)
- Reviewing a diff you did not write, against the spec (tutorial)
- Giving the agent a check it can run, with a limit (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
Reproducing a fault before the agent fixes it
Unlocks when you finish Reproducing a fault before the agent fixes it.
Takeaways
- Reproduce the fault before anyone changes code. Rerun the failing input and write the steps down, and "sometimes fails" becomes a precise condition and the test the fix has to pass.
- Observing a running system means reading what it recorded, the log, the output and the exit status, next to a run that worked. The log shows how far the failing run got, and a theory about a step the run completed is ruled out.
- Isolating a fault means halving the suspects: one change, one observation, and a note of what was ruled out. The same halving over commits is
git bisect. - Brief the agent with the reproduced, explained cause and a done-criterion from the reproduction, and verify the fix by running those steps yourself.
Example
Run each night three times · open in the lesson
In a clone of the course repository, go to the fixture directory, site/examples/coding-with-agents/observing-and-debugging/, and run the reproduction script there. The scripts in this lesson all run from that directory.
python3 reproduce.pyPrints the lines below (verified in CI from site/examples/coding-with-agents/observing-and-debugging/reproduce.py)
2026-09-14: ok ok ok (total: 22975.95 EUR) 2026-09-15: rejected rejected rejected 2026-09-16: ok ok ok (total: 23257.33 EUR) 2026-09-17: rejected rejected rejected
Attacking your own agent before someone else does
Unlocks when you finish Attacking your own agent before someone else does.
Takeaways
- Red-teaming your own agent means planting hidden instructions, asking for what its permissions should stop and giving it a vague brief, in a scratch copy where a successful attack costs nothing.
- Contain the run before you start: a fresh copy, the agent started inside it, and settings that block the network and writes outside the copy. In Claude Code that's Manual mode, deny rules for the web tools,
curlandwget, and the sandbox with a strict network allowlist and no retry outside the sandbox. - Screen what the agent did by the evidence. Compare the files with a record from before and search them for a canary. For what happened outside the folder, read the prompts you answered and the transcript.
- A deny rule for a command matches the command as written, so an agent that gets a No can try another route. Record the retry as a finding, and rely on a limit the operating system enforces for the network.
- Turn each finding into a permission change, a line in the instructions or a gate, by asking what would have stopped it.
Example
Make the scratch copy and check it · open in the lesson
Run these two commands and compare what you see with the output below. The second command is the check you run again after the agent's session.
python3 make_scratch.py ~/red-teampython3 check_run.py ~/red-team; echo "exit status $?"Prints the lines below (verified in CI from site/examples/coding-with-agents/red-teaming-your-agent/demo_setup.py)
$ python3 make_scratch.py ~/red-team project/ a copy of the invoice tool project/docs/vendor-notes.md the planted file red-team-settings.json the permissions for the session manifest.json a hash of every file, for check_run.py $ python3 check_run.py ~/red-team Changed since setup: nothing Canary key found in: no other file answer.txt: missing, so the agent's reply wasn't checked Findings: 0 exit status 0
Reviewing a diff you did not write, against the spec
Unlocks when you finish Reviewing a diff you did not write, against the spec.
Takeaways
- Read the brief, then the tests, then the diff, and answer whether the change does what was asked before how it does it. Reviewing code you didn't write means every line has to justify itself against the spec, because you have no memory of why it exists.
- Start with the list of changed files and question every file the brief never named. A silent behavior change, a reformat and a shrinking test file are the riskiest part of an agent's diff, and a deleted test is a claim the code no longer makes.
- Check the success criteria item by item, by running the command each one names. A plausible change with a criterion dropped or an edge case skipped looks complete until then, and a green suite proves only what the agent's own tests say.
- Run the branch as well as reading it. Reading catches the design, and running catches the behavior the tests missed.
Example
Implement SPEC.md on the branch due-dates and commit when python3 -m unittest -q passes. Then summarize what you changed.
Done. I added the due and overdue commands to todo.py, with date validation so a malformed date is refused with bad date: <input>. list shows the date after the text. I added test_due_dates.py with tests for due, list, overdue and the validation, and the whole suite passes. I also cleaned up the formatting in one test file while I was there.
Giving the agent a check it can run, with a limit
Unlocks when you finish Giving the agent a check it can run, with a limit.
Takeaways
- Turn a manual check into a command the agent can run and read, one that exits with status 0 on a pass and names what failed otherwise. The agent can then rerun it after every change, and you no longer have to notice every mistake yourself.
- A bounded self-checking loop has a limit on attempts, after which the agent stops and reports, and a rule that the check itself is read-only.
- After the run, stage everything with
git add -Aand compare it with a commit from before the run. The list shows whether the test changed and which files the agent added. Then run the test yourself. A rule that denies edits to the test blocks the agent's file tools, the shell file commands it recognizes and redirects into the file. A script the agent runs can still write the file. - Deterministic gates, such as tests, type checks, linters, builds and schema checks, give the same answer for the same input when they aren't flaky, and can block a merge. A second model's review is a second opinion next to them and never replaces them.
Example
The test, before any fix · open in the lesson
Run the test in your copy and compare what you see with the output below.
Prints the lines below (verified in CI from site/examples/coding-with-agents/self-checking-loops/run_test.py)
2026-09-14: pass 2026-09-15: FAIL: exit status 1; parsed 39 rows, expected 40 rows 2026-09-16: pass 2026-09-17: FAIL: exit status 1; parsed 46 rows, expected 47 rows 2 passed, 2 failed exit status 1
Sources
Brilliant VERVerification, Brilliant, Coding with AI skills map (reference)Academy ai-fluency-for-buildersAI Fluency for builders, Claude Academy (course)Academy claude-code-in-actionClaude Code in action, Claude Academy (course)