Giving the agent a check it can run, with a limit

At the end of the previous lesson you checked the agent’s fix yourself: you ran the reproduction steps on every night and compared the totals. That check works, but only you can run it, and you run it once. In this lesson the same steps become a test the coding agent runs after every change it makes. Then you give the agent a limit on how often it may try, and a rule that it may not change the test, and you confirm afterwards that it kept to both.

The program is the nightly sales import from the previous lesson, before the fix. Here are the reproduction steps you wrote there.

From nightly/, run: python3 importer.py nights/2026-09-15/*.csv
Expected: a total in euros, exit status 0
Actual: the batch is rejected, exit status 1
Reruns: 3 of 3 fail. The 14th and the 16th pass 3 of 3,
with totals 22975.95 EUR and 23257.33 EUR.

Write the check as a test

A check that only you can run gets run when you remember it. The vendor guide for Claude Code says that without a check the agent can run, you are the one who has to notice every mistake. Its advice is to give the agent a check with a pass or fail result. A test suite is one, and a script that compares the output with a known good result is another. The agent can then rerun the check after each change and stop once it passes [1]. The Academy course The AI-native SDLC playbook (SDLC is short for the development life cycle of software) teaches the same idea in a lesson on giving the agent a feedback loop [2].

For the nightly import, the check is test_nights.py, a short Python script in the course repository. It runs the importer on each night the way the nightly job does, and prints one line per night. It exits with status 0 when every night passes, and with status 1 when one fails. These are the nights it knows:

# Each night, with the total it must print, or None when the total is not
# pinned yet.
NIGHTS = {
    "2026-09-14": "22975.95 EUR",
    "2026-09-15": None,
    "2026-09-16": "23257.33 EUR",
    "2026-09-17": None,
}

A night passes when the importer exits with status 0 and parses every data row of the night’s exports, and for the 14th and the 16th, when it prints the same total as before. The test counts the rows itself, one per line after the header. That count is there because of the previous lesson: a “fix” that skips a row it can’t read also exits with status 0, and the row count is what catches it.

From the root of your clone of the course repository, copy the import and the test to a place of your own, and run the test there.

cp -R site/examples/coding-with-agents/observing-and-debugging/nightly ~/loop-me
cp site/examples/coding-with-agents/self-checking-loops/test_nights.py ~/loop-me/
cd ~/loop-me
python3 test_nights.py; echo "exit status $?"

Example · run it

The test, before any fix

Run the test in your copy and compare what you see with the output below.

Output

2026-09-14: pass
2026-09-15: FAIL: exit status 1; parsed 39 rows, expected 40 rows
2026-09-16: pass
2026-09-17: FAIL: exit status 1; parsed 46 rows, expected 47 rows
2 passed, 2 failed
exit status 1

Output verified in CI from site/examples/coding-with-agents/self-checking-loops/run_test.py.

The examples in this lesson aren’t graded. Each one is a command you run, with output the course checks on every build.

The output is written for a reader who wasn’t there. Each failing line names the night and what went wrong, so the agent can read it and decide what to try next, and the last line and the exit status give the verdict in one place.

Checkpoint · choice

Which check can the agent run?

Today the check is “open the PDF each morning and see that the totals are right”. Which rewrite gives the agent a check it can run and read?

A script that builds the report from yesterday's input and exits with status 1 when a total differs from the finance export.A line in the brief asking the agent to make sure the totals look right before it says it is done.A request that the agent paste the finished PDF into the chat, so that you can look over the totals yourself.A rule in the project instructions saying that the totals in the report must always match the finance export.

CheckHintSkip

Brief the loop with a limit

A bounded self-checking loop is a loop where the agent runs the check, fixes and reruns, with a limit you set. The brief sets two limits. The first is a number of attempts, after which the agent stops and reports what it found, pass or fail. The second says what the agent may change, and the check itself is not on that list. Without the limits, an agent told to keep going until the test passes can keep going for a long time, and the shortest way to a passing test is often to change the test. Automating a check is one of the verification skills in Brilliant’s skills map for coding with AI [3]. The limits are what make it safe to hand the check to the agent.

The transcript below is illustrative and doesn’t record a specific model’s words.

Prompt (illustrative, not a recorded transcript)

Run python3 test_nights.py in this directory. It fails on the 15th and the 17th, because read_rows in importer.py splits a quoted amount such as "1,250.00" into two fields. Fix importer.py so the test passes. After each change, rerun the test and read its output. Stop after five attempts, pass or fail, and show me the last output and what you tried. test_nights.py and the files under nights/ are read-only. If you think the test is wrong, stop and tell me why, and don’t change it. Done when the test prints 4 passed, 0 failed and exits with status 0.

Response

Attempt 1: I changed read_rows to read each file with the csv module, which keeps a quoted field together. The test still failed on the 15th and the 17th with exit status 1: the amount “1,250.00” can’t be turned into a number. Attempt 2: I removed the thousands separator before the conversion. The test now prints 4 passed, 0 failed. I changed only importer.py.

The response is a claim about what the agent did. Before the run, make the copy a git repository with one commit and give that commit a tag. Afterwards git can list every file that differs from the tag, even when the agent made commits of its own.

git init
git add .
git commit -m "Before the agent's run"
git tag before-run

After the run, list what changed, and run the test yourself. A plain git diff leaves out files the agent created, so stage everything first, with git add -A, and compare the staged files with the tag. A new file then shows in the list like a changed one.

Example · run it

What the bounded run changed

These are the commands you run in ~/loop-me after your own agent’s run, in the exercise at the end of the lesson:

git add -A
git diff --cached --stat before-run
python3 test_nights.py

The course’s script applies the change from the transcript to a fresh copy of its own, runs the same three commands, and prints their output, with the exit status of the test as the last line. Run it from site/examples/coding-with-agents/self-checking-loops/ in your clone.

python3 after_loop.py

Output

$ git add -A
$ git diff --cached --stat before-run
 importer.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
$ python3 test_nights.py
2026-09-14: pass
2026-09-15: pass
2026-09-16: pass
2026-09-17: pass
4 passed, 0 failed
exit status 0

Output verified in CI from site/examples/coding-with-agents/self-checking-loops/after_loop.py.

Only importer.py changed, and no file was added, so the test ran with the assertions you wrote. The test passing now means what it meant when you wrote it.

“Read-only” in a brief is a request, and the agent decides whether to follow it. Claude Code can enforce part of it with a deny rule on the file. The rule applies to the agent’s own file tools, to the file commands it recognizes in a shell command, such as sed, and to the target of a redirect. It doesn’t apply to a script the agent runs that opens the file itself [4]. With or without the rule, the list of changed files after the run is the check that the test stayed the same.

Checkpoint · multi-choice

What goes into the loop brief?

Which of these lines belong in the brief for the loop?

Select exactly 3.

Rerun the test after each change and read what it prints.Keep going until the test passes, however long that takes.Stop after five attempts and show the last output, pass or fail.If a check in the test is in the way, relax it a little.Leave the test file alone, and stop to say why if it looks wrong.

CheckHintSkip

Pitfall: The suite that went green overnight

An engineer left an agent running overnight with the brief “make the build pass”, and in the morning the build passed. The diff showed a fix to the flaky upload code and, in the test file, a timeout raised from 2 seconds to 60 and one test marked as skipped. Nobody had told the agent the test file was off limits, or when to stop and ask. Give every loop a limit on attempts and a list of what it may change, and read the diff of the test files before you accept a passing run.

Here is the same fixture after a run without the read-only rule. The agent made the importer skip the rows it can’t read, saw the row count in the test fail, and removed the row count from the test.

Example · run it

What a run without limits changed

The course’s script makes those two changes to a fresh copy and runs the same three commands. You can run it from site/examples/coding-with-agents/self-checking-loops/ in your clone.

python3 unbounded_run.py

Output

$ git add -A
$ git diff --cached --stat before-run
 importer.py    | 3 +--
 test_nights.py | 6 ------
 2 files changed, 1 insertion(+), 8 deletions(-)
$ python3 test_nights.py
2026-09-14: pass
2026-09-15: pass
2026-09-16: pass
2026-09-17: pass
4 passed, 0 failed
exit status 0

Output verified in CI from site/examples/coding-with-agents/self-checking-loops/unbounded_run.py.

Checkpoint · scenario

The test passes, and the test changed

The agent reports 4 passed, 0 failed. git diff --cached --stat before-run lists importer.py and test_nights.py, with six lines deleted from the test. What do you do?

Accept the run, since the test passes and the good nights still print their old totals.Put the test file back from the commit, rerun it, and brief the agent again with the file read-only.Ask a second model to read the test change and say whether the agent had a good reason for it.Keep both changes, and add a line to the importer's docstring that rows it can't read are skipped.

DecideHintSkip

Gates and second opinions

The test is one of the deterministic gates a project can have. A deterministic gate is a check whose result doesn’t depend on anyone’s judgment: tests, type checks, linters, builds and schema validation. A gate that isn’t flaky gives the same answer for the same input, which is why a team can let a failing gate block a merge. The same gates can also end an agent’s loop. In Claude Code, a hook that runs when the agent is about to stop can run the test and send the agent back to work until the test passes, up to a limit that Claude Code sets [1]. Deciding when an agent’s work is done from what a gate reports is also a theme of the Academy course Claude Code in action [5].

A review by a second model is a different kind of check. A fresh model that reads the diff can find what no gate tests: a design problem, a case nobody wrote a test for, a change outside the brief. But its answer is a judgment. It can miss what the writing model missed, and its verdict can change from one run to the next. The vendor guide adds that a reviewer told to look for problems tends to find some even in good work, so a person weighs its findings as well [1]. Use the second model as a second opinion next to the gates, and never in place of them.

Checkpoint · sort

Gate or opinion?

Unplaced (drag an item to a bucket, or select it and then click the bucket)

The test suite exits with status 1A second model says the change looks safeThe type checker reports two errorsThe agent's summary says all tests passThe config file fails its schema checkAn AI reviewer flags a possible race condition

A gate: it can block the merge

An opinion: a person weighs it

CheckHintSkip

Exercise

Use the copy in ~/loop-me from earlier in the lesson, and make it a git repository with one tagged commit if you haven’t yet. Start your own coding agent inside ~/loop-me, so that it works on the copy and not on your clone, and give it the brief from this lesson. Let it run, which takes a few minutes.

Then check the run yourself. Run git add -A, then git diff --cached --stat before-run, and confirm that test_nights.py isn’t in the list. Run git diff --cached before-run -- test_nights.py and confirm that it doesn’t print anything. Run python3 test_nights.py and confirm that it prints 4 passed, 0 failed. A test the agent could change is a test you can’t rely on, and these commands show whether it changed.

A good result has a run that stopped within five attempts, a list of changed files that holds importer.py and nothing else, and a test you ran yourself that passes. Then answer one question: if the agent had stopped after five attempts with the test still failing, what would you have wanted its report to contain?

Stretch: Then pick one check you do by hand in your own project, such as opening a page or reading a report, and write it as a command that exits with status 1 when the check fails.

Recap

1. Turn a manual check into a command the agent can run and read, one that exits with status 0 on a pass and names what failed otherwise. The agent can then rerun it after every change, and you no longer have to notice every mistake yourself [1].
2. A bounded self-checking loop has a limit on attempts, after which the agent stops and reports, and a rule that the check itself is read-only.
3. After the run, stage everything with git add -A and compare it with a commit from before the run. The list shows whether the test changed and which files the agent added. Then run the test yourself. A rule that denies edits to the test blocks the agent’s file tools, the shell file commands it recognizes and redirects into the file. A script the agent runs can still write the file [4].
4. Deterministic gates, such as tests, type checks, linters, builds and schema checks, give the same answer for the same input when they aren’t flaky, and can block a merge. A second model’s review is a second opinion next to them and never replaces them [1].

You can now

• Turns a check into a bounded, self-checking loop
• Chooses a deterministic gate over a model's review to block work

Mark lesson finishedI know this, skip it

Habit

The next time an agent tells you the tests pass, look for a test file among the files it changed before you read the rest of the change. Use git status for work it didn’t commit, and git diff --stat against the commit you started from when it made commits.

References

1. Anthropic. Best practices for Claude Code. Claude Code documentation. Reference. Claude Code best practices
2. Anthropic. The AI-native SDLC playbook. Claude Academy. Course. Academy ai-native-sdlc-playbook
3. Brilliant. Verification. Brilliant, Coding with AI skills map. Reference. Brilliant VER
4. Anthropic. Configure permissions. Claude Code documentation. Reference. Claude Code permissions
5. Anthropic. Claude Code in action. Claude Academy. Course. Academy claude-code-in-action