Reproducing a fault before the agent fixes it

In this lesson you get a bug report that says “sometimes fails”, and you turn it into a confirmed cause before a coding agent changes a line. The program is a small nightly import that adds up the sales exports of eight stores. The course keeps it as a fixture, so everyone debugs the same code and the same data. Here is the report.

The nightly sales import failed on Tuesday the 15th and on Thursday the
17th, and ran fine on the 14th and the 16th. Nobody changed the code. I
think the exchange-rate download is sometimes late at night and leaves a
half-written rates.json. Can you ask the agent to make the download more
reliable?

The report contains a symptom, two dates, and a theory. An agent handed this report gets a theory to work from and a request for a fix, and the fix it writes is aimed at the theory. Asking for a reproduced, explained cause before any patch keeps the fix from becoming a second bug [1].

Reproduce it first

“Sometimes” means you don’t yet know what the failure depends on. The first step is to run the failing case again, the same way, and see whether it fails every time. The fixture keeps each night’s exports in nightly/nights/<date>/, one comma-separated values (CSV) file per store, and the nightly job runs python3 importer.py nights/<date>/*.csv from nightly/. The course’s script runs that command three times for each of the four nights and prints the result of each run, with the total for a night that passes.

Example · run it

Run each night three times

In a clone of the course repository, go to the fixture directory, site/examples/coding-with-agents/observing-and-debugging/, and run the reproduction script there. The scripts in this lesson all run from that directory.

python3 reproduce.py

Output

2026-09-14: ok ok ok (total: 22975.95 EUR)
2026-09-15: rejected rejected rejected
2026-09-16: ok ok ok (total: 23257.33 EUR)
2026-09-17: rejected rejected rejected

Output verified in CI from site/examples/coding-with-agents/observing-and-debugging/reproduce.py.

The examples in this lesson aren’t graded. Each one is a command you run, with output the course checks on every build. The checkpoints ask what you do with what the commands show.

Every rerun gives the same result as the night it reruns. The failure now has a precise condition: it happens every time, on the exports of the 15th and the 17th. Note the totals of the two good nights as well, because the fix must not change them. Write the steps down now, in a form someone else can follow, because they’re the test the fix must pass at the end.

From nightly/, run: python3 importer.py nights/2026-09-15/*.csv
Expected: a total in euros, exit status 0
Actual: the batch is rejected, exit status 1
Reruns: 3 of 3 fail. The 14th and the 16th pass 3 of 3,
with totals 22975.95 EUR and 23257.33 EUR.

Checkpoint · choice

What do the reruns tell you?

The 15th and the 17th fail three runs of three, and the 14th and the 16th pass three runs of three. What does that tell you about the cause?

That the rates file was half-written on the 15th and the 17th and still is today.That the failure follows the exports of the 15th and the 17th, because the same input fails on every run.That the failure follows the time of day, and the reruns happened to land on a calm moment.That nothing is known yet, because a fault that fails only sometimes can never be made to fail on demand.

CheckHintSkip

Read what the system recorded

Observing a running system means looking at what the program did, and not only at what its code says it should do: the log it writes, the files it reads, the output and the exit status. The code tells you what should happen, and the log tells you what did [1]. The importer writes one log line per step. Put the log of a failing night next to the log of a good one.

Example · run it

The log of a good night and a failing one

From the fixture directory, the course’s script runs the importer on the 14th and on the 15th, from nightly/, and prints each log with its exit status. To run the two commands yourself instead, cd nightly first, follow each command with echo $?, and cd .. when you are done. The next steps start from the fixture directory again.

python3 night_log.py

Output

$ python3 importer.py nights/2026-09-14/*.csv
INFO reading 8 files
INFO rates: 3 currencies loaded
INFO parsed 46 rows
INFO total: 22975.95 EUR
exit status 0
$ python3 importer.py nights/2026-09-15/*.csv
INFO reading 8 files
INFO rates: 3 currencies loaded
INFO parsed 39 rows
ERROR 1 row(s) with the wrong number of fields, batch rejected
exit status 1

Output verified in CI from site/examples/coding-with-agents/observing-and-debugging/night_log.py.

Read the two logs line by line. Both read eight files, and both load three currencies from the rates file. The failing run gets past the rates step and stops at the next one, where one row has the wrong number of fields. The rerun rules out the colleague’s theory: the rates load in full, and the batch is still rejected at the next step. Write that down too, with the evidence, so nobody has to rule it out again.

The log doesn’t say which of the eight files holds the bad row. It narrowed the cause from the whole program to one step, and the step still has eight suspects.

Pitfall: The handler the request never reached

A checkout page sometimes shows an error, and the coding agent starts to rewrite the order handler with more checks for a field that could be missing. The engineer opens the service’s request log first, and the failing requests aren’t in it. A proxy in front of the service rejected them for headers that were too large, so the handler never saw them. The rewrite would have passed its tests and changed nothing for the user. Before anyone changes the code a theory points at, check in the log that the failing run reached that code.

Checkpoint · scenario

The agent proposes retries

The agent says: “The rate download is unreliable at night. I’ll add three retries with a pause between them, and check that rates.json is complete before it is used.” The log of the rerun of the 15th is open next to it. What do you do?

Let it add the retries, since a more reliable download is useful even if it isn't this bug.Ask it to add detailed logging to the download, so the next failing night shows what the download did.Ask it to rewrite the import to be more tolerant, so that one bad row no longer rejects the batch.Stop it and point at the log of the rerun, where the rates load in full and the parsing step fails after them.

DecideHintSkip

Halve what is left

Isolating a fault means narrowing it down to its cause in steps, where each step changes one thing and gives one observation. With eight files to suspect, the fastest way is to halve them. Run the importer on half of the files. If that half passes, the bad row is in the other half, and if it fails, the bad row is in this one. Each run rules out half of what is left, so eight files take three runs, and a fourth run confirms the last one. The shell pattern store-0[1-4].csv picks four files by name.

cd nightly
python3 importer.py nights/2026-09-15/store-0[1-4].csv
python3 importer.py nights/2026-09-15/store-0[5-6].csv
python3 importer.py nights/2026-09-15/store-05.csv
python3 importer.py nights/2026-09-15/store-06.csv
cd ..

Example · run it

Halve the files of the 15th

From the fixture directory, the course’s script runs the four commands above from nightly/ and prints ok or rejected for each.

python3 halve.py

Output

store-01 to store-04: ok
store-05 and store-06: rejected
store-05: ok
store-06: rejected

Output verified in CI from site/examples/coding-with-agents/observing-and-debugging/halve.py.

Keep a note of what each run ruled out, one line per run: stores 1 to 4 are ruled out, stores 7 and 8 are ruled out, store 5 is ruled out, and store 6 fails alone. Halving works best when the failure has one cause. If both halves had failed, the note would show it at once, and you would halve each of them.

Now store-06.csv is short enough to read.

date,store,currency,amount
2026-09-15,store-06,GBP,449.30
2026-09-15,store-06,GBP,642.20
2026-09-15,store-06,GBP,"1,250.00"
2026-09-15,store-06,GBP,432.53

The fourth line has an amount of a thousand or more, which the store’s system writes in quotes and with a comma. Read read_rows in importer.py with that line in mind: it splits each line on every comma, so "1,250.00" becomes two fields and the row has five. This cause explains the whole report: the nights that failed are the nights with a big sale, and nothing about the timing matters. The condition is now precise. The import fails when an export holds an amount of 1,000 or more.

The same halving works on commits as well as on files. When a fault appeared between a version that worked and one that doesn’t, git bisect does a binary search over the commits between them. It checks out one commit at a time, you mark it good or bad, and it stops at the commit that brought the fault in [2].

Checkpoint · choice

Which run comes next?

On the 15th, store-0[1-4].csv passes on its own. Which run do you do next?

The importer on store-05 and store-06 only, because the bad row is now among stores 5 to 8, and half of that is two files.The importer on stores 1 to 4 once more, because one passing run of a failure that seemed random proves nothing.The importer on store-01 alone, because the first file is where a bad row most often turns up in a batch.The importer on stores 5 to 8 with a change that makes it skip bad rows, to see whether the total then looks right.

CheckHintSkip

Brief the agent with the cause

Only now does the agent get the task. The brief contains everything the reproduction and the halving found: the command that fails, the file and the line, the reason, what was ruled out, and what done means. The transcript is illustrative and doesn’t record a specific model’s words.

Prompt (illustrative, not a recorded transcript)

The nightly import in nightly/importer.py rejects the batch for 2026-09-15 every time. To reproduce it, run python3 importer.py nights/2026-09-15/*.csv from nightly/. It logs 1 row(s) with the wrong number of fields and exits with status 1. The cause is line 4 of nights/2026-09-15/store-06.csv, the amount "1,250.00": read_rows splits on every comma, so the quoted amount becomes two fields. The rates download is ruled out, because a rerun with a complete rates.json fails the same way. Fix the parsing so that a quoted amount with a thousands separator is read as one amount, and change nothing else. The 17th fails with the same log line. Done when all four nights under nights/ exit with status 0, and the 14th and the 16th still print 22975.95 EUR and 23257.33 EUR.

Response

I changed read_rows to read each file with the csv module, which keeps a quoted field together, and removed the comma from the amount before it is turned into a number. Nothing else changed. All four nights now pass.

The response is a claim, and the reproduction steps are the check. Run them again yourself, on every night, and compare the totals of the nights that already passed. The course’s script applies the same fix to a copy of nightly/ and reruns the reproduction.

Example · run it

The reproduction, after the fix

From the fixture directory, run the script that applies the fix on a copy and prints each night’s result and total.

python3 after_fix.py

Output

2026-09-14: ok, total: 22975.95 EUR
2026-09-15: ok, total: 18505.15 EUR
2026-09-16: ok, total: 23257.33 EUR
2026-09-17: ok, total: 23213.13 EUR

Output verified in CI from site/examples/coding-with-agents/observing-and-debugging/after_fix.py.

The two failing nights pass, and the two good nights print the same totals as before. The fix is verified against the steps that showed the fault, which is a stronger check than the agent’s summary of it.

Checkpoint · multi-choice

What goes into the brief?

Which of these belong in the brief you give the agent?

Select exactly 3.

The command that fails and the output it gives now.The colleague's theory about the rates download, as a second thing to fix.The file and the line with the quoted amount, and why a split on commas breaks it.Permission to change whatever the agent needs to change until the import passes.All four nights pass, and the totals of the good nights stay the same.

CheckHintSkip

Exercise

From the root of your clone of the course repository, copy the fixture’s nightly/ directory to a place of your own, and work only in the copy.

cp -R site/examples/coding-with-agents/observing-and-debugging/nightly ~/debug-me
cd ~/debug-me

Take the 17th, which the lesson left alone, through the same steps. Spend about ten minutes. Write the reproduction steps, with how many reruns failed. Compare its log with a good night’s and write one line on what the log rules out. Halve the eight files, one run at a time, and keep a line per run for what it ruled out. Then read the file that is left. This is the order that keeps a guessed fix out of your code, and it works the same for a bug in your own project.

A good result has a reproduction that fails three runs of three, a note that the rerun of the 17th loads the rates in full, and a halving that ends at store-03.csv, where line 4 holds the amount "2,180.40". The cause is the same as on the 15th. Then answer one question: if you had briefed the agent with the colleague’s report as it was, what would it have changed?

Stretch: Then start your own coding agent inside `~/debug-me`, brief it with what you found, and verify its fix against your reproduction steps. Compare its diff with the fix the lesson describes.

Recap

1. Reproduce the fault before anyone changes code. Rerun the failing input and write the steps down, and “sometimes fails” becomes a precise condition and the test the fix has to pass [1].
2. Observing a running system means reading what it recorded, the log, the output and the exit status, next to a run that worked. The log shows how far the failing run got, and a theory about a step the run completed is ruled out.
3. Isolating a fault means halving the suspects: one change, one observation, and a note of what was ruled out. The same halving over commits is git bisect [2].
4. Brief the agent with the reproduced, explained cause and a done-criterion from the reproduction, and verify the fix by running those steps yourself.

You can now

• Observes the running system and isolates a fault systematically

Mark lesson finishedI know this, skip it

Habit

The next time a bug report says a thing sometimes fails, rerun the failing case three times and write down the steps before you open the code or ask an agent for a fix.

References

1. Brilliant. Verification. Brilliant, Coding with AI skills map. Reference. Brilliant VER
2. The Git project. git-bisect. Git reference documentation. Reference. Git docs git-bisect