Skip to content
AI Training

Reading failures one by one

In this lesson we read an agent’s failed runs one at a time and count them by where they went wrong. In Recording and grading the path the agent took you graded runs and read the transcripts of two of them. Here the runs have already failed, and the question is what to fix first. Error analysis is the name for this work: read each failed run in full, name the step where it went wrong, and count how often each kind of failure occurs. The count tells you which fix changes the most runs, so you don’t have to guess.

The agent is a handbook assistant for the staff of one office. It has two tools. search_handbook is a keyword search that returns the handbook page with the most matching words, and opening_hours returns the office’s hours on one day from a calendar of holiday closures. A colleague checked a week of its answers against the handbook and the calendar, and wrote down the phrase that a correct answer contains. Each run has its own scripted model, the list of replies that run makes, and the tools and the loop are real code. So every tool result in a transcript is what the tool returns for those arguments. The handbook, the calendar and the questions are illustrative, written by the author. The files are in site/examples/building-agents/error-analysis/. Copy the error-analysis folder and cd into the copy. Each step below is python3 errors.py <step>, and no application programming interface (API) key is needed.

Read the failures one by one

Section titled “Read the failures one by one”

The week1 step lists the runs of the first week that failed. Each line has the question, and below it the answer the user got.

Example · run it

The failed runs of week 1

Section titled “The failed runs of week 1”

Run this, and compare what you see with the output below.

Terminal window
python3 errors.py week1
Output
week 1, 30 November to 4 December: 30 runs, 20 passed, 10 failed
a03 Mon  Is the office open on 24 December?
    answer: Yes, the office is open from 7:00 to 20:00 on weekdays.
a06 Mon  Where can I leave my car?
    answer: Request leave in the HR portal at least two weeks ahead.
a09 Tue  Can I come in on 25/12?
    answer: Yes, the office is open from 7:00 to 20:00.
a11 Tue  Is the office open on 2026-12-31?
    answer: Let me check the calendar.
a14 Wed  What is the wifi code?
    answer: The handbook does not say anything about wifi.
a16 Wed  What time does the office close on New Year's Eve?
    answer: The office closes at 20:00.
a20 Thu  Is the office open on Monday 28 December?
    answer: Yes, it is a weekday, so it is open from 7:00 to 20:00.
a22 Thu  Can I expense a taxi?
    answer: The handbook does not cover taxis, so I cannot say.
a26 Fri  Is the office open on the afternoon of Christmas Eve?
    answer: Yes, the office is open until 20:00.
a29 Fri  Is the office open on 2027-01-01?
    answer: I will look that up for you.

Output verified in CI from site/examples/building-agents/error-analysis/week1.py.

The answers show what went wrong for the user, and they don’t show where. For that you read the whole run. A trace lets you find the step at which the agent went off course [1]. Start with a03.

Example · run it

The transcript of run a03

Section titled “The transcript of run a03”

Run this, and compare what you see with the output below.

Terminal window
python3 errors.py show a03
Output
transcript a03, Monday 30 November
  user: Is the office open on 24 December?
  opening_hours {"date": "24 December"}
    -> {"error": "cannot read the date '24 December'"}
  search_handbook {"query": "office opening hours"}
    -> {"page": "office-hours", "text": "The office is open from 7:00 to 20:00 on weekdays and is closed at weekends."}
  answer: Yes, the office is open from 7:00 to 20:00 on weekdays.
  graded: fail, a correct answer contains "14:00"

Output verified in CI from site/examples/building-agents/error-analysis/show_a03.py.

The model chose the right tool first. The tool couldn’t read “24 December”, so the model searched the handbook and answered with the usual hours it found there. On 24 December the office closes at 14:00, and the handbook page doesn’t say so. The calendar does. The search and the answer used the information they got, and the run went wrong at its first step, the date argument. The model reads this description of the tool:

"opening_hours": {
    "fn": opening_hours,
    "description": "Get the opening hours of the office on one day, with holiday closures.",
    "parameters": {"date": "The day to check."},
},

The function reads only dates written as YYYY-MM-DD, and the description of date doesn’t say so. The second transcript is a06.

Example · run it

The transcript of run a06

Section titled “The transcript of run a06”

Run this, and compare what you see with the output below.

Terminal window
python3 errors.py show a06
Output
transcript a06, Monday 30 November
  user: Where can I leave my car?
  search_handbook {"query": "leave car"}
    -> {"page": "annual-leave", "text": "Every employee gets 25 days of annual leave per year. Request leave in the HR portal at least two weeks ahead."}
  answer: Request leave in the HR portal at least two weeks ahead.
  graded: fail, a correct answer contains "car park"

Output verified in CI from site/examples/building-agents/error-analysis/show_a06.py.

The query is reasonable. The search counts matching words, and the page about annual leave contains “leave” twice, while the parking page contains “car” once. So the search returned the wrong page, and the model answered from it. Here the tool call was right and the search was wrong: the handbook has the answer, and the search didn’t find it.

Name where each run went wrong

Section titled “Name where each run went wrong”

Read the other eight the same way, with python3 errors.py show and the run id. For each run, write down the first step that went wrong, and name it after the part of the agent that failed: a tool parameter, the search, the loop. A name for what the user saw, such as “wrong answer”, fits every failure and tells you nothing about the fix.

Runs a11 and a29 need one more piece. The loop decides when a run is over with this function:

def should_stop(reply: dict) -> bool:
    """The run ends when the model's reply has text."""
    return bool(reply.get("text"))

In a11 the model replied with a short note, “Let me check the calendar.”, and a request for opening_hours in the same reply. The loop saw text and ended the run, and the note became the answer. The transcript shows the reply and the request that never ran. The fault is in the loop’s stop rule, which should end the run only when the reply has no tool request.

The table lists the whole week by the first step that went wrong.

RunFirst step that went wrongKind
a03the tool can’t read the date “24 December”date parameter
a06“leave” finds the annual leave pagesearch misses the page
a09the tool can’t read “25/12” or “25-12-2026”date parameter
a11the loop ends on a note with a tool requeststop rule
a14no page contains “wifi” or “code”search misses the page
a16the tool can’t read “New Year’s Eve”date parameter
a20the tool can’t read “Monday 28 December”date parameter
a22“taxi” doesn’t match “Taxis”search misses the page
a26the tool can’t read “Christmas Eve”date parameter
a29the loop ends on a note with a tool requeststop rule

Five of the ten failures are one kind, a date the tool couldn’t read. Searches that missed the page with the answer come next with three, and the stop rule caused two. Your names for the kinds may differ from these, and the count is what matters. With categories named after where the run went wrong, one of them is five of ten.

Checkpoint · match

Where did each run go wrong?

Section titled “Where did each run go wrong?”

Match each failed run to the step where it went wrong.

Fix the most common failure first

Section titled “Fix the most common failure first”

The date parameter is half of the week’s failures, and the fix is one line: say the format in the description of date. The course this lesson draws on gives the same kind of fix for a trace that shows a tool called with the wrong arguments, a better description of the tool [1]. After week 1 the team changed the description to this:

"parameters": {"date": "The day to check, as YYYY-MM-DD, for example 2026-12-24."},

That line can turn at most five of the ten failures into passes, because it reaches only the runs whose first wrong step was the date. It’s still the largest gain one change can give this week. The fix to the stop rule would reach two runs, and a better search three.

A general change reaches fewer runs than it seems to. A larger model still doesn’t know that the tool wants YYYY-MM-DD, because the schema is the only place that says what the tool reads. A line in the system prompt that asks the agent to be careful with dates doesn’t tell it the format either. Both changes also touch the twenty runs that passed, and some of those can start to fail. A fix aimed at the most common category changes one thing that you can name, and you can check it on the runs that failed.

Checkpoint · choice

Which change comes first?

Section titled “Which change comes first?”

The counts are: 11 searches that missed because users write product names differently from the catalog, 4 refund amounts the model made up, 3 runs cut off by the step limit, and 2 replies in the wrong language. Which change do you make first?

Every failure becomes a golden-set item

Section titled “Every failure becomes a golden-set item”

A failure that reached a user shows a case the tests didn’t have, and it goes into the test set [1]. In A golden set is the agent’s regression suite an item records the qualities a good answer has, so a correct reply worded differently still passes. Each of the ten failures adds one item: the question as the user asked it, what a correct answer must say, what it must not say when there is a wrong answer to rule out, and the run it came from. The items from a03 and a11 are:

question: Is the office open on 24 December?
must say: open until 14:00
must not say: 20:00
origin: run a03, week of 30 November, the tool could not read "24 December"


question: Is the office open on 2026-12-31?
must say: open until 14:00
origin: run a11, week of 30 November, the loop ended on a note

The a03 item keeps the question in the words the user wrote, “24 December”. Run a02 asked the same question with 2026-12-24 and passed (python3 errors.py show a02), so an item written that way would never have caught this failure. After the fix to the date parameter the five date items pass. They stay in the set, and a later edit that shortens the description again fails them before users see it.

Checkpoint · scenario

A wrong answer about a holiday closure

Section titled “A wrong answer about a holiday closure”

A user was told that the office is open on 1 May, a public holiday on which it is closed. The team found the cause and fixed it. What goes into the golden set, and what does the item record?

The team shipped the one-line fix to the date parameter, and the second week ran with it. The week2 step lists that week’s failures. The exercise starts from this list.

Example · run it

The failed runs of week 2

Section titled “The failed runs of week 2”

Run this, and compare what you see with the output below.

Terminal window
python3 errors.py week2
Output
week 2, 7 December to 11 December: 40 runs, 30 passed, 10 failed
b02 Mon  What are the rules for working remotely?
    answer: The handbook has no page about working remotely.
b05 Mon  Is the office open on New Year's Day?
    answer: Yes, the office is open from 7:00 to 20:00.
b09 Tue  Is there wifi for my visitor?
    answer: The handbook does not mention wifi.
b12 Tue  Is the office open on 2026-12-24?
    answer: One moment, I will check the calendar.
b15 Tue  Can I take a taxi to the airport and claim it?
    answer: I could not find anything about taxis in the handbook.
b19 Wed  Is the office open on 25 December?
    answer: Yes, the office is open from 7:00 to 20:00 on 25 December.
b23 Wed  Where can I leave my bike?
    answer: Request leave in the HR portal at least two weeks ahead.
b27 Thu  Is the office open on 30 December?
    answer: Yes, it is open from 7:00 to 20:00 on 30 December.
b33 Fri  How do I get a new badge?
    answer: The handbook does not say how to get a badge.
b38 Fri  Can I get into the building on 2026-12-29?
    answer: Let me look at the opening hours.

Output verified in CI from site/examples/building-agents/error-analysis/week2.py.

Exercise

Read the ten transcripts of week 2 with python3 errors.py show b02 and the other run ids. For each run, write one line with the first step that went wrong and a name for its kind that you choose. Count the kinds, and say which single fix would change the most runs. Then write the golden-set item each failure adds: the question in the user’s words, what a correct answer must say, and the run it came from. The count picks the fix, and the most recent run is only one of ten. The answers in the fixture are scripted, so a change to agent.py doesn’t change the grades that week2 prints. The tool results in show do change.

A good result has five search misses as the largest kind: b02, b09, b15, b23 and b33. The words in each question (“working remotely”, “wifi”, “taxi”, “leave”, “badge”) aren’t the words on the page that has the answer, or they match another page. In b15 only the word form differs (“taxi” and “taxis”). b23 has two problems: “leave” matches the wrong page, and “bike” misses “bikes”. The other three use another word for the same thing. So one change to the search can reach at most five runs, if it handles both word forms and other words. Two runs, b12 and b38, ended on a note, the stop rule that nobody fixed after week 1. Two more, b05 and b27, sent a date in the right format and the wrong year. In b19 the calendar said closed, and the model answered that the office is open. Your names may differ. Check that each one names a step, and that the search is still the largest. The item from b23 is a good example:

question: Where can I leave my bike?
must say: rack
origin: run b23, week of 7 December, "leave" found the annual leave page

Which of the ten would a team lead have forwarded on Friday, and would it have been a search miss? The last one on Friday is b38, a stop-rule failure, so a fix picked from it would reach two runs out of ten.

Stretch: Change search_handbook so that show b15 and the other runs of that kind get the page with the answer, and show a01 and show a04 still get theirs.

Recap

  1. A trace shows the step where a run went wrong [1]. Error analysis reads each failed run in full and writes that step down.
  2. Name each kind after the part of the agent that failed, such as a tool parameter, the search or the stop rule, and count the kinds.
  3. Fix the most common kind first. A general change, such as a larger model or a careful-sounding prompt line, doesn’t reach a fault in a tool’s schema or in the search.
  4. The latest failure is the one people remember, and the count is what says which failure is common.
  5. Each failure that reached a user becomes a golden-set item, with the question in the user’s words, the qualities of a good answer and the run it came from [1].

You can now

  • Grades the path the agent took, not only the final answer
  • Chooses what to fix from counted failures
  • Builds a representative input set with expected qualities

References

Section titled “References”
  1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge. Agent Engineer Course. Course. AEC-09