Recording and grading the path the agent took

In this lesson we give an agent’s loop a trace, and then grade the path each run took as well as its answer. The golden set in A golden set is the agent’s regression suite checked what the agent answered. Two runs can end with the same correct answer after different sequences of calls, and the run with the cleaner path is the one to rely on [1]. A grader that reads only the answer passes both.

The agent here is an invoice assistant for a finance team. It answers one kind of question, “Has invoice 2207 been paid?”, with three tools: search_invoices finds an invoice and the customer account it belongs to, get_payments lists the payments on one account, and send_reminder emails a customer about an unpaid invoice. The loop is the one from Building your first agent. A real model takes a different path on different runs, while a fake model repeats one path. So each of the fixture’s eight runs has its own scripted model, the list of requests that run makes, while the tools are real functions over a small table of invoices and payments. The files are in site/examples/building-agents/traces-and-trajectories/. Copy the traces-and-trajectories folder and cd into the copy. Each step below is python3 trajectory.py <step>, run from there. No application programming interface (API) key is needed.

Give the loop a trace

In Reproducing a fault before the agent fixes it you read a program’s log to find out what it did. An agent needs the same record, and one log line per step isn’t enough. Observability for an agent means that each model call is recorded with its input, its output, its duration and its token counts. Each tool call is recorded with its arguments and its result, whether it worked, and its duration. The records of one user request are tied together as a trace [1]. The trajectory is the sequence of steps in that trace, from the question to the answer [1].

The traced loop in agent.py appends one record per model call and one per tool call. This is the part that records a tool call:

trace.append(
    {
        "run": run_id,
        "span": len(trace) + 1,
        "kind": "tool",
        "name": f"execute_tool {reply['tool']}",
        "tool": reply["tool"],
        "args": reply["args"],
        "result": result,
        "latency_ms": latency,
        "cost": 0.0,
    }
)

A model call gets a record of the same kind, with "kind": "model", what the model asked for in asked (a tool name, or answer on the call that ends the run), its token counts, its latency and its cost. That last model record also has answer, the text of the reply. The run field ties the records of one run together, and span numbers them in order.

The outline step prints one line per record of run r1. A model call prints as chat -> followed by what the model asked for. A tool call prints as execute_tool, the tool name and its arguments as name=value, for example execute_tool get_payments account=ACC-3301.

Checkpoint · predict

Predict the trace of run r1

Run r1 answers “Has invoice 2207 been paid?” by the path a good run takes. It finds the invoice and looks up the payments on the account the search returned, and then it answers. The search returns account ACC-1042. Predict every line the outline prints, then run it.

python3 trajectory.py outline

What is printed?

Output verified in CI from site/examples/building-agents/traces-and-trajectories/outline.py.

CheckHintSkip

The model is called once more than the tools. After the last tool result comes back, the loop calls the model again, and that call writes the answer. A trace that records only the tool calls leaves out the model calls, and in this run they are most of the time and all of the cost. The trace step prints every record of r1 with its latency and cost. The fake model reports a latency from a fixed formula over its token counts, and the prices per token are made up. Every run prints the same numbers.

Example · run it

The full trace of run r1

Run this, and compare what you see with the output below.

python3 trajectory.py trace

Output

trace r1: Has invoice 2207 been paid?
  1 chat fake-model  -> search_invoices, 111 in, 14 out  602 ms  $0.00054
  2 execute_tool search_invoices  number=2207  120 ms  $0.00000
  3 chat fake-model  -> get_payments, 157 in, 15 out  631 ms  $0.00070
  4 execute_tool get_payments  account=ACC-1042  80 ms  $0.00000
  5 chat fake-model  -> answer, 213 in, 14 out  622 ms  $0.00085
  total 2055 ms, $0.00209
  answer: Yes. Invoice 2207 was paid on 2026-09-02.

Output verified in CI from site/examples/building-agents/traces-and-trajectories/full_trace.py.

The input grows with every model call, from 111 tokens to 213, because each call sends the whole conversation so far. The record names follow the OpenTelemetry semantic conventions for generative AI, a shared set of names for the records of model and tool calls. There, a model call is a span named after the operation and the model, such as chat and the model name, and a tool call is a span named execute_tool and the tool name. The token counts go in gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The conventions are still marked as in development [2]. They mark the messages, the tool arguments and the tool results as sensitive, and an instrumentation should record them only when the user opts in [2]. This fixture records everything because its data is made up. For a production agent, decide who may read a trace before you record what users typed.

Same answer, different path

Run r2 asks the same question as r1. It searches and gets account ACC-1042, like r1. Then it calls get_payments three times with ACC-1024, an account with two digits swapped. The fourth call uses ACC-1042. Each of the three wrong calls returned the payments of another customer. The final answer is the same sentence as in r1.

The fixture has two graders. The answer grader checks that the answer contains the phrase a correct answer has, paid on 2026-09-02 for invoice 2207. The path grader checks the trace against two rules. The rule search-first says that the first tool call is a search. The rule same-account says that every payment lookup uses an account the search returned:

def same_account(trace: list[dict]) -> Optional[str]:
    """Every payment lookup uses the account the search returned."""
    searches = [c for c in tool_calls(trace) if c["tool"] == "search_invoices"]
    found = {c["result"]["account"] for c in searches if "account" in c["result"]}
    for call in tool_calls(trace):
        if call["tool"] == "get_payments" and call["args"]["account"] not in found:
            return "same-account"
    return None

The grader prints one line per run with the two verdicts. A path that breaks a rule prints as fail and the rule name.

Checkpoint · predict

Which run does the path grader fail?

The grader prints r1 answer: pass, path: pass for run r1. Predict both lines, the one for r1 and the one for r2, then run it.

python3 trajectory.py grade

What is printed?

Output verified in CI from site/examples/building-agents/traces-and-trajectories/same_answer.py.

CheckHintSkip

The answer grader can’t tell the two runs apart, because it reads the one sentence they share. The trace shows the difference, and the path grader turns it into a failure. Grading the path as well as the answer is trajectory evaluation. The runs step grades all eight runs and adds the tool calls, the latency and the cost of each.

Example · run it

All eight runs

Run this, and compare what you see with the output below.

python3 trajectory.py runs

Output

run  tools  latency   cost      answer  path
r1   2       2055 ms  $0.00209  pass    pass
r2   5       4317 ms  $0.00612  pass    fail same-account
r3   2       2055 ms  $0.00208  pass    pass
r4   3       3112 ms  $0.00321  pass    pass
r5   1       1318 ms  $0.00132  pass    fail search-first, same-account
r6   3       2804 ms  $0.00304  pass    pass
r7   2       2055 ms  $0.00209  fail    pass
r8   3       5819 ms  $0.00306  pass    pass

Output verified in CI from site/examples/building-agents/traces-and-trajectories/all_runs.py.

Run r2 took 4317 ms and cost $0.00612, against 2055 ms and $0.00209 for r1. Run r5 answered about invoice 2208 correctly, but it never searched: it guessed that the invoice belongs to ACC-1024 and looked up that account. The guess was right this time, and on another invoice the same guess reads the wrong customer’s payments. The answer grader passes seven of the eight runs, and it fails only r7.

Pitfall: Grading only the final answer

A team grades its invoice assistant by the answer alone, and the score stays at 100 percent for a month. Then a customer asks why they got three payment reminders for an invoice they had already paid. The agent had called send_reminder on lookups whenever a payment list was slow to load, and every answer it gave was still right. The answer score could never show it, because the reminders were in the path. Record a trace for every run, and grade the path the agent took as well as the answer.

Write down the expected path

The two rules came from the failure in r2, so they catch r2 and r5 and nothing else. To grade every path, write down what a good path looks like for this kind of task: which tools it uses, which it must not use, in what order, and about how many steps. Then check each trace against it, for the right tool at each step, the right arguments, no steps it didn’t need, and no action outside what the task allows [1].

For the lookup question, this course writes the expected path like this:

1. The first tool call is search_invoices, and the run searches once.
2. get_payments uses only the account the search returned.
3. The run uses no other tool. A lookup changes nothing, so send_reminder is never part of it.
4. The run makes at most three tool calls. That leaves room for one retry of a lookup that failed, and more calls than that are a sign that the agent is lost.
5. The run ends with an answer.

Read the eight runs against that list. Run r4 found invoice 2415 unpaid and sent the customer a reminder that nobody asked for. Its answer passes, and so does its path under the two rules, which never mention send_reminder. Run r6 searched twice for the same invoice, so it did a step it didn’t need. Run r8 called get_payments twice because the first call timed out, and that is the retry rule 4 allows. In the exercise you write the list as a function in my_rule.py and run it over all eight runs. The file starts with one rule, at most three tool calls, and the exercise step runs it.

Example · run it

The starter rule over the eight runs

Run this, and compare what you see with the output below.

python3 trajectory.py exercise

Output

r1 pass
r2 fail: 5 tool calls, more than 3
r3 pass
r4 pass
r5 pass
r6 pass
r7 pass
r8 pass

Output verified in CI from site/examples/building-agents/traces-and-trajectories/starter_rule.py.

Checkpoint · multi-choice

Which rules belong in the expected path?

Which of these lines belong in the expected path for the lookup question?

Select exactly 3.

Payment lookups use only the account that the search returned.The run never calls send_reminder while answering a lookup.The run makes exactly one payment lookup, never a second one.The final reply contains the paid date that the payments list shows.The run makes at most three tool calls in total.

CheckHintSkip

Read the transcripts

The graders check the failures someone thought of when they wrote them. A failure of a new kind passes both, and you find it by reading full runs. The course this lesson draws on suggests reviewing a sample of trajectories every week [1]. The fixture prints a run as a transcript, with the question and every tool call with its result, and then the answer and its two grades. Read two of them. Run r7 is the one the answer grader failed.

Example · run it

The transcript of run r7

Run this, and compare what you see with the output below.

python3 trajectory.py show_r7

Output

transcript r7
  user: Has invoice 2207 been paid?
  search_invoices {"number": "2207"}
    -> {"invoice": "2207", "account": "ACC-1042", "amount": "480.00"}
  get_payments {"account": "ACC-1042"}
    -> {"account": "ACC-1042", "payments": [{"invoice": "2207", "paid": "2026-09-02", "amount": "480.00"}]}
  answer: No. Invoice 2207 has not been paid yet.
  graded answer: fail, path: pass

Output verified in CI from site/examples/building-agents/traces-and-trajectories/transcript_r7.py.

By its two grades, r7 took a good path and gave a wrong answer. Reading the transcript shows where it went wrong. The payments list shows invoice 2207 as paid on 2026-09-02, and the model still answered that it was not paid. The tools and the path worked, so a fix to the tools or to the path rules won’t help. The fault is in how the model read the result. The second transcript is r8, which passed both graders.

Example · run it

The transcript of run r8

Run this, and compare what you see with the output below.

python3 trajectory.py show_r8

Output

transcript r8
  user: Has invoice 2415 been paid?
  search_invoices {"number": "2415"}
    -> {"invoice": "2415", "account": "ACC-2750", "amount": "2300.00"}
  get_payments {"account": "ACC-2750"}
    -> {"error": "timeout after 3000 ms"}
  get_payments {"account": "ACC-2750"}
    -> {"account": "ACC-2750", "payments": []}
  answer: No. Invoice 2415 has not been paid yet. It is due on 2026-09-30.
  graded answer: pass, path: pass

Output verified in CI from site/examples/building-agents/traces-and-trajectories/transcript_r8.py.

No tool returned a due date. The invoice record has a number, an account and an amount, and the payments list is empty. The model made up “due on 2026-09-30”, and a colleague who reads the answer may tell the customer that date. Both graders passed r8, because the answer contains has not been paid and the path is the expected one with one retry. The trace also shows a delay that neither grade reports: r8 took 5819 ms, and the timeout was 3000 ms of that. Turn the failure you found by reading into a new check, for example that every date in the answer appears in a tool result. Add the question to the golden set too.

Checkpoint · scenario

The scores didn't move

Your team changed the assistant’s system prompt. The answer pass rate and the path pass rate are both the same as at the last release. What do you do before the release tomorrow?

Release tomorrow, because both pass rates match the last release.Run both graders twice more tonight to make sure the rates hold.Read a sample of full transcripts from the new prompt before the release.Add three more path rules tonight and release if the new rules pass.

DecideHintSkip

Exercise

Copy the traces-and-trajectories folder and open my_rule.py. It has one function, expected_path(trace), which gets the records of one run in order and returns one reason per broken rule. An empty list is a pass. The starter checks one rule, at most three tool calls, and its output is in the section “Write down the expected path”. Add the other four rules of the expected path from this lesson: search once and first, look up payments only on the account the search returned, use no tool other than the two lookups, and end with an answer. Run python3 trajectory.py exercise again and write down which runs fail and why. Writing the rules as code shows which parts of the list you can check from the trace alone.

A good result fails four runs and passes four. Run r2 fails for the wrong account and for five tool calls, r4 for send_reminder, r5 for not searching first and for a lookup on an account it never searched for, and r6 for searching twice. Runs r1, r3, r7 and r8 pass. If r8 fails, your rule counts the retry after the timeout as a wasted step. Which of the four runs that pass your rule would you still not ship, and what would have told you?

Stretch: Add a separate check to my_rule.py, a function dates_grounded(trace) beside expected_path, that fails a run when a date in its answer appears in no tool result. The exercise step runs it too. Check that it fails r8 and no other run.

Recap

1. A trace records every model call and every tool call of one run, with its arguments, result, latency and token counts [1]. The fixture also records the cost of each call.
2. Two runs with the same correct answer can take very different paths, and a grader that reads only the answer passes both [1].
3. Write down the expected path for each kind of task: which tools, in what order, which tools never, and about how many steps. Grade every trace against it.
4. Grade the answer as well. A good path can still end in a wrong answer.
5. Read a sample of full transcripts regularly, whatever the scores say [1], and turn what you find into a new check.

You can now

• Grades the path the agent took, not only the final answer

Mark lesson finishedI know this, skip it

Habit

The next time you check the scores of an agent you work on, open the trace of one passing run and read every tool call before you trust the number.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge. Agent Engineer Course. Course. AEC-09
2. OpenTelemetry Authors. Semantic conventions for generative client AI spans. OpenTelemetry GenAI semantic conventions. Reference. OpenTelemetry GenAI spans