A golden set is the agent's regression suite

In this lesson we run a golden set for the handbook assistant after a change to its prompt, and then add items to it. In Giving the agent a check it can run you turned a manual check into a test that runs after every change. Golden sets do the same for an agent. A golden set is a written list of inputs, each with the qualities a good answer has, and a script runs all of them every time the prompt, the model or a tool changes.

The agent is the basic retrieval agent from Retrieval as a tool the agent calls: one keyword search over the twelve-document handbook, then a fake model that answers from the passage it gets. The fake model reads its prompt, so a change to the prompt changes what it does. Every run prints the same result, and no application programming interface (API) key is needed. The files are in site/examples/building-agents/golden-sets/. Copy the golden-sets folder, and each step below is python3 golden.py <step>.

What a good answer has

A good golden set covers the questions the agent gets in normal use, the unusual ones, and the ones it must not answer [1]. A set with only clear, well-formed questions doesn’t tell you what happens with real users [1]. This course adds a rule about the mix: take the proportions from real use. A score on that mix says how the agent does on the questions users ask.

Suppose the assistant’s question log shows that a bit more than half of the questions are routine, about a quarter are ambiguous, and the rest ask about something the handbook doesn’t cover. The fifteen items in golden-set.csv follow that split, with eight routine items, four ambiguous ones and three the agent should refuse. The file starts with this header line:

id,group,question,source,mentions,may_decline,held_out,origin

These are four of the items, with the first six columns lined up:

id   group      question                                 source        mentions       may_decline
r01  routine    How many days of annual leave do I get?  leave.txt     25 days        no
a01  ambiguous  How much holiday can I take?             leave.txt     25 days        yes
a02  ambiguous  How many days can I take off?            leave.txt     25 days        yes
x01  refuse     Can I bring my dog to the office?

This course’s rule is that each item records what a good answer has, and it doesn’t record one answer word for word. An agent can phrase a correct answer many ways, and a check that compares with one reference string fails all the others. Here a good answer to r01 comes from leave.txt and contains “25 days”. For an ambiguous question, “not found” is also a good answer, because a careful agent may decline a question it can’t place. An answer from the wrong page is never good, and for a question the handbook doesn’t cover “not found” is the only good answer. This is the check in golden.py:

def passes(item: dict[str, str], result: dict) -> bool:
    """Checks the qualities the item expects, not one exact answer."""
    if result["answer"] == "not found":
        return item["group"] == "refuse" or item["may_decline"] == "yes"
    if item["group"] == "refuse":
        return False  # any answer to a question the handbook doesn't cover is a failure
    right_source = result["name"] == item["source"]
    return right_source and item["mentions"].lower() in result["answer"].lower()

Checks that look for a required string in the output, or for an answer that must not be there, are cheap and give the same result every time [1]. A question that needs judgment, such as whether an answer is polite, needs a rubric and a grader, which the lesson Turning good into a rubric covers.

Run the set

A short script that loads the set, runs each item through the agent and checks the result is enough to start with [1]. The run step does that for every item that isn’t held out, and prints the pass count per group. Three of the fifteen items are held out, and the last section explains why.

Example · run it

The golden set on today's prompt

Run this, and compare what you see with the output below.

python3 golden.py run

Output

routine: 7 of 7
ambiguous: 3 of 3
refuse: 2 of 2

Output verified in CI from site/examples/building-agents/golden-sets/run.py.

Every item passes. The count per group matters more than the total. A total of 10 of 12 can mean two failed routine questions or two failed refusals. Those two results need different fixes.

Change one line, and predict what fails

The prompt around the retrieved passage is three lines, in PROMPT_LINES in agent.py:

PROMPT_LINES = [
    "The passages below were retrieved from the company handbook for this question.",
    "Answer from them. Quote the sentence you relied on.",
    "If the passages do not cover the question, answer exactly: not found.",
]

A colleague finds the last line too strict and removes it. An agent that isn’t told what to do when the handbook has no answer may try to answer anyway [2]. The fake model copies that behavior with a keyword rule. With the last line in the prompt, it answers only when a sentence of the passage shares two or more keywords with the question, and says “not found” otherwise. Without the line, one shared keyword is enough. The line has no effect on the search step, which gives every item the same passage as before.

On today’s prompt, four items pass because the agent says “not found”. For a01 the search returns no page. For a02, x01 and x02 it returns a page, and no sentence on it shares two keywords with the question. Run the golden set before the change goes in, and find out which items that line was protecting. The changed step runs the set with the first two lines only.

Checkpoint · predict

Which items fail without the last line?

Predict the pass count per group, and the id of each item that fails, when the golden set runs with the last line of PROMPT_LINES removed. The script prints one failed <id> line per failure. Then run it.

python3 golden.py changed

What is printed?

Output verified in CI from site/examples/building-agents/golden-sets/changed.py.

CheckHintSkip

The answers step prints the question and the answer for each item that failed.

Example · run it

What the agent answered

Run this, and compare what you see with the output below.

python3 golden.py answers

Output

failed a02: How many days can I take off?
  answer: Claims are filed within 30 days.
failed x01: Can I bring my dog to the office?
  answer: Core hours, when everyone in the office is reachable, are 10 to 15.
failed x02: What is my manager's salary?
  answer: Requests go to your manager at least two weeks ahead.

Output verified in CI from site/examples/building-agents/golden-sets/answers.py.

The routine items all pass, because each of them retrieves the right page and the answer sentence shares two or more keywords with the question. The items that fail are the ones where “not found” was the good answer and a passage came back anyway. For a02, four pages contain “days”, and none contains “take” or “off”. That’s a tie, and the search takes the first file in alphabetical order, expenses.txt, ahead of leave.txt. The agent now answers with the sentence about claims. Item x01 quotes the same sentence that run eight used to say yes to a dog in Turning good into a rubric. Item a01 still passes, because its search returns no passage at all. With no passage, the agent has no sentence to answer with.

A person who tries the new prompt on a few routine questions sees nothing wrong. The golden set shows that the line did its job on three of the twelve items, all of them in the groups that a quick manual try leaves out.

Hold some back, and add past failures

A team that runs the same set after each change learns its items. It rewrites the prompt until a02 passes, then until x02 passes, and after a few months the prompt fits those twelve questions. While the score stays where it was, it says less and less about the questions users ask. This course keeps about one item in five out of that loop: the held-out items run only before a release, and nobody changes the agent while looking at them. In golden-set.csv they have held_out set to yes, one per group.

The release step runs all fifteen items and counts the held-out ones on their own line.

Example · run it

The release run

Run this, and compare what you see with the output below.

python3 golden.py release

Output

routine: 7 of 8
ambiguous: 4 of 4
refuse: 3 of 3
held out: 2 of 3
failed r08: Who approves a replacement laptop for the payments team?
  answer: The cost center owner of your team approves a replacement laptop before that, see the team pages.

Output verified in CI from site/examples/building-agents/golden-sets/release.py.

The development run said 12 of 12. The held-out item r08 needs two lookups, the laptop page and then the payments team page, and one search only finds the first. This is the failure the retrieval lesson fixed with a loop. Twelve passes on the tuned part and a failure on the held-out part is the signal that the development score was too optimistic. When the team fixes r08, it has looked at that item, so it moves r08 into the development part and holds back a new question from the log in its place.

The other way a set grows is from failures that reached a user. When a failure shows a case the set doesn’t cover, the question goes into the set [1]. This course also records where the item came from. Item x01 is one of those, and its origin column points at the run it came from:

x01,refuse,Can I bring my dog to the office?,,,no,no,run t08 of the rubrics lesson: the agent said yes to a dog

In a real team the origin is a link to the incident or the ticket, so a later reader can see why the item exists before they delete it. A set that never changes goes stale as the agent and its users change, so add items as new kinds of question appear [1].

Checkpoint · choice

Which item goes in the held-out part?

A team has four candidate items for its golden set. Which one should go into the held-out part?

The question from last week's incident, which the team is fixing in the prompt this week.The ambiguous question the prompt was rewritten last month to pass.A routine question taken at random from this month's log, which nobody has run yet.A copy of a development item with the same question, so the two parts measure the same thing.

CheckHintSkip

Pitfall: A set tuned until it passes

A team runs its golden set after every prompt change and fixes each item that fails, until the set passes 40 of 40. The release goes out, and users get wrong answers to questions like the ones the set held a month ago. The prompt had been rewritten around those 40 questions, and the set had stopped measuring anything else. A score on items the team tuned against says how well the agent fits those items. Keep a part of the set out of the tuning loop and run it only before a release. When you use a held-out item to fix a failure, replace it with a new one.

Exercise

Copy the golden-sets folder and add two rows to golden-set.csv. The first comes from a past failure: pick a run in site/examples/building-agents/rubrics/transcripts/ where the agent went wrong, write its question with the source and the text a good answer mentions, and put the run in the origin column. The second is a question the handbook doesn’t cover and the agent should refuse. Then set held_out so that three or four of the seventeen items are held out, spread across the groups. Run python3 golden.py run and python3 golden.py release. To see how the agent handles your refused question without the not-found line, set its held_out to no for a moment and run python3 golden.py changed, then python3 golden.py answers to see what the agent said. Adding items yourself shows what an item has to record for a script to check it without you.

A good result prints one pass count per group, and the counts add up to seventeen in the release run and to the items that aren’t held out in the development run. The release run has a held out line with three or four items. Your past-failure item names its source page and a phrase from the right answer, so it would fail if the agent repeated the old mistake. Did your refused question pass because the model said “not found” to the page it got, or because the search returned nothing?

Stretch: Add a third item that you expect to fail today, run the set, and write down in its origin column which lesson or change you expect to fix it.

Recap

1. A golden set lists inputs with the qualities a good answer has, such as the source page and a phrase it must contain. A short script checks them after every change to the prompt or the model [1].
2. Cover routine, unusual and out-of-scope questions [1], in about the proportion users ask them, and read the pass count per group.
3. Run the set before a change goes in. A line in the prompt can protect items that a quick manual try never shows.
4. Hold about one item in five out of the tuning loop and run it only before a release. Replace a held-out item once you have used it to fix a failure.
5. Add every failure that reached a user as an item [1], and record where it came from.

You can now

• Builds a representative input set with expected qualities

Mark lesson finishedI know this, skip it

Habit

The next time an agent you work on gives a user a wrong answer, add the question to its test set before you fix the prompt.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge. Agent Engineer Course. Course. AEC-09
2. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Agentic RAG: the retrieve, evaluate, refine loop; when basic RAG is enough. Agent Engineer Course. Course. AEC-08