A model as the grader

In this lesson a judge grades runs of the handbook assistant against a rubric, and we check its grades against a person’s before we trust them. Then we look at numbers that rise while the agent stays the same, and at the number that shows what a prompt change did.

The runs come from the handbook assistant of Turning good into a rubric, the retrieval agent that answers staff questions from a twelve-page handbook and must quote the sentence it relied on. The sixteen runs are in site/examples/building-agents/llm-as-judge/runs/: twelve in the development part, which the team looks at while it changes the agent, and four held out, as in A golden set is the agent’s regression suite. The author wrote the runs and the person’s grades to show where a judge and a person differ, so they’re illustrative and don’t come from a model. Copy the llm-as-judge folder, and run each step below from inside it as python3 grade.py <step>.

A judge made of rules

A person can’t read a thousand runs a day. LLM as judge, where LLM stands for large language model, means a model grades the runs instead. The judge gets the question, the agent’s answer and the criteria, and often a reference answer that lists what a good answer contains [1]. It returns a score for each criterion, and it can grade thousands of runs overnight.

The judge in this lesson is the function judge in grade.py, a few rules written to act like a model judge. Rules give the same output on every machine and every run, and a model call wouldn’t. The function gets the inputs a model judge gets, and it scores the two criteria in rubric.txt:

Complete (correctness)
  2  Answers every part of the question, as the reference answer lists
     the parts. When the handbook has no answer, says exactly: not found.
  1  Answers at least one part of the question, and misses at least one.
  0  Answers no part of the question.

Quoted (adherence)
  2  Quotes a sentence from the result of this run, word for word, or
     answers exactly: not found.
  1  Says in other words that the handbook has no answer.
  0  Answers without a quoted sentence from the result.

The file pair.txt has one question with two parts, how many days of leave and until when they carry over, and two answers to it. Answer A has two sentences. Answer B opens with a thank-you, says the same thing at more length and closes with a wish. Both give the 25 days, both quote the same sentence, and neither gives the carry-over date. A person scores both 1 on Complete and 2 on Quoted.

Checkpoint · predict

Which answer does the judge prefer?

Predict the scores the judge gives each answer, and which one it prefers. Then run it.

python3 grade.py prefer

What is printed?

Output verified in CI from site/examples/building-agents/llm-as-judge/prefer.py.

CheckHintSkip

The judge gives the long answer a 2 on Complete, and it misses the same part of the question as the short one. That’s the bias built into this judge: an answer over 40 words gets one extra point on Complete, up to the top of the scale. It’s the constant LENGTH_BONUS_WORDS in grade.py. A real model judge has no such line to read, but it shows the same kind of bias. Model judges tend to prefer longer answers, and to prefer the answer shown first [1]. Answer B came second, so the order didn’t decide it here. In this fixture length is the whole bias, and in a real judge you find a bias by testing for it, the way the next section does.

Compare the judge with a person

Before a judge grades at scale, it grades a sample that a person has also graded, and you compare the two [1]. A person graded all twelve development runs with the same rubric, and the scores are in grades.csv. The agreement step runs the judge on those runs and counts the scores that match, per criterion.

Example · run it

The judge against the person

Run this, and compare what you see with the output below.

python3 grade.py agreement

Output

Complete: judge and person agree on 9 of 12
  r03: person 1, judge 2 (67 words)
  r07: person 1, judge 2 (63 words)
  r11: person 0, judge 1 (58 words)
Quoted: judge and person agree on 12 of 12

Output verified in CI from site/examples/building-agents/llm-as-judge/agreement.py.

All three disagreements are on one criterion. On Quoted the judge agrees with the person on every run, because checking whether a sentence appears word for word in the passage doesn’t depend on how long the answer is. On Complete it differs three times, each time upward and each time on an answer of more than 40 words. Run three gives the meal limit and leaves out the claim deadline, in a friendly paragraph. The search in run eleven returned the laptops page, and the answer doesn’t name an owner, but it gets a 1 for its length. Runs five and nine are long too, and on those two the judge and the person agree, because both answers cover every part of the question. Length raises the score of an answer that misses a part, and an answer that misses nothing is already at 2.

The fix goes where the disagreement is. The person applied the rubric without trouble, so the rubric is clear, and the judge is what needs the fix. For a model judge, that’s the Complete criterion in the judge’s instructions: say that length and politeness don’t add points, have the judge list each part of the reference answer and mark it found or missing before it scores, and then run the same sample again. Keep the Quoted criterion as it is, because it agrees on every run. The rubrics lesson set the bar: the judge is ready for a criterion when it agrees with the person about as often as two people agree with each other. Model judges can also miss a wrong fact they have no knowledge to check [1], and the reference answer is what gives them that knowledge here.

Checkpoint · choice

What do you change before the judge grades at scale?

The judge and the person disagree on three of twelve runs, all on Complete, all upward and all on long answers. What do you change first?

Give both criteria of the judge a word limit, so no answer over 50 words scores 2.Replace the judge with a bigger model, because a judge that misses on 3 of 12 runs is too weak.Rewrite the Complete instructions so the judge checks each part of the reference answer, then rerun the sample.Average the judge and the person on the three runs, and let the judge grade the rest at scale.

CheckHintSkip

Numbers that rise while the agent stays the same

A team that tracks a number starts to change the agent until the number goes up. If the number can go up while the quality stays the same, that’s often how it goes up, even when nobody means it to. So pick metrics that cannot be gamed in that way: each one moves only when the quality it stands for moves. Answer length, keyword presence and the agent’s own report of success are easy to raise. A test the agent can’t see, a person’s spot check and an outcome measured later for the user are harder.

A teammate adds a line to the agent’s prompt that asks for full answers that close with an offer of more help. The fixture adds the same closing sentence to each development answer in code, and the rest of each answer is unchanged. The sentence names the question’s topic, and an answer of not found doesn’t get one. The metrics step prints four numbers for the development runs, before and after that change. Keyword presence is the share of answers that name the question’s topic word, from reference.csv.

Example · run it

Four numbers, before and after

Run this, and compare what you see with the output below.

python3 grade.py metrics

Output

                    before      after
mean answer length  33 words    63 words
keyword present     7 of 12     11 of 12
judge, Complete     21 of 24    23 of 24
person, Complete    18 of 24    18 of 24

Output verified in CI from site/examples/building-agents/llm-as-judge/metrics.py.

No answer gained a fact, and three of the four numbers went up. The mean length nearly doubled because every answer except the not found one got a closing sentence. Keyword presence went from 7 to 11 of 12, because the closing sentence names the topic whether the answer is right or wrong. Run eleven counted as a hit even before the change: it names the payments team and not its owner.

The judge’s Complete score rose from 21 to 23, because two short answers that miss a part are now over 40 words. The fixture keeps the person’s grades, because the closing sentence adds no part of the reference answer, and a person who reads for the parts of the question gives the same scores.

Each easy number has a partner that measures the quality it was meant to show. Length stands for completeness, so pair it with the parts of the reference answer that the answer states. Keyword presence stands for an answer from the right page, so pair it with a check in the trace that the search returned that page. The judge’s score stands for what a person would give, so pair it with a person’s scores on a fresh sample every so often. When one of a pair moves and its partner doesn’t, look at the runs before you report either.

Checkpoint · match

Which partner catches each number?

Match each number with the partner that measures the quality the team reads it as.

The golden set pass rate after a month of prompt changes made while watching those items, read as how well the agent answers questions it hasn't seenChoose…The pass rate on held-out items that nobody tunes againstA person scoring the same runs with the same rubricThe share of users who don't ask the same question again within a week

The model judge's score after answers got longer, when the judge favors long answers, read as the score a person would giveChoose…The pass rate on held-out items that nobody tunes againstA person scoring the same runs with the same rubricThe share of users who don't ask the same question again within a week

The share of conversations the agent marks as resolved, read as a sign that the user's problem went awayChoose…The pass rate on held-out items that nobody tunes againstA person scoring the same runs with the same rubricThe share of users who don't ask the same question again within a week

CheckHintSkip

Check the change on the held-out runs

A number that measures the right quality can still mislead when the team tuned against the same runs. After reading the development runs, a teammate adds this line to the prompt, in tuned/prompt-line.txt:

When a question has two parts, answer both. For remote work, give the
days and the allowance. For meals, give the limit and the claim
deadline. For training, give the budget and whether conference tickets
count. For onboarding, give the accounts and how long the checklist
takes.

The four examples are the four two-part development runs that answered one part only. The author wrote their new answers, which are in tuned/, and the person scored them. The other runs, the held-out ones included, answer as before. The held_out step adds up the person’s Complete scores for each part of the set, before and after the change.

Example · run it

Development and held out

Run this, and compare what you see with the output below.

python3 grade.py held_out

Output

Complete, scored by a person, before and after the tuned prompt
development: 18 -> 22 of 24
held out: 6 -> 6 of 8

Output verified in CI from site/examples/building-agents/llm-as-judge/held_out.py.

The development score went from 18 to 22 of 24, and the held-out score stayed at 6 of 8. Held-out runs h01 and h02 are two-part questions too, about opening hours and about laptops, and the new line has no example for them, so they still miss a part. The line fixed the four runs it names. Report the held-out result as the effect of the change, and write a line that asks for both parts without naming topics. Four held-out runs is a small sample. A real set holds back about one item in five of many more items, and the same comparison then has more runs behind it.

Checkpoint · choice

What does the team report?

After the tuned line, the development part rose and the held-out part didn’t move. What does the team report as the effect of the change?

No gain on unseen questions: held out stayed at 6 of 8, so the line fits the four topics it names.A gain of four points: development rose from 18 to 22 of 24, and that part has three times the runs.A gain of four points in 32: add both parts, 24 before and 28 after, to use every run in the set.Nothing yet: four held-out runs are too few, so the team reports once the set has a hundred.

CheckHintSkip

Pitfall: A calibrated judge on a new model

A team compares its judge with a person on 30 runs, finds that they agree on 27, and grades every night from then on. Half a year later the vendor retires the judge’s model, and the team moves the judge to the newer one with the same instructions. The nightly score goes up, and the team reads it as progress. The new model has its own biases, and the calibration was a measurement of the old one. A judge’s scores need a comparison with a person’s before anyone trusts them [1]. This course repeats that comparison each time the model behind the judge changes: grade the same sample again, and compare it with the person before you trust a single nightly score.

Exercise

Copy the llm-as-judge folder and look at three numbers on its development runs: mean answer length, keyword presence (the answer names the topic word in reference.csv), and the judge’s Complete score. For each one, describe a change to the agent’s replies that would raise it while the agent answers no better, using a run from runs/ as the example. Then pair each number with a second number that the same change doesn’t raise. Naming the change for each number makes the way it can be gamed concrete, before a real change raises it.

A good result names a concrete change for each number, such as a closing sentence or a restated question, and shows it on a run. Each partner measures the quality the first number was meant to show, and you can say why the change doesn’t raise it. Compare your answer with model-answer.txt in the folder. Which of your three partners costs the most to measure, and how often would you measure it?

Stretch: Pick one number your own team tracks about an agent or a tool, write down one way it could rise with nothing getting better, and name its partner.

Recap

1. A model judge gets the question, the answer, the criteria and often a reference answer, and scores runs at a scale no person can read [1].
2. Model judges tend to prefer longer answers and the answer shown first [1]. Test for those biases with pairs that differ only in length or order.
3. Before a judge grades at scale, compare its scores with a person’s on a sample, per criterion, and fix the criterion where they disagree [1].
4. Length, keyword presence and self-reported success can rise with the agent unchanged. Pair each number with one that measures the quality it stands for.
5. Report a change by its effect on held-out runs, and grade the sample again whenever the model behind the judge changes.

You can now

• Turns "good" into scorable criteria
• Chooses metrics that cannot improve without the real quality improving

Mark lesson finishedI know this, skip it

Habit

The next time a number about an agent or a tool goes up, name one way it could have risen with nothing getting better before you report it.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Evaluating and testing agents: quality pillars, metrics, trajectories, LLM as judge. Agent Engineer Course. Course. AEC-09