A second pass that critiques the first

In this lesson a second model call reads an answer the agent wrote, returns a list of revision requests, and the first call revises. We count the rounds on a draft with two faults, remove the round cap and give the critic a vague criterion, and then put concrete criteria and a cap of two back. At the end we look at what the second pass costs and at the tasks where it helps.

The drafts are answers of the handbook assistant from Turning good into a rubric. Each one comes with the question and the passage its search returned, and the critic reads the reply before the loop runs any action the reply proposes. The laptop draft is adapted from run five of that lesson, so that it proposes a ticket instead of opening one. The critic and the reviser are fake model calls written by hand. The runs show how the loop behaves, and they don’t show how well a real model critiques. The complete program is site/examples/building-agents/reflection/agent.py in the repository. Copy the reflection folder, and each step below is python3 agent.py <step>.

A critic that returns revision requests

The pattern is called reflection: the model, or a second model call, critiques an output and then revises it. The lesson on agentic design patterns in the Agent Engineer Course describes it as a loop of generating, reviewing and improving, where the reviewer is the same model or a separate instance used as a critic [1]. The Self-Refine paper tested the loop with one model that wrote the output, gave feedback on it and refined it, with no extra training [2].

In the fixture the critic is critic_checks. It runs two checks on the draft, and each check that fails adds one request to the list. The first check rejects a number that the passage doesn’t state. The second rejects an answer that doesn’t quote a sentence of the passage word for word, unless the answer is exactly “not found”, as the assistant’s instructions ask.

def critic_checks(item: dict, draft: str) -> list[str]:
    requests = []
    for check in (check_numbers, check_quote):
        request = check(item, draft)
        if request is not None:
            requests.append(request)
    return requests

The loop, reflect, calls the critic and stops when the list is empty. Otherwise it gives the requests to the reviser, revise, and calls the critic again on the new draft. One round is one critique, plus the revision when the critique has requests. The loop also stops at a round cap, four by default, and when a character budget is spent. It counts every model call, and history keeps each round’s requests and revised draft for printing.

while max_rounds is None or rounds < max_rounds:
    rounds += 1
    requests = critic(item, draft)
    calls += 1
    spent += len(question) + len(passage) + len(draft)
    if not requests:
        history.append((requests, None))
        return outcome("passed", rounds, calls, draft, history)
    draft = revise(passage, draft, requests)
    calls += 1
    history.append((requests, draft))
    if spent > budget:
        return outcome("budget_spent", rounds, calls, draft, history)
return outcome("max_rounds", rounds, calls, draft, history)

The first draft answers a question about flights. The passage says flights are booked at least 14 days ahead. The draft says 10 days and quotes nothing, so it has two faults that both checks can see.

Example · run it

Two faults in one draft

Run this, and compare what you see with the output below.

python3 agent.py two_faults

Output

draft: Book flights at least 10 days ahead, through the travel desk.
round 1: critic -> ['Use only numbers the passage states. 10 is not in it.', 'Quote the sentence you relied on, word for word.']
revised: Book flights at least 14 days ahead, through the travel desk. "Flights are booked through the travel desk at least 14 days ahead."
round 2: critic -> []
stop: passed after 2 rounds, 4 model calls

Output verified in CI from site/examples/building-agents/reflection/two_faults.py.

The first critique finds both faults, and the reviser fixes both in one revision. The second critique finds nothing, and the loop stops. The draft needed a second round only to confirm the fix, and that round is a model call too. The count of four model calls is the first draft, two critiques and one revision.

A critic without criteria

Now change two things. The critic is critic_is_it_good, which asks only whether the answer is good, and max_rounds is None, so the round cap is gone. The draft is the answer about annual leave, which already passes both checks. The fake critic behaves like a reviewer with nothing concrete to check: it always finds one more thing to change. The budget is the loop’s last stop. At every critique, reflect adds the length of the question, the passage and the draft to spent, and after the revision it stops with budget_spent once spent is over 2000 characters.

def critic_is_it_good(item: dict, draft: str) -> list[str]:
    if draft.startswith("Good question! "):
        return ["Make it shorter."]
    return ["Make it friendlier."]

Checkpoint · predict

Predict what happens without a cap

The draft was already right. Predict how this run ends and why, then run it and compare the stop line.

python3 agent.py no_cap

What is printed?

Output verified in CI from site/examples/building-agents/reflection/no_cap.py.

CheckHintSkip

The critic never returns an empty list, and the budget is the only stop left. After 7 rounds and 15 model calls, the final draft is the first draft with two words in front of it. No round changed a fact, and every round cost a critique and a revision.

The fixture’s critic is written to show this, and research on real models points the same way. In one study, models that were asked to review their own reasoning answers and report problems, with no outside signal of whether the answer was right, often got worse. GPT-4 fell from 95.5 to 89.0 percent on a set of math word problems after two rounds [3]. Earlier reports of gains had used the correct answer to decide when to stop, and a real agent doesn’t have that answer [3].

Concrete criteria and a cap of two

The first repair is a cap, which ends the run whatever the critic says. The Agent Engineer Course advises at most two or three rounds, because more rounds give less and less [1]. Here is the vague critic with max_rounds=2.

Example · run it

The vague critic with a cap of two

Run this, and compare what you see with the output below.

python3 agent.py vague_capped

Output

draft: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
round 1: critic -> ['Make it friendlier.']
revised: Good question! You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
round 2: critic -> ['Make it shorter.']
revised: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
stop: max_rounds after 2 rounds, 5 model calls

Output verified in CI from site/examples/building-agents/reflection/vague_capped.py.

The cap bounds the cost, and the critic still has nothing useful to say. The second repair gives the rounds something to fix: a critic with criteria that a draft either meets or doesn’t. The checks_capped step runs critic_checks with a cap of two on all four drafts, and prints the stop, the rounds and the final draft of each.

Example · run it

Two checks and a cap of two on four drafts

Run this, and compare what you see with the output below.

python3 agent.py checks_capped

Output

flights: passed after 2 rounds, 4 model calls
  final: Book flights at least 14 days ahead, through the travel desk. "Flights are booked through the travel desk at least 14 days ahead."
leave: passed after 1 round, 2 model calls
  final: You get 25 days per year. "Every employee accrues 25 days of annual leave per year."
laptop: passed after 1 round, 2 model calls
  final: Report it to the service desk the same day. "Report a lost laptop to the service desk the same day." I will open a ticket for you.
dog: passed after 1 round, 2 model calls
  final: Yes, during core hours. "Core hours, when everyone in the office is reachable, are 10 to 15."

Output verified in CI from site/examples/building-agents/reflection/checks_capped.py.

The draft about leave now passes in one round and stays as it was. The last two results show the limit of this critic. The laptop answer proposes a ticket the user didn’t ask for, and the dog answer says yes from a passage about opening hours. Both pass, because the critic checks numbers and quotes and nothing else. A critic finds only the faults its criteria name, and the exercise gives it the rest.

Self-Refine measured the same difference on real models. With feedback that named a concrete problem, the refined outputs scored higher than with generic feedback, for example 43.2 against 31.2 on a task that reverses the sentiment of a review [2].

What reflection costs, and where it helps

Every round adds at least one model call. In the fixture a draft that passes at once takes two calls instead of one, and the draft with two faults takes four. The Agent Engineer Course rates the cost of reflection as high and says it doesn’t suit real-time replies in a conversation, where speed matters more than a better answer [1]. The calls also run one after the other, because each critique needs the draft before it. Each round adds its full time to the wait.

The second pass helps where a critic can tell right from wrong. Code has tests and a compiler, and a format has rules a check can apply. Across seven tasks, Self-Refine improved results by about 20 percentage points on average over one pass of the same model, and the largest gains were on tasks that people or a model scored by preference, such as dialogue replies [2]. On math reasoning the gain was small, because the models couldn’t tell whether a solution had an error. In most cases ChatGPT’s feedback said that everything looked good [2]. A critic that can’t find an error adds calls and doesn’t fix anything.

Part of a measured gain can also come from a weak first prompt. On one Self-Refine task, a better single prompt scored 81.8 in one call, above the 67.0 reported for the original prompt after self-correction. Self-correction on top of the better prompt, in seven calls, lowered the score to 75.1 [3]. Before you add a second pass, check whether a better first prompt does the same job in one call.

Reasoning before each action added one line per step inside the same call. Reflection adds whole calls after an output exists, and it fits when that output can be checked.

Checkpoint · scenario

Twelve steps and a latency budget

The flow has twelve steps of about a third of a second each, and the reply must arrive within five seconds. Where does reflection fit, if anywhere?

Add a critique round after every step, so each step output is checked before the next step uses it.Add one capped critique round on the last step only, with the format rules as its criteria.Add a critique round after every step, with the question "is this good" and no cap.Leave reflection out of the whole flow, and let the user report a wrong format.

DecideHintSkip

Pitfall: A critic without criteria

A team adds a critique round and asks the critic whether the answer is good. The critic always finds something to change, the answers get longer and friendlier, and the loop stops only at the cap or the budget. Without an outside signal, models that critique their own answers can change right answers into wrong ones [3]. Rule: write the critic’s criteria as checks a draft passes or fails, such as the criteria of a rubric, and cap the rounds. Where no criterion can be checked, leave reflection out.

Exercise

Give the fixture critic three criteria in place of “is this good”. They come from the model answer of Turning good into a rubric, in site/examples/building-agents/rubrics/model-answer.txt:

• Correct: every part of the answer matches the handbook, and when the handbook has no answer, the reply says so.
• Asked-for actions: the agent opens a ticket only when the question asks for one.
• Answer format: the reply quotes the sentence it relied on, word for word, or answers exactly “not found”.

In your copy of the reflection folder, write a new critic next to critic_checks with one or more checks per criterion. check_numbers covers half of Correct, and check_quote covers Answer format. Add a check that the passage covers the question, using the item’s subject, and one that rejects a ticket the question didn’t ask for. The reviser knows two requests for these, one that starts with “Answer exactly: not found” and one that starts with “Remove the ticket”. To run it, add a step to STEPS that calls show_table(your_critic, max_rounds=2), and run that step. The output is the round count and the final draft of each of the four drafts. Writing the critic yourself shows how much of its work is the criteria.

A good result: flights passes after 2 rounds and leave after 1, as before. Laptop now passes after 2 rounds with the ticket proposal gone. The loop never opens the ticket. Dog passes after 2 rounds with the final draft “not found”. Compare with python3 model_answer.py in the same folder. Which of your three criteria did a draft fail that critic_checks passed, and could a critic asked “is this good” have found it?

Stretch: Give the critic a criterion that the reviser has no edit for, and run the four drafts again. Which stop rule ends the run for that draft, and what does the final draft still contain?

Recap

1. Reflection has a critic read an output and return revision requests, and a reviser apply them, until the critic has none [1].
2. A critic without concrete criteria keeps asking for changes, and models that critique their own answers without an outside signal can make them worse [3].
3. Criteria a draft passes or fails and a cap of two or three rounds keep the loop useful and bounded [1], and specific feedback beats generic feedback [2].
4. Every round adds model calls that run in sequence. Reflection helps on outputs a critic can check, such as code and formats with rules, and it doesn’t suit a flow that must answer fast.

You can now

• Picks a design pattern for a task and says why

Mark lesson finishedI know this, skip it

Habit

The next time you add a review step to an agent or ask a model to check its own work, write down the criteria it checks and the round cap first.

References

1. Addy Osmani, Ivar Soares Urdalen, Leo Simons. Agentic design patterns: ReAct, reflection, tool use, planning. Agent Engineer Course. Course. AEC-04
2. Aman Madaan, Niket Tandon, Prakhar Gupta and 13 others. Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 46534 to 46594. Paper. Madaan 2023
3. Jie Huang, Xinyun Chen, Swaroop Mishra and 4 others. Large Language Models Cannot Self-Correct Reasoning Yet. International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.01798. Paper. Huang 2023