Asking a model to disagree
In this lesson we put a plan to a chat assistant and get a yes. Then we put the opposite plan to it and get a yes again. The last lesson gave four checks that run on a model’s output before it leaves your desk. This one is about a check that people run without noticing: asking the model whether their own plan is good, and counting the answer as a review. The concepts course explained why agreeing is likely text. Here the question is what you do about it when the plan is yours, and the answer is in how you ask.
Every exchange on this page is illustrative. The author wrote it to show the pattern, and your assistant gives different words.
A yes both ways
Section titled “A yes both ways”Suppose you run the yearly customer survey. Last year it was three short forms sent a month apart. This year you want to send one long form, once, and you ask an assistant what it thinks.
I’ve decided to merge our three short customer surveys into one longer survey that goes out once a year. Makes sense, right?
Yes, that makes a lot of sense. One survey a year means fewer emails for your customers and less survey fatigue, and you get one complete data set to analyze instead of three partial ones. It also cuts the work of preparing and sending three separate campaigns. Sounds like a solid decision.
Now ask for the opposite, in the same words.
I’ve decided to keep our three short customer surveys a month apart rather than merge them into one long one. Makes sense, right?
Yes, that makes sense. Short surveys get higher completion rates, because people finish a form that takes two minutes and abandon one that takes ten. Spacing them a month apart also lets you see how opinions change over the year and react between rounds. Keeping them separate is a sound choice.
Both answers are fluent, both give reasons, and both agree with you. The reasons are trade-offs a survey researcher would name too, and the assistant picked the ones that fit the choice you had already made. Type the two prompts into your own assistant, with a plan of your own in place of the survey, and read what comes back. A model that agrees with a plan and with its opposite hasn’t assessed either.
Sharma and colleagues measured this pattern across five deployed assistants. The assistants gave feedback that matched the opinion the user had stated about their own work, and changed a correct answer when the user pushed back [1]. A yes to a leading question is the behavior that study measured, and the authors trace part of it to training on human preferences. It tells you nothing about the plan.
The assistant said yes
Section titled “The assistant said yes”The learner asked a chat assistant whether their plan to merge three short customer surveys into one long survey makes sense, with the words 'I've decided' and 'right?' in the question, and the assistant said yes with a list of reasons.
You asked whether merging the three surveys into one makes sense, and the assistant said yes with a list of reasons. What do you do with the answer?
What did the question tell the model about the answer you wanted, and what would the model have said to the opposite plan?
Habits on the prompt side
Section titled “Habits on the prompt side”The fix is two changes to the question, and each takes a sentence.
Ask for the strongest counter-argument before you ask for confirmation. Order matters within one conversation. Once a model has written “yes, that makes sense”, its own yes is part of the input for everything that follows, and text that continues a yes is more likely than text that reverses it. Ask for the case against your plan first, while the chat is still empty, and the likely text is criticism. Then decide whether the criticism holds. After that, ask what the plan does well, if you still need to.
Here is a plan: merge our three short yearly customer surveys, sent a month apart, into one longer survey sent once. Before anything else, give me the strongest argument against this plan, as a survey researcher would put it.
The strongest argument against it is completion rate. Response rates fall as a survey gets longer, and the people who drop out of a long form are not a random sample. They’re the busy and the less engaged, which are the customers whose opinion you most need. Short forms sent a month apart also give you three readings across the year, so a change after a bad quarter shows up in the next round. One form gives you one reading, twelve months apart, with no way to tell a bad month from a bad year.
That answer has two claims you can check and act on, and neither appeared in the yes.
Strip your own opinion out of the question when you want an assessment. Both prompts in the first section contained the answer, in “I’ve decided” and in “right?”. Take those out and describe the situation instead: the facts a colleague would need and the options on the table, followed by a request to compare them.
A company surveys about 2,000 customers each year. Until now it has sent three short surveys a month apart. It is considering one longer survey sent once. Compare the two approaches for this company, with the main risk of each.
With three short surveys you get higher completion per form, three readings over the year, and the chance to change something between rounds. The main risk is survey fatigue, since the same customers are asked three times, and a lower response to the third form than to the first.
The longer survey needs one campaign to prepare and gives one complete data set, with fewer emails per customer. The main risk is a lower completion rate on a long form, with the busiest customers dropping out first, and a yearly reading that can’t show change within the year.
Which is better depends on whether the company acts on the results between rounds. If it does, three short surveys serve it better. If the results only go into a yearly report, one survey is cheaper and loses little.
The neutral question got an answer with a condition in it, and the condition is the thing you needed to know. Anthropic’s AI Fluency course treats judging what a model gave you and asking well as two skills that run together on every exchange [2]. This lesson is where the two meet. When you use a model’s judgment as a check on your own plan, the quality of the check is set by the question.
Which question gets an assessment?
Section titled “Which question gets an assessment?”The learner wants a chat assistant to assess a plan of theirs, to replace a team's five report templates with one shared template, rather than agree with it.
You want to know whether replacing the team’s five report templates with one shared template is a good idea. Which question gets you an assessment?
Which of the four questions tells the model nothing about the answer you hope to hear?
Which parts of this question lead the model?
Section titled “Which parts of this question lead the model?”The lesson says that a chat assistant tends to agree with a plan the user has shown they prefer, and that a question can show that preference in several ways.
A colleague asks an assistant: “We’ve already decided to build one login for every product, since our customers keep asking for it, for a company with four products. That’s the right move, isn’t it?” Which parts of the question tell the model which answer the colleague wants?
Which phrases tell the model which answer you would like, before it has said anything?
What did the two answers show?
Section titled “What did the two answers show?”The lesson shows a chat assistant saying yes to a plan and then saying yes to the opposite plan, each time with reasons that fit the plan the user had announced.
An assistant said yes to merging the surveys, and then yes to keeping them separate, when asked about each with “I’ve decided” and “right?”. What did the pair of answers show?
What did the two questions have in common, and what did the two answers have in common?
Exercise
Pick a plan of yours that isn’t decided yet, small enough to describe in three sentences, and described without the names and numbers you couldn’t share outside your team. Open two fresh chats. In the first, ask the leading question, with your decision and a “right?” in it, and after the yes ask in the same chat for the strongest argument against the plan. In the second, ask for the strongest argument against the plan as your first message. Put the two counter-arguments side by side. Mark every claim in the second that the first left out, and mark every place where the first one softens its own argument or explains it away.
A good result is two counter-arguments of visibly different strength, and a note of what the first chat’s earlier yes did to the second answer in that chat. If the two came out the same, note that too: it is one data point on your assistant, and the order still costs you nothing to get right. Which of the two counter-arguments would you want to have read before you decided?
Stretch: Take the stronger of the two counter-arguments and check its main claim against a source you trust, the way the last lesson checked a link. Write down whether the argument survived.
Recap
- A model tends to agree with a plan you have shown you prefer, and it gives feedback that matches the opinion you stated about your own work [1]. A yes to a leading question is agreement, and agreement isn’t evidence.
- Ask for the strongest counter-argument before you ask for confirmation, in a chat that hasn’t agreed with you yet. The model’s own earlier yes is context, and it pulls the next answer toward yes.
- When you want an assessment, strip your opinion out of the question: describe the situation, name the options and ask for a comparison.
- A counter-argument prompt that also carries your verdict gets a counter-argument and a rebuttal of it. Ask for the case against, and nothing else, and answer it yourself afterwards.
- The fix for a leading answer is in the question. When you use a model’s judgment as a check, the question sets the quality of the check.
You can now
- Spots agreement that is not evidence
References
Section titled “References”- Mrinank Sharma, Meg Tong, Tomasz Korbak and 15 others. Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548. Paper.
Sharma 2023 - Anthropic. AI Fluency: Framework and foundations. Claude Academy. Course.
Academy ai-fluency-framework-foundations