Skip to content
AI Training

Same prompt, different answer

In the lesson on how a model works you saw the widget score five candidate tokens and pick one, and you saw that the pick is where the randomness is. In this lesson we run that pick ten times in a row and count. The toy model answers one fixed question, and we run it at two temperatures, then with a small wobble in the scores, and then with a check applied to every answer. By the end you can say what one good answer proves about a prompt, and what a process built on a model needs instead.

You need only this page. The toy model is a widget further down, and it runs in your browser. It draws new random numbers on every press, so your runs differ from the ones this page shows. Each set of runs shown on the page is one recorded set from the same toy model.

Run it at temperature 0

Section titled “Run it at temperature 0”

The toy model holds five candidate answers to “What is the capital of Australia?” with made-up scores, in the same spirit as the widget in the lesson on how a model works. Four of the candidates name Canberra in different wordings, and the fifth, with the lowest score, says Sydney. Each run turns the scores into probabilities at the temperature you set, draws one answer, and lists it. The summary line counts how many runs gave the same text as the first run.

At temperature 0 the probabilities collapse onto the top score. Before you press the button, predict the summary line for ten runs.

Checkpoint · choice

Predict the summary at temperature 0

Section titled “Predict the summary at temperature 0”

You set the toy model to temperature 0, leave both boxes unchecked, and press the button for ten runs. What does the summary line say?

Now try it. Set the temperature to 0, leave both boxes unchecked, and press the button.

What is the capital of Australia?

      Every line reads Canberra, because with all the probability on one candidate the draw has only one place to land. The summary line is the same on every press:

      10 of 10 runs give the same answer as run 1

      Raise the temperature

      Section titled “Raise the temperature”

      Now raise the temperature. Move the slider in the widget above to 1.0 and press the button again. The scores are unchanged, and so is the order of the candidates. The candidates below the top one keep more of the probability. Here is one recorded set of ten runs at 1.0:

      What is the capital of Australia? (temperature 1.0, 10 runs)
       1  Canberra
       2  Canberra.
       3  Canberra
       4  The capital of Australia is Canberra.
       5  Canberra
       6  Canberra.
       7  It's Canberra.
       8  Canberra
       9  Sydney
      10  The capital of Australia is Canberra.
      4 of 10 runs give the same answer as run 1

      Ten runs, five different texts, and one of them wrong. This is non-determinism: the same prompt gives different answers on different runs, because each answer is a draw from the probabilities and the draw can land anywhere the probability is above zero [1]. Nothing about the prompt changed between run 1 and run 9. Run 9 says Sydney because Sydney had a few percent of the probability and, on the ninth draw, the number came up.

      Press the button a few more times and tally how often the ten runs agree with run 1. The count moves. The exact value is the toy’s, but the picture is the real one, and it is the picture behind the exercise in the lesson on how a model works, where the opening sentence of a book came out differently in two fresh chats.

      Variation at the lowest temperature

      Section titled “Variation at the lowest temperature”

      Temperature 0 gave ten identical answers, so it is tempting to decide that a workflow only has to set the temperature to 0 and the variation is gone. On a real model it isn’t. The scores the toy holds as fixed numbers come, in a real model, from billions of multiplications done on hardware that rounds. The order of those multiplications changes with the other requests served alongside yours, and rounding in a different order gives a score that differs in the last decimal places. When two candidates are close, that last decimal is enough to swap which one is on top, and the greedy pick follows the swap.

      The box “Shake the scores before each pick” in the widget imitates that wobble. Before each pick it shakes every score a little. The shake is far larger than the rounding wobble in a real model, so ten runs are enough to see the effect. Set the temperature back to 0, check the box, and press the button a few times. Here is one recorded set:

      What is the capital of Australia? (temperature 0.0, 10 runs)
       1  Canberra
       2  Canberra
       3  Canberra
       4  Canberra
       5  Canberra.
       6  Canberra
       7  Canberra
       8  Canberra
       9  Canberra
      10  Canberra
      9 of 10 runs give the same answer as run 1

      In this set one run in ten differs from the rest, at the lowest temperature there is. In the toy the difference is a full stop. In a real model the swapped token is somewhere in the middle of a long answer, and every token after it is predicted from a slightly different prefix, so two temperature-0 answers can share their first sentence and part ways after it. The Messages API reference says the same about the real thing: even at a temperature of 0 the results aren’t fully deterministic [2].

      Put the two results together. At temperature 1 a run can pick a wrong answer that has a few percent of the probability. At temperature 0 a run can still differ from the run before it. So one run tells you which answer came up once. It tells you nothing about how often that answer comes up. A prompt that gave the right answer once has been tested zero times for what a workflow needs from it: a usable answer every time it runs.

      Checkpoint · scenario

      One good run

      Section titled “One good run”

      A colleague ran their invoice prompt once, got the right total, and wants to put it in the payment process today. What do you tell them?

      The check a workflow needs

      Section titled “The check a workflow needs”

      A person reading the ten recorded runs at temperature 1.0 sees at once that nine of them are right. A program comparing them with the first answer sees four matches and six failures, and a program comparing them with the full sentence sees two matches and eight failures. Neither count is the one the workflow cares about. The workflow needs a check that accepts the variation that doesn’t matter and rejects the variation that does.

      For this question the check is short: does the answer name Canberra? Any wording that names it passes, and the Sydney answer fails. A check this loose is fine here because a person reads the answer, and a person reads “It’s Canberra.” and “Canberra” as the same fact. When a program reads the answer, as in the ticket system of the pitfall, the wording is part of what the check has to pin down. The box “Check each answer” in the widget applies the check to every run and adds a second summary line. Set the temperature to 1.0, uncheck “Shake the scores before each pick”, check “Check each answer”, and press the button. Here is the check applied to the recorded set from temperature 1.0:

      What is the capital of Australia? (temperature 1.0, 10 runs)
       1  pass  Canberra
       2  pass  Canberra.
       3  pass  Canberra
       4  pass  The capital of Australia is Canberra.
       5  pass  Canberra
       6  pass  Canberra.
       7  pass  It's Canberra.
       8  pass  Canberra
       9  FAIL  Sydney
      10  pass  The capital of Australia is Canberra.
      4 of 10 runs give the same answer as run 1
      9 of 10 runs pass the check: the answer names Canberra

      The summary lines answer different questions. The first asks whether the text is identical, and a model gives a “no” to that on a regular basis without being wrong. The second asks whether the answer does the job, and it is the count a workflow can act on: nine items go through and one is held for a person. The check lives outside the model, in code, and it runs on every answer, so a change in the model’s variation shows up as a change in the pass count instead of as a surprise.

      What the check looks at depends on the job. A classification prompt checks that the answer is one of the allowed labels and nothing else, and an extraction prompt checks that the field parses as a number or a date. The checker in the lesson on structured output is a check of this kind for action items. A program can run each of them a hundred times without anyone reading a hundred answers.

      Checkpoint · choice

      Which check does the workflow need?

      Section titled “Which check does the workflow need?”

      The ticket workflow from the pitfall needs a check. Which one does the job?

      Exercise

      With “Check each answer” checked in the widget, press the button three times at temperature 0.3 and three times at temperature 1.5. The temperature scale is the toy’s own, and a real model’s scale differs. Each press shows both counts, how many runs agree with run 1 and how many pass the check, so note both. Put the twelve numbers in a small table. Then write the check a workflow that used this model would need, in one or two sentences, and say which wording differences it accepts and which answers it rejects. Ten minutes is enough. Doing this once with numbers in front of you is what makes you ask “how often?” the next time someone shows you one good answer.

      A good result is a table where the agreement count falls as the temperature rises while the pass count changes little, and a check that names the fact to test for and the wording it lets through. Which of the two counts would you have reported after one run, and what would it have told you?

      Stretch: Take a stricter check that passes only the bare word Canberra. Apply it by hand to the recorded ten runs at temperature 1.0 above, and decide whether the stricter check helped the workflow or hurt it.

      Recap

      1. Each answer is one draw from the probabilities the model assigns, so the same prompt gives different answers on different runs. This is non-determinism, and it comes from how the answer is produced. A better prompt narrows the variation and doesn’t end it.
      2. Temperature 0 puts the probability on the top candidate and removes most of the variation. It doesn’t remove it all, because a real model computes its scores in batches with other requests, the rounding follows the order of the arithmetic in the batch, and a near tie can swap.
      3. One good run is a sample of size one. It shows that an answer can come up, and says nothing about how often it does or what a bad run looks like.
      4. A process built on a model needs a check that runs on every answer, accepts the variation that doesn’t matter, and rejects the answers that don’t do the job. Its pass count, over many runs, is the reliability of the prompt.

      You can now

      • Explains tokens, context and sampling in plain words
      • Names the common ways output goes wrong and why

      References

      Section titled “References”
      1. Anthropic. AI capabilities and limitations. Claude Academy. Course. Academy ai-capabilities-and-limitations
      2. Anthropic. Messages. Claude Platform documentation. Reference. Claude docs messages