How a language model works
You type a question, and a paragraph comes back that reads like a person wrote it. It is tempting to picture a mind on the other side that looked up the answer. Nothing like that happens. In this lesson we replace that picture with the real one. It is simpler and it explains more: a language model predicts the next piece of text, one piece at a time.
Text becomes tokens
Section titled “Text becomes tokens”A model never sees letters or words. Its input is cut into tokens, short chunks of text that are usually a word, part of a word, or a punctuation mark. “Unbelievable” might be three tokens, and “the” is one. Each token has a number, and the model works only with those numbers.
This has consequences you notice. Models are bad at counting letters in a word, because they never saw the letters. They’re priced per token, not per word. And rare names and non-English text often cost more tokens per character than common English, because the token vocabulary was built around the text that appears most often in training data.
One token at a time
Section titled “One token at a time”Given the tokens so far, the model outputs a score for every token in its vocabulary: how likely is each one to come next? Those scores become a probability distribution. The model picks one token from it, appends it to the input, and the whole thing runs again. A hundred-word answer is a hundred or so rounds of “what comes next?”.
Try it. The widget below is a toy with five candidate tokens and made-up scores, but the mechanism is the real one.
The cat sat on the …
The input fixes the distribution, so the randomness is in the pick alone. Along the same line, temperature doesn’t add knowledge: it flattens or sharpens the same distribution. At low temperature the model almost always picks the top token and sounds confident and repetitive. At high temperature it wanders.
Where the scores come from
Section titled “Where the scores come from”A neural network with billions of adjustable numbers, the weights, produces the scores. During training the network reads enormous amounts of text and is nudged, each time, so that the token that really came next scores higher, and it doesn’t store facts or rules. The result is a compressed statistical model of the text it saw: grammar, style, common facts, common code, common arguments.
So models know a great deal about things that appear often in writing and are unreliable about things that appear rarely, recently, or only in your organization’s private documents. Knowledge is frequency-weighted and frozen at the training cutoff.
That first training stage produces a text continuer, and a text continuer has no idea that you want an answer. Type a question into a raw model and it may reply with more questions, in the pattern of a worksheet. A second, shorter stage fixes this: the network is trained on example replies that people have rated, and it learns that your text is a question to answer. Those ratings have side effects. People tend to rate agreeable replies and long replies higher. The finished model leans toward both [1].
What did the model do?
Section titled “What did the model do?”The lesson has a widget that shows a model scoring five candidate tokens, picking one, and repeating.
A colleague says: “The model looked up the answer in its database and returned it.” Which correction is right?
Look back at the widget. What is the one operation it performs, over and over?
Put the steps in order
Section titled “Put the steps in order”The lesson explains what a model does during inference, when it writes an answer to a prompt.
- Read the prompt and the answer so far as tokens
- Score each possible next token
- Pick one token from those scores
- Add the picked token to the answer
- Start again with the longer text, until the answer ends
What does the model need before it can score anything, and what does it do with the token it picks?
Failure modes, named
Section titled “Failure modes, named”Once you see generation as prediction, the famous failures stop being mysterious.
- Hallucination. The most likely continuation of “The 2019 paper by Smith et al. showed” is a plausible citation, even if no such paper exists. Confidence in the prose isn’t confidence in the fact. Made-up details concentrate in the specifics, such as a release date, a page number, or a quoted line. A common concept has been written about thousands of times and the prediction is well constrained, but one exact figure has few or no examples behind it, and the model fills the gap with something of the right form. A detail the model saw once comes out in the same confident prose as a fact it saw ten thousand times [1].
- Sycophancy. Agreeable replies were rated well in the second training stage, so agreeing is likely text. Push back and the model often folds, even when its first answer was right.
- Instruction dilution. Everything in the context is just tokens. A rule you stated on page one competes with everything said since, and long conversations drift. Position in the context window matters too. A rule buried on page six of a twelve-page paste weighs less in the prediction than the same rule at the top or at the bottom. Liu and colleagues measured this. When the one document that holds the answer was in the middle of a long context, models answered less accurately than when it was at the start or at the end, and models built for long contexts showed the same curve [2].
- Cutoff and staleness. Anything after training is invisible unless a tool puts it in the context.
Name the failure
Section titled “Name the failure”You ask for the release date of a library version that came out last month. The model answers with a specific date, stated confidently, and it is wrong. Which failure mode is the best name for this?
When did the model's training data end, and when did the event happen?
Name each failure
Section titled “Name each failure”The lesson names four failure modes that follow from generation as prediction: hallucination, sycophancy, instruction dilution, and cutoff and staleness.
Match each reply to the failure mode it shows.
Is the model filling a gap, agreeing with you, losing an old rule, or missing what came after training?
Exercise
Open any chat assistant you have access to. Ask it to count the letter “r” in “strawberry” and to spell the word backwards. Then ask it for the opening sentence of a book you know well, and ask that last question a second time in a fresh chat. Note which answers are right and, for the wrong ones, which failure mode from this lesson explains it. If the two answers about the book differ, the difference comes from sampling, the pick from the distribution you saw in the widget.
A good result names one failure mode per wrong answer and can say, in a sentence, why prediction rather than lookup produces it.
Stretch: Ask the assistant to explain why it got one of them wrong, then decide whether you believe the explanation and why.
Recap
- Input is tokens rather than words, and that alone explains a number of odd weaknesses.
- Output is one token at a time, sampled from a distribution the model scores. Temperature reshapes the distribution and doesn’t add knowledge.
- The scores come from weights fitted to training text, so knowledge is frequency-weighted and stops at the cutoff.
- A second, shorter training stage on rated replies turns the text continuer into an assistant, and the ratings have side effects.
- Hallucination, sycophancy, dilution, and staleness all follow from those two stages [3].
You can now
- Explains tokens, context and sampling in plain words
- Names the common ways output goes wrong and why
References
Section titled “References”- Anthropic. AI capabilities and limitations. Claude Academy. Course.
Academy ai-capabilities-and-limitations - Nelson F. Liu, Kevin Lin, John Hewitt and 4 others. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157-173. Paper.
Liu 2024 - Addy Osmani, Ivar Soares Urdalen, Leo Simons. How agents think: tokens and context, reasoning strategies, model choice, system prompts. Agent Engineer Course. Course.
AEC-02