Explains how a language model produces text and where it fails

Concepts · competency concepts/explains-models

Taught in: the Concepts course

Draws on: How language models work, Capabilities and limits, Grounding and memory

Learning objectives

Explains tokens, context and sampling in plain words (base)

Claim	Why	Example
A model produces text one token at a time, each chosen from a probability distribution over what could come next.	Once the learner sees output as a chain of likely next pieces, both fluent nonsense and surprising variation stop being mysterious.	Asked why the same question gave two different answers, the learner says the model picked a different likely token early on and the rest followed from it, rather than saying the model "changed its mind".
The context window is the whole of what the model can see for one response, and anything outside it does not exist for the model.	Most "it forgot" and "it ignored my file" complaints are context problems, and the fix is different from a prompting problem.	When a long chat starts contradicting itself, the learner suspects earlier messages fell out of the window and restates the key facts instead of scolding the model.
Temperature and sampling settings trade variety for predictability and do not change what the model knows.	Learners otherwise reach for temperature to fix accuracy, which it cannot do.	For a summary that must be the same on every run the learner asks for low temperature. For brainstorming names they raise it, and in neither case expect it to add facts.

Served by: What the model can see, How a language model works, Where memory comes from, Same prompt, different answer, What every token costs

Names the common ways output goes wrong and why (base)

Claim	Why	Example
A hallucination is a fluent, confident statement with nothing behind it, and fluency is no signal of truth.	Confident wording is exactly what the training rewards, so the learner must stop using tone as a check.	Given a citation with a real-looking journal name, author and year, the learner looks it up before trusting it and is not surprised when it does not exist.
The knowledge cutoff means the model's built-in facts stop at a date, and anything after that must be supplied or retrieved.	Questions about recent events or versions get plausible but stale answers unless the learner knows to bring the facts.	Asking for the current version of a library, the learner pastes the changelog or asks the tool to look it up rather than accepting the number the model recalls.
Sycophancy is the model agreeing with the person in front of it, so the learner's own framing shapes the answer.	A leading question produces a leading answer, which feels like confirmation and is not.	Instead of "this plan is solid, right?" the learner asks "what are the three strongest objections to this plan?" and compares the two responses.

Served by: How a language model works, Same prompt, different answer, Getting a straight answer, Where a model makes things up

Explains what grounding and retrieval add and what they do not fix (base)

Claim	Why	Example
Retrieval puts relevant documents into the context window at question time, so the model answers from text it can see rather than from what it memorized.	Learners otherwise expect a model "trained on our documents", and the difference decides what a stale or missing document does to the answer.	When a product assistant gives an outdated price, the learner asks which document it retrieved rather than asking for the model to be retrained.
Grounding reduces invented facts about the retrieved material but does not stop the model from misreading it or filling gaps outside it.	A grounded system still produces fluent text, and a wrong summary of a real document looks the same as a right one.	The learner checks the cited passage against the answer and finds the model combined two clauses from different contracts into one.
Retrieval finds text that looks similar to the question, so a question that uses other words than the document may miss it, and keyword search misses it most often.	Knowing this turns "the assistant does not know our policy" into "rephrase or fix the index", which is a fixable problem.	The assistant says it has no travel policy, so the learner asks about "expense rules for trips", gets the document, and reports the naming gap.

Served by: Where memory comes from, Answers from documents the model never saw

Alignment

Framework	Code	Asks	Objectives here
AI Fluency 4D (Dakan and Feller)	Discernment	Judge the output, the process and the behavior of the AI critically	names-failure-modes