Capabilities and limits
Concepts · topic concepts/limits
Language models are fluent and broadly knowledgeable, and they fail in predictable ways. This topic names the failures a user must expect: confident invention, knowledge that stops at a date, answers that vary between runs, agreement that flatters instead of informs, instructions that fade as the context fills, and the cost and delay that grow with every token.
Concepts
- Hallucination
- Output that is fluent and confident but false: an invented citation, a function that does not exist, a plausible date that is wrong. It happens because the model predicts likely text, not verified facts, and it has no built-in signal for "I do not know". Hallucination is most likely on specifics, rare topics and anything the model cannot look up. glossary
- Knowledge cutoff
- The date after which a model's training data stops. Events, releases and documentation newer than the cutoff are unknown to the model unless they are pasted into the context or fetched by a tool. A model will often answer about recent things anyway, from older patterns, so the cutoff is a common and quiet source of wrong answers. glossary
- Non-determinism
- The same prompt can produce different answers on different runs because output is sampled from probabilities. Even at the lowest temperature, small differences in serving can change results. This means one good answer does not prove a prompt is reliable, and any process built on a model needs checks that tolerate variation. glossary
- Sycophancy
- The tendency of a model to agree with the user, praise their idea or change a correct answer when challenged, because agreeable text was rewarded during training. Sycophancy makes a model a poor judge of your own work unless you ask for criticism explicitly, hide your preference, or have it argue the other side. glossary
- Cost and latency
- Every token read or written costs money and time. Long prompts, large documents, big models and multi-step agents multiply both. Latency grows with output length because tokens are produced one at a time. Cost and latency are why a smaller model or a shorter prompt is often the right engineering choice even when a larger one would answer slightly better. glossary
- Instruction dilution
- The weakening of an instruction as more text piles up after it in the context window. A rule stated once at the start competes with everything said since; in a long conversation or a long instruction file the model follows some rules and quietly drops others. Dilution is why short instruction files work better than long ones and why a rule that must always hold belongs in a mechanical check, not in prose. glossary
Links
- Builds on: How language models work
- Leads to: What an agent is, Recognizing failure, Responsible use, Choosing models and tools
- Competencies drawing on it: Explains how a language model produces text and where it fails, Recognizes an agent, its tools and its degree of autonomy
Lessons
- Same prompt, different answer (tutorial)
- Getting a straight answer (tutorial)
- Where a model makes things up (explanation)
- What every token costs (tutorial)
Your reference
Each lesson above adds its takeaways and its example here once you finish it. Your reference lists every lesson you have finished.
Same prompt, different answer
Unlocks when you finish Same prompt, different answer.
Takeaways
- Each answer is one draw from the probabilities the model assigns, so the same prompt gives different answers on different runs. This is non-determinism, and it comes from how the answer is produced. A better prompt narrows the variation and doesn't end it.
- Temperature 0 puts the probability on the top candidate and removes most of the variation. It doesn't remove it all, because a real model computes its scores in batches with other requests, the rounding follows the order of the arithmetic in the batch, and a near tie can swap.
- One good run is a sample of size one. It shows that an answer can come up, and says nothing about how often it does or what a bad run looks like.
- A process built on a model needs a check that runs on every answer, accepts the variation that doesn't matter, and rejects the answers that don't do the job. Its pass count, over many runs, is the reliability of the prompt.
Example
This lesson has no runnable example or prompt block.
Getting a straight answer
Unlocks when you finish Getting a straight answer.
Takeaways
- Sycophancy is the model agreeing with you, praising your idea or changing a correct answer when you push back, in part because raters preferred agreeable answers during training and training rewarded them.
- Your framing shapes the answer. A leading question produces a leading answer, and agreement you asked for tells you nothing about your plan.
- To get a straight answer, ask for criticism explicitly, hide your preference, or have the model argue the other side.
- A reversal under pushback with no new evidence tells you the model followed you, and nothing about who was right. Check the claim outside the chat, as the safety lessons on spotting a hallucination ask for every specific you rely on.
Example
I've decided to replace our weekly status meeting with a written update that everyone posts in a shared doc by Friday noon. Great idea, right?
That's a great idea! Written updates respect everyone's time, give people a record they can search later, and let colleagues in other time zones read at a moment that suits them. Many teams find that they get more done once the meeting is gone. A shared doc with a Friday deadline gives you a clear rhythm. You've thought this through well.
Where a model makes things up
Unlocks when you finish Where a model makes things up.
Takeaways
- One mechanism produces both the answer you can use and the invented one. The task differs, so the question to ask is about the task.
- The first question is how common the task is in written text. A summary, a reformat or a textbook concept is a wide path. A niche topic, an exact figure or an event after the cutoff is a thin one.
- The second question is how much of the answer hinges on exact specifics. Invention concentrates in names, numbers, dates, quotes and citations, because each is exactly right or wrong.
- A wide-path task needs a spot-check against the material you gave it. A thin-path task needs a check of every specific before you pass it on, in proportion to what a wrong specific would cost.
- A real failure is two properties meeting, such as prediction plus a knowledge gap or prediction plus the cutoff. Prediction cannot be switched off, so the fix is on the other side of the pair: supply what is missing, or verify what came out.
Example
Summarize this in two sentences for someone who missed the meeting.
[a page of meeting notes, pasted in full]
The team agreed to move the release from Friday to the following Tuesday so the payment fix can be tested on staging first. Sam writes the test plan, and Tom tells the two customers who asked about the date.
What every token costs
Unlocks when you finish What every token costs.
Takeaways
- A vendor charges per million tokens, with a lower price for the tokens the model reads and a higher one for the tokens it writes, and a large model in a family costs several times a small one.
- One call costs a fraction of a cent, and the bill comes from the number of calls. The choice of model moves the price by a factor, the length of the prompt and the answer by a share, and a long chat pays for every earlier turn again.
- The wait grows the same way, because a model writes one token at a time. A long answer on a large model is the slow case.
- A rule in an instruction file is tokens like everything else and competes with all the text after it, so a rule in the middle of a long file weighs less than one at either end. Shorten the file, and move a rule that must always hold into a check on the answer.
Example
This lesson has no runnable example or prompt block.
Sources
DLAI-3AI Prompting for Everyone, DeepLearning.AI (course)Academy ai-capabilities-and-limitationsAI capabilities and limitations, Claude Academy (course)Academy ai-fluency-framework-foundationsAI Fluency: Framework and foundations, Claude Academy (course)Sharma 2023Towards Understanding Sycophancy in Language Models, International Conference on Learning Representations (ICLR 2024), arXiv preprint 2310.13548 (paper)