Why AI Makes Things Up: It Was Trained to Guess
Ask an AI for the publication date of a slightly obscure paper and you will often get one. Confident, specific, formatted exactly like a real date, and wrong.
The usual explanations are that the model is broken, or that it is lying. Neither is right, and the actual explanation is more useful than either. Researchers at OpenAI and Georgia Tech put it in one line: models behave “like students facing hard exam questions,” guessing when uncertain because guessing scores better than admitting you do not know.
It is not a malfunction and it is not deception
Start by clearing away the two intuitive readings.
A The industry’s word for a model stating something false as though it were fact. It is a bad name, because it suggests a perceptual glitch, something going wrong inside the machine. Nothing goes wrong. The system produces a fluent, plausible continuation of your text, exactly as it does when it is right. The output that is correct and the output that is invented are produced by the same process, which is why the wrong one arrives with the same confidence as the right one. is not a system failing. Lying is worse still as a description, because lying requires knowing the truth and choosing otherwise. There is no moment where the model knows the real date and decides to supply a different one.
What there is, according to the research, is a training and scoring environment that quietly rewards a guess.
Some facts cannot be learned, only guessed
The first half of the argument is about what is learnable at all.
The authors show, using learning theory, that facts which lack repeated support in the training data produce unavoidable errors, while recurring regularities do not. That distinction is worth translating, because it is the single most useful thing in the paper for anyone using these tools.
Like people, models learn through patterns and repetition. Spelling and grammar appear millions of times, so patterns like that get learned properly and models are correspondingly reliable at them. But an arbitrary one-off fact, a particular person’s birthday, the exact date of a minor paper, a specific figure from one report, may appear once in the training data or never. There is no pattern underneath it to generalize from, much as you would not expect to recall a phone number you saw once, on a sign, years ago.
So the error rate is not spread evenly across everything you ask. It concentrates in exactly the places you would least expect a computer to struggle: the specific, the obscure, and the arbitrary. A model that can write a competent paragraph about employment law can invent a case citation, and those two facts are not in tension. One is a pattern, the other is a lookup it never had.
The part that makes it persist: guessing scores better
The second half is the bit that explains why this has not been fixed.
Models are ranked on benchmarks, and benchmarks are overwhelmingly scored on accuracy. Get it right, get a point. Get it wrong, get nothing. Say “I am not sure”, and also get nothing.
Look at that scoring from the model’s perspective during training. A wrong answer and an Declining to answer, or flagging that you are not confident. It is the behavior you actually want from a colleague who does not know something, and under most benchmark scoring it earns exactly the same score as being wrong: zero. Since a guess has some chance of being right and an abstention has none, optimizing for the score means guessing. earn the same score, but a guess sometimes wins. Over enough training, the strategy that maximizes the number is the one that always answers. In the authors’ phrase, models are “optimized to be good test-takers.”
This is why the paper frames hallucination as an incentive problem rather than a technical defect. The behavior is not a flaw in the system. It is the system doing well at what it was measured on.
Which is also why a better model alone will not fix it
The authors’ proposed remedy is not another hallucination benchmark. It is changing how the existing benchmarks score, so that confident errors cost more than admitting uncertainty. They describe this as a socio-technical fix, which is a polite way of saying the problem is partly about what the industry has agreed to reward.
There is evidence the behavior is tunable rather than inevitable. In Chroma’s testing of long-context performance, Claude models tended to abstain when they were unsure, while GPT models more often produced a confident wrong answer. Same underlying technology, different dispositions, which tells you this is a choice somebody makes rather than a law of physics.
What this changes about using it
The useful consequence is that hallucination is predictable. It is not scattered randomly through the output, so you can aim your checking.
- Suspect the specific and the obscure. Names, dates, citations, case numbers, figures from a single report, anything that would have appeared rarely. Fluent general explanation is the safe part. The precise detail dropped into the middle of it is the risk.
- Treat any reference as unverified until you open it. An invented source looks exactly like a real one, because both are generated the same way.
- Asking it to admit uncertainty helps a little. Telling a model to say when it does not know does shift the behavior, but you are pushing against the grain of how it was scored. Do not treat it as a guarantee.
- Confidence tells you nothing. This is the hardest habit to build, because in people confidence and knowledge usually travel together.
For what a check actually has to catch, How to Check AI’s Work covers the errors people miss, and AI Gave You a Bad Answer covers what to do once you have caught one.
The takeaway
AI makes things up because one-off facts were never learnable from the training data, and because every scoreboard it was measured against gave a guess a chance of points and gave “I don’t know” none.
So check the specifics, not the prose. The paragraph explaining a concept is where these systems are strongest. The confident little detail sitting inside it, the date, the citation, the number, is where they were trained to take a swing.
Sources: both mechanisms are from the same body of work by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, published as Why Language Models Hallucinate (opens in a new tab) (preprint, September 2025) and then peer-reviewed as Evaluating large language models for accuracy incentivizes hallucinations (opens in a new tab) (Nature, 22 April 2026). The “students facing hard exam questions” and “optimized to be good test-takers” phrasings are theirs. The learning-theory result about facts lacking repeated support is described in the Nature abstract; I was not able to read the full paper, so the account above rests on the two abstracts rather than the complete argument. The abstention contrast between model families is from Chroma’s context rot research (opens in a new tab), which was testing long-context behavior rather than hallucination specifically, and which is vendor research. The checking advice is mine, drawn from the mechanism rather than tested by either paper.
Related: Why AI Gives Different Answers to the Same Question is the other half of why these systems are unpredictable in ways people do not expect. Workslop: The Polished Report That Wastes Everyone’s Time is what happens when nobody checks the specifics before passing the work on.