Why AI Gives Different Answers to the Same Question: It Isn't Randomness
You got a genuinely good answer yesterday. Today you asked the same thing and got something different. Not wrong: just not the same.
This is not you misremembering the prompt, and it isn’t the model being creative. Thinking Machines Lab ran a single identical prompt 1,000 times, on the setting specifically designed to remove randomness, and got 80 distinct answers back. The most common one appeared just 78 times out of a thousand.
Understanding why matters more than it sounds, because the usual advice, “write a better prompt”, doesn’t touch it.
Where the answers actually split
The detail worth sitting with is where the divergence happened.
All thousand responses agreed perfectly for the first 102 The chunks a model reads and writes in: roughly three-quarters of a word each. A model doesn’t produce an answer all at once; it picks one token, then the next, each one influenced by everything before it. . Word for word, identical. Then at token 103, describing where Richard Feynman was born, they split: 992 said “Queens, New York” and 8 said “New York City.”
Both are correct. That’s the part people miss when they hear “AI gives different answers” and picture it inventing facts. What happened here was two true statements, and a coin-flip between them, and once the sentence went one way rather than the other, everything downstream drifted too.
So the risk isn’t usually that one answer is wrong. It’s that you cannot get the same one twice.
The cause is boring, and it isn’t randomness
Here’s the mechanism, and it’s mundane in a way that’s oddly reassuring.
When you send a question, it doesn’t get processed alone. The server bundles your request with whatever other requests arrived at that moment into a A group of requests processed together for efficiency. How many are in the group depends on how busy the service is at that instant, which is why the same question at 9am and at 9pm isn’t necessarily doing the same arithmetic. . And the math inside the model isn’t perfectly associative: adding a very long list of decimal numbers in a different order produces a very slightly different total.
Different batch, different order, microscopically different numbers. Usually that changes nothing. Occasionally two candidate words are so close that the rounding decides between them, and then you get “Queens” instead of “New York City,” and the rest of the answer follows a different path.
As the researchers put it, the reason nearly all AI services behave this way is that “the load — and thus batch size — nondeterministically varies.”
Your answer depends, very slightly, on how busy the server was. Not on anything you did.
The dial you’ve heard of won’t save you
If you’ve read anything about this, you’ve probably met A setting that controls how adventurous the model is when choosing between likely next words. Turn it down and it always takes the most probable option; turn it up and it takes more chances. Zero is supposed to mean “no randomness at all.” . It’s the standard answer, and it fails here twice over.
First, the 1,000-run experiment above was already at temperature zero. The variation happened anyway, because batch effects sit underneath the setting rather than being governed by it.
Second, and more practically: you almost certainly can’t change it. If you use ChatGPT, Claude, Gemini or Copilot in a browser or an app, temperature isn’t exposed to you at all. It’s a developer parameter, and on some of the newest models it’s being retired even there.
The researchers did solve it: by rewriting the low-level math so results don’t depend on batch size, after which all 1,000 runs came back identical. That’s a fix for people building inference systems. It is not a setting you’ll find in your sidebar.
What this changes about how you work
Four things follow, and they’re all about designing around variance rather than trying to eliminate it.
Save the output, not just the prompt. This is the big one. If a result is good enough to reuse, keep the result. A prompt is not a recipe that returns the same dish: it’s a request that returns something in the same family. The instinct to save clever prompts and re-run them is exactly backwards for anything you need to be consistent.
Treat big variation as a diagnosis. Run the same request twice. If you get two answers that differ in substance rather than phrasing, the question was underspecified. You left something open and the model picked differently each time. That’s useful information, and the fix is more context, not a better prompt. It’s the same lesson as giving AI context, arrived at from the other direction.
Stop being surprised when it doesn’t transfer. A colleague follows your exact instructions and gets something worse. Usually that’s not user error and it isn’t your instructions: it’s the same variance you’d have got yourself on a second attempt.
Never promise byte-identical output. If someone is expecting the same document every month, generate it once and template it. Anything downstream that depends on exact wording: a system that parses the response, a formula that expects a header: will break eventually, and it will break on a Tuesday for no visible reason.
The honest counter-argument
It would be easy to read all this as “AI is unreliable,” and that’s an overcorrection.
Look at the numbers again. A thousand runs, and 992 of them agreed on the substantive point. The disagreement was between two accurate descriptions of the same fact. For most workplace tasks (summarize this, draft that, pull the themes out of these notes) variation at that level is genuinely irrelevant, and treating it as a crisis would cost you more than it saves.
The place it matters is narrow and specific: anything you need to reproduce, anything another system consumes, and anything where you’d be embarrassed to find the second version differs from the one you already sent.
The takeaway
The same question can give you a different answer because your request was batched with different neighbors and the arithmetic rounded differently. Not randomness, not creativity, and not something you caused.
You can’t fix it from where you sit. What you can do is stop expecting repeatability from a tool that doesn’t offer it: save good outputs instead of re-running good prompts, treat large variation as a sign your question was vague, and never build a process that assumes the wording will come back the same.
Sources: the experiment is from Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference” (opens in a new tab), 1,000 completions of one prompt (“Tell me about Richard Feynman”) at temperature 0 on Qwen3-235B-A22B-Instruct-2507, producing 80 unique completions, the most common appearing 78 times, diverging at token 103 with 992 continuations reading “Queens, New York” against 8 reading “New York City.” Their batch-invariant kernels made all 1,000 runs identical. That consumer chat apps don’t expose temperature is straightforwardly checkable in each product. The four working rules are mine, not the paper’s: it’s a systems paper and makes no recommendations about how to use a chatbot.
Related: How to Check AI’s Work: You’re Looking for the Wrong Mistake is the companion problem. That one is about output being wrong, this one is about output being different, and they need different habits. Build a Reusable AI Assistant for the Task You Do 20 Times a Week is the practical answer to “save the setup,” which gets you consistency of approach even when the wording moves.