LLMs hallucinate. It’s just the way they are. This is obviously a problem for systems that need dependable results. The first reaction is to limit the LLM to your own data, AKA Retrieval Augmented Generation (RAG). Unfortunately, it’s not that simple. A model can find the exact right paragraph in a fifty-page document and still invent a different answer. Finding information and returning the right answer are not the same thing. Even the best-performing model still fabricates it’s share of its answers when the correct source is sitting right in front of it. RAG doesn’t fix hallucination. It changes which hallucinations you get.
Hallucination is at least three distinct failure modes with different causes, and fixing a solution means knowing which one you’re actually measuring. This post covers the two computational ones. The third, what happens when a confident user pushes back, is interactional rather than architectural, and I’ll talk about it in another post.
Hallucination is a multi-headed beast
Most people think about hallucination as a single dial. Turn the model’s confidence down and hallucinations go down with it. There are two main reasons that doesn’t work, Fabrication and Grounding.
Fabrication is inventing a fact that exists nowhere in the source or the training data. Grounding failure is different: the right information is sitting in the context window and the model fails to find or use it.
These two have different causes and different rates across models so you can’t solve them with one approach. If you increase your retrieval quality you can fix grounding failure while doing nothing for fabrication. A model can be excellent at finding the right passage and still invent the answer to a question that the underlying data never addresses.
Model family predicts fabrication resistance better than model size does. Qwen3-Next-80B-A3B, a mixture-of-experts model with only 3 billion active parameters performs better at long context retrieval than dense models many times its total size, including Llama 3.1’s 405-billion-parameter version.
Grounding and truth-telling are different
Let’s take a look at an example fifty-page vendor contract. Now let’s ask it a question that’s not in the contract, something about a fee structure that isn’t in there. A model that’s excellent at the finding existing text can still confidently invent an answer when there is no relevant text. Finding information and refusing to invent it when there’s nothing to find are not the same.
The problem gets worse when you start looking at cross document aggregation. Synthesizing an answer from multiple documents requires the model to pay attention to parts of multiple, disjointed documents at the same time. This is the more common use case for RAG. A vector search will return chunks from multiple documents and the model has to reason across all of them. Assuming you are evaluating retrieval responses for accuracy (and you should), you need to make sure you are evaluating multi document retrievals and not just looking at single documents
This is also why the standard retrieval benchmarks (like RULER) that measure whether a model can find a needle in a long document, don’t tell you anything about fabrication rate. They score whether the model finds the right answer when one exists, at a single-lookup level of difficulty. If your evaluation suite is built the same way, you aren’t looking at what happens when the answer isn’t there or when it takes real synthesis to assemble.
Fabrication gets worse with longer contexts. The 172B study found the best model fabricating at roughly 1 percent of answers at 32,000 tokens, and that rate roughly triples by 128,000 tokens, crossing 10 percent for every model tested at 200,000 tokens. GLM 4.6, a dense model, fabricates on close to three-quarters of unanswerable questions once you reach 200,000 tokens. This is due to the way the model works. Dense models attend across an enormous key-value cache as context grows, and the signal any single fact contributes gets diluted alongside everything else. Mixture-of-experts models route each token through a small subset of specialized parameters instead of spreading attention across the whole cache, and they degrade more gracefully as a direct consequence. That’s why a 3-billion-active-parameter MoE model can perform better at long context better than a dense model with a hundred times the active parameters. Model family beating model size isn’t a training artifact. It’s about the token routing.
There’s a reason why fabrication doesn’t improve much no matter which technique you throw at it. A recent impossibility proof, working from mechanism design theory and the actual math of how a transformer aggregates competing signals, shows that no inference mechanism, RAG included, can, at the same time, be fully truthful, fully information-preserving, complete, and constrained to what it actually knows. You can trade between those four properties but you can’t eliminate all of them at once. That’s not a limitation of this generation of models. It’s closer to a conservation law.
Temperature is a tradeoff, not a dial
The common belief is that lower temperature means fewer hallucinations. This depends entirely on which hallucination you’re worried about.
For deterministic output, setting the temperature to zero, gives the best overall accuracy in roughly 60 percent of cases in the 172B study, so for simple factual retrieval, turning the dial down usually helps. But higher temperature reduces fabrication for most models, probably because it raises the odds of an “I don’t know” instead of a confident invented answer. And temperature zero comes leads to coherence collapse. Infinite generation loops happened at roughly 48 times the rate at temperature zero compared to temperature one.
The type of use case makes a big difference. Code generation research finds a clear increase in hallucination as temperature rises, which is the opposite for document QA. There’s no single correct temperature setting for “reducing hallucinations.” There’s a setting that trades retrieval accuracy against fabrication confidence against output coherence and the answer depends on what you’re building.
What can you do?
Prompting is not the fix people think it is. It can help with grounding but does almost nothing for fabrication.
Grounding failure is a retrieval and attention problem, so it responds to changes in the pipeline. Split retrieval into two steps instead of one. Have the model pull literal quotes first, with no synthesis, then answer using only those quotes. Testing whether it found the passage separately from whether it correctly reasoned about the passage turns a single failure into two things you can actually measure. You should rerank after a vector search, since dense retrieval alone can often puts the correct chunk lower on the list of retrieved data. If the question requires combining data across documents, break it into sub-questions and answer each against its own retrieval before combining. Don’t hand the model five disjointed chunks and hope it does the proper cross-referencing in one pass.
Put your most likely chunk first or last in the context window. Lost-in-the-middle is a real risk, and it gets worse as context grows. Shrink your context wherever you can. As above, fabrication roughly triples between 32,000 and 128,000 tokens. Fewer, better-ranked chunks reduce grounding failure and fabrication risk at the same time. This is one of the only methods that affect both issues.
Fabrication is a different animal, and the impossibility proof above tells us why prompting can’t solve it. If no inference mechanism can be simultaneously truthful, information-preserving, complete, and constrained to what it actually knows, no system prompt is going to change that. It’s not a matter of finding the right wording. Research on citation generation shows this. Prompted citation generation doesn’t produce reliable citations, but models fine-tuned specifically on citation-grounding data do. An 8-billion parameter model trained this way beat GPT-4o on citation accuracy. Training closes the gap. Prompting narrows it, slightly, and only some of the time.
So the fixes that actually work on fabrication live outside the generation step. Generate the answer, then generate a separate set of verification questions about its own claims, and answer those independently against the source. Revise anything that doesn’t hold up. This catches a good chunk of fabrications, though it adds a second call and doesn’t help much with recognizing when a question has no answer at all. Force the model to cite a specific paragraph or span for each claim, then check in code, not in the model, whether that span actually exists and actually supports the claim. Don’t ask the model if it’s sure. Ask your code to check. Score each claim in an answer separately instead of scoring the whole answer for faithfulness, since a paragraph with two grounded claims and two invented ones will still score fine on an aggregate metric. If you have the volume, sample the same question multiple times at a higher temperature and check whether the answers agree. Disagreement is a cheap and reasonably reliable fabrication signal, since you already know higher temperature makes a model more willing to say it doesn’t know instead of inventing something.
None of this eliminates fabrication. Nothing does, and that’s the point of the proof. What it does is move the check from hoping the model is right to verifying it before the answer reaches anyone. That’s the only lever that actually exists.
Sources
Roig, J.V. (March 2026). How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-billion-token study across 35 models. arxiv.org/abs/2603.08274
Hsieh, C. et al. (2024). RULER benchmark. arxiv.org/abs/2404.06654
Karpowicz, M.P. (2025). On the fundamental impossibility of hallucination control in LLMs. arxiv.org/abs/2506.06382
Gramacki et al. (2024). Hallucination rate vs. temperature in code generation. researchgate.net/figure/Hallucination-rate-vs-temperature_fig3_381484725