Hallucinations: why they happen and what actually helps
The source of the problem
The model produces the most plausible continuation, not the truth. When it has no basis, the output still looks confident — and that's exactly the danger: the error doesn't look like an error.
Four things that help
Attach a relevant source to the prompt instead of relying on the model's memory.
Require a source citation for every claim, and verify in code that the source exists in the material provided.
Give an explicit way to say “there's no answer in the material,” and reward it with examples.
Verify discrete facts — numbers, dates, identifiers — against a structured source of truth, not against text.
What doesn't help
Writing “don't make things up” in the prompt. That's a request, not a mechanism. Confidence claims from the model aren't a reliable measure either — they're part of the output, not a measurement of it.
Going deeper
Build a dedicated test category of questions with no answer in the material, and measure how often the system said so correctly. It's the metric that's almost always forgotten, and it's the one that decides whether you can rely on the system in real use.