Hallucinations, why AI makes things up and how to protect yourself
Non-existent rulings, wrong numbers, quotes never said. Where hallucinations come from, when they are most likely, and a six-step method to avoid being fooled.
Mattia PanattaUpdated 15 October 202610 minBeginner
In short
- A hallucination is false information written with the same confidence as true information.
- It stems from how the model works: it produces the most plausible text, not necessarily the true one.
- It is most likely on rare, recent, very specific facts (numbers, dates, names, legal references, quotes).
- You reduce it by giving the model sources, letting it say “I don’t know” and always checking the data you will use.
A serious problem with real consequences
In 2023 a New York lawyer filed a brief with six court precedents invented by ChatGPT, complete with party names and quotes. The judge noticed and sanctioned the firm (Mata v. Avianca). Similar episodes have since happened in many countries, Italy included.
This isn’t a rare defect of a few poor models. It happens to the best too, less often than before, but it happens. The good news: knowing why it happens makes it easy to know when to trust.
Why the model makes things up
Remember how an LLM works? It picks, one piece at a time, the most likely continuation. If you ask “Which Supreme Court ruling says X?” and its data holds no clear answer, the model doesn’t “feel” a gap: it sees that a plausible answer looks like Court, section X, no. NNNNN/YYYY and produces it, with plausible numbers.
There are two subtler causes:
- Training rewards guessing. An analysis published by OpenAI in 2025 showed many tests score only correct answers: “I don’t know” scores zero, a guess sometimes scores. Models learn that guessing pays. Labs are fixing this incentive, which is one reason 2026 models say “I don’t know” more often.
- Sycophancy. Models are also trained on what people prefer, and people prefer being agreed with. If your question assumes something false (“Since flat-rate freelancers can deduct expenses…”), the model tends to go along.
Test yourself
Three statements like those you read in a chat every day. Would you trust them?
"According to Supreme Court ruling no. 18,442/2023, ChatGPT subscriptions are always 100% deductible for flat-rate freelancers."
Two red flags: a precise ruling number (easy to invent) and a rule that makes no sense under a flat-rate regime, where costs are not deducted item by item. Always check the primary source.
"The Transformer was introduced in 2017 in the paper 'Attention Is All You Need' by Google researchers."
Correct, and very famous: it appears thousands of times in training data. Hallucinations are rare on well-known facts and common on rare or specific ones.
"The company's 2025 revenue (from the PDF you uploaded) is 4.82 million, up 12% on the previous year."
Maybe right, maybe not: figures extracted from a document must be checked on the cited page. Ask "which page?" and verify at least the numbers you will use.
When the risk is highest
| Situation | Risk | Why |
|---|---|---|
| Very well-known facts (history, basic science) | Low | Seen thousands of times in training data |
| Summary of a text you pasted | Low-medium | The text is in context, but details can be distorted |
| Numbers extracted from long documents | Medium | It can confuse rows, years, units |
| Recent news without web search | High | Not in training data |
| Precise references: rulings, statutes, DOIs, ISBNs | High | Easy-to-imitate format, rare content |
| Verbatim quotes from people | High | Tends to paraphrase and misattribute |
| Questions containing a false premise | High | Sycophancy: it goes along |
The six-step method
1. Give sources rather than asking for them
The most effective way to cut hallucinations is to put the information in the context: upload the document, paste the statute, turn on web search. The model moves from remembering to reading, and reading is far more reliable.
2. Allow “I don’t know”
Say it explicitly. It sounds trivial but measurably changes the model’s behaviour.
Base your answer only on the information I provided (or that you find via web search, citing it). If something is missing or you are unsure, write “I don’t know” or “To be verified” instead of guessing. Give the source for every number.
3. Ask where it came from
“Which page of the document?”, “What’s the link?”. If the model can’t point to it, or points to a page that doesn’t say that, you’ve found the problem.
4. Check the primary source
A citation is not verification. Open the link, look up the ruling in the official database, check the statute in the official journal. For AI model data, go to Artificial Analysis or Arena and check the date.
5. Ask the same question differently (or to another model)
If rephrasing or asking a second assistant gives different answers, the model isn’t sure. If they match, the chance of error drops (not to zero: models share much training data).
6. Don’t build in false premises
Ask “Can flat-rate freelancers deduct expenses?” rather than “Since they can deduct expenses, which ones can they deduct?”. The first leaves the model room to correct you.
A rule for professionals
Anything that ends up in a document you sign (opinion, tax return, filing, report) must be checked against the original source. Italy’s AI law (Law 132/2025, Art. 13) says it clearly for intellectual professions: AI can help, but the work and the responsibility remain the professional’s.
Will hallucinations decrease?
Yes, and they already are: 2026 models use web search more often when needed, state their limits more often and cite pages when working on documents. But given how the technology works they won’t disappear entirely: a system that generates the most plausible text will always, now and then, produce something plausible and false.
So the skill to build isn’t “find an AI that never errs”, but knowing when an error is likely and what it would cost. For a slogan, no problem. For a tax deadline, you check.
Sources
Informational content, current as of the date shown. Prices, models and rules change often: always check official sources. Written with the support of AI tools and reviewed by the author.
