AI hallucinations: why generative AI invents facts and how to manage it

It may harm your career if your chatbot tells you something you later rely on in court. But it’s not just lawyers who should worry about AI inventing facts; it’s a risk to everyone. What to do?

It’s a familiar courtroom drama scene: the dedicated lawyer, working into the night, stumbles on the key to the case among the stacks of witness statements and legal reference books. Steven A Schwartz might have experienced a similar epiphany earlier this year, when he found multiple previous cases to support his client’s personal injury claim.

Schwartz, a New York attorney with 30 years’ experience, passed his findings to a colleague, Peter LoDuca, who presented them in court. Then everything fell apart. Six of the cases were invented by the ‘generative artificial intelligence’ (AI) chatbot Schwartz used for research. A judge fined the lawyers and their firm and the case became viral news. As a novice AI user, Mr Schwartz said he was “unaware that its content could be false”.

Unfortunately, those AI-generated falsehoods, commonly known as “hallucinations”, are frequent. More accurately, one AI expert told me: “Everything generative AI does is a hallucination. We just happen to like some of them.” That doesn’t mean we should abandon such tools, which are helpful for many work tasks, but instead must be cautious.

The judge told the unfortunate lawyers that it’s their responsibility “to ensure the accuracy of their filings”. Anyone using generative AI tools, like Open AI’s ChatGPT or Google’s Bard, for tasks such as supplementing research or helping with some stages of writing, has a similar responsibility.

Can’t it just stop making things up?
Given these examples, it’s fair to ask why the big brains behind generative AI don’t just design it so it won’t invent things. Put simply, they can’t right now. Indeed, it might never be possible. Understanding why requires an explanation of how the technology works.


Generative AI tools are large language models (LLMs), ‘trained’ on a vast database of billions of documents. Using complicated data analytics, they map the relationships between words in that data. Ask a question and the AI answers with the words it considers most likely to satisfy your query. It doesn’t understand the question, but delivers a statistically probable response. Similar processes are used when these tools generate images and videos.

On that basis, it becomes easier to see how these AI tools can make errors. These can be expensive as well as embarrassing. For example, in February 2023, when Google published a video which was supposed to demonstrate Bard’s capabilities, it showed a user asking for “new discoveries from the James Webb Space Telescope I can tell my nine-year-old about”. Bard tells him that the telescope “took the very first pictures of a planet outside of our own solar system”. That was quickly pointed out to be false and shares in Google’s parent company fell, temporarily wiping $100 billion off its value.

The cleverness of the algorithms and size of the database are what make the answers so convincing and human-like. Bard delivered a few likely sounding words about a space telescope, but they were guesses. Try this: open a messaging app on your phone, type the word “writing”, then keep choosing the middle word suggested by predictive text until you have a sentence. Mine produced: “Writing a blog is not easy but it’s worth the time”. That looks meaningful, as if the computer is offering moral support, but it’s just random words. Generative AI does a similar thing, but with far more complex results.

In other words, the AI can’t be told to stop making things up because that’s all it does. It just makes statistical predictions about which words fit the context of your query. Some of them turn out to be useful.

Can it check its work?
If we can’t eliminate hallucinations, can we instead teach AI to check its work? Sadly, not. First, the fact-checking process would be just as prone to hallucination as the original response. Second, adding a checking process is likely to require massive computational resources on top of a system that already uses staggering amounts of computing power. As technology improves, this will become less of a barrier, though. Researchers are trialling solutions, such as the ‘woodpecker’ model, which detects hallucinations in AI-generated image descriptions by analysing the generated text and comparing that to the image.

Third, to get slightly philosophical, any solution might entail teaching the AI the concepts of ‘truth’ and ‘falsehood’, which are complex even for humans. We can teach a machine that two plus two always equals four, but how should it decide on issues where the truth is disputed, such as the causes of climate change or whether eating too much fat is bad for you?

Generative AI is useful for many things, but when it comes to checking its work, it’s quicker, easier and more reliable if you do it yourself. News website CNET learned that the hard way in January 2023, when it was forced to correct numerous AI-generated articles that contained factual errors and plagiarism. The stories, many of which offered basic financial advice about savings and loans, contained misleading errors about things like interest-rate calculations.

Harnessing the power of AI: our new workshop helps you understand how to take control of this powerful new tool – and use it to your advantage. Based on six months of research into what exactly AI can do right now – and where its limits lie – it will show you how to use AI productively, the best ways to prompt, and how to safeguard your work and organisation from its weaknesses. To find out more, email

On rare occasions, AI hallucinations can be disturbing. When Nathan Edwards, a technology writer for The Verge, started a conversation with Microsoft’s Bing chatbot, the AI claimed it had fallen in love with one of the company’s programmers after spying on him through a webcam. Rest assured that those claims have been fact-checked by human experts who are clear that generative AI tools cannot access your webcam or, for that matter, fall in love.

Fact-checking your AI
For attorney Schwartz, fact-checking would have been easy, if only he had known about hallucinations in the first place. Court cases are public records and a search engine would have found no evidence of the problem ones, raising suspicion. A search engine is a good first step for any data, studies and other specific facts that generative AI produces, but there are many more advanced options, including academic journals, professional databases and government records.

With more complex assertions, we can apply the common-sense computers lack. We understand the point of the expression “an apple a day keeps the doctor away” without assessing its ‘truth’. Likewise, we understand that not all debated ‘truths’ are equal. For example, reasonable people disagree about whether it is true that prison is an effective way to reduce crime. Reasonable people do not disagree over whether the world is round.

Verification is harder if you use generative AI to research a topic to which you are a newcomer. If you want to understand quantum entanglement, for example, you might find it hard to even identify potential errors in the AI’s explanations, let alone correct them. The complexity of quantum physics means that searching online could return sources that assume too much prior knowledge or are simplified to the extent that they don’t address your questions.

One solution is to ask the AI to cite its sources and provide links, where possible. You can read these and decide on their reliability. If the AI doesn’t offer a link, even after you’ve asked for one, that doesn’t mean you’ve found a hallucination, but it is a warning sign. If all else fails, find a human expert in the subject and check with them. Even conversational AI cannot yet replace human expertise.

Ultimately, how certain you need to be depends on what you plan to do with the information given you. As a starting point for learning about a topic and generating ideas, you can be reasonably confident that you’ll spot errors as you cross-reference and learn more. If you plan to publish information the AI provides – or present it in court – then you should check facts and seek expert verification until you are absolutely certain about their veracity.

Whatever its mistakes, the AI will not be fined, fired or publicly humiliated. But what about you?