AI Hallucination
Output from an AI model that is presented confidently but is false, unsupported by its sources, or inconsistent with the context it was given.
Hallucinated facts in contracts, support answers, reports or code can cause legal, financial and reputational damage, so output verification must be part of any process that relies on generative AI.
A legal operations team requires its contract-review assistant to quote the exact clause behind each finding, drops any finding without a matching quote, and has a lawyer review every flag on contracts above €50,000.
Mitigations reduce hallucinations but do not remove them, and retrieval can still return wrong or outdated documents. A confident tone is not a reliable signal of accuracy, and some errors can only be spotted with domain expertise.
NIST's Generative AI Profile names the risk confabulation: systems generate and confidently present erroneous or false content, which may also contradict the prompt or earlier statements. It arises because models predict likely text rather than check facts. Controls include grounding answers with Retrieval-Augmented Generation (RAG), requiring quotes or citations, allowing the model to say it does not know, AI Guardrails and Human-in-the-Loop (HITL) review.
AI Governance
NIST — Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf; Claude Platform Docs — Reduce hallucinations — https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations