What if the AI just makes things up? How we keep an agent from hallucinating in production

Hallucination is a design problem, not an inevitability

Here is the one idea I want you to walk away with: an agent that makes things up is an agent that was poorly fenced in.

The model does not know what your company knows. It knows what it saw during training, all mixed together, with no date, no source. Ask it about your return policy when you never gave it one, and it will produce something that sounds like a return policy. Plausible, well written, and false.

That is why I do not fight the model. I work on the environment around the model. And there, I do have concrete levers.

Let me walk you through them one by one. None of them is magic. Together, they bring the risk down from "anything goes" to something you can manage.

Lever 1: anchor the agent in YOUR data, not in its memory

The first rule: the agent should not answer off the top of its head. It should answer from your documents.

The technique is called document retrieval (in English, RAG). The principle is easy to grasp. Before answering, the agent searches your sources (your knowledge base, your PDFs, your catalog, your procedures) for the snippets that relate to the question. We hand those snippets to the model. And we ask it to answer using only that.

The difference is huge. Without this, the agent improvises from its fuzzy memory. With this, it reads your text and summarizes it.

One upside people underestimate: you can ask it to cite where each piece came from. "According to document X, section Y." When an agent cites its source, two things happen. The user can verify. And you spot it fast when something does not add up.

Honest caveat: this works if your data is clean. If your knowledge base is messy, contradictory or outdated, the agent will retrieve garbage and summarize garbage. The boring work of tidying up your information is half the project. There is no shortcut.

Lever 2: give it permission to say "I don't know"

By default, a model would rather invent than stay quiet. You have to reprogram that reflex.

In the agent's instructions we write something explicit. If the answer is not in the retrieved sources, do not make it up. Say you do not have the information. And, where relevant, hand the conversation over to a human.

Sounds obvious. Almost nobody gets it right the first time.

Because you have to test it with trick questions. Things outside your domain, data that does not exist, edge cases. And check that the agent really answers "I don't have that information" instead of improvising. An agent that knows how to say "I don't know" is worth more than one that seems to know everything.

Lever 3: a closed scope

The broader the agent, the more it invents. It is mathematical.

So I define a scope. The agent answers on what we authorize it to handle, and nothing more. A support agent for your product does not weigh in on the competition, does not give legal advice, does not improvise on topics that are not its own. When something falls outside, it says so and redirects.

This is not just safety. It is quality. An agent specialized in three things does them well. An agent that tries to do everything does it all halfway, and at the edges, it hallucinates.

Closing the scope also protects you from detours. People test agents. They ask for cooking recipes, political opinions, a poem. A well fenced-in agent replies, in a friendly way, that this is not its subject.

Lever 4: a human on the risky cases

Here is the question that changes the whole design: what is the cost of an error?

If your agent suggests the wrong blog article, not much happens. If it confirms a refund that should not be granted, or gives guidance on a sensitive topic, the cost is real.

So I map the cases. Where an error is cheap, the agent acts on its own. Where an error is expensive, the agent prepares the answer but a human validates it before it goes out. Or it simply hands the case to a person.

We call that a human in the loop. It is not admitting that AI fails. It is deciding, deliberately, where full automation makes sense and where it does not. A good agent knows when it is its turn to step aside.

Lever 5: log everything and test before production

What you do not measure, you do not control.

Every conversation the agent has is logged. Question, retrieved sources, answer. So when something goes wrong, we do not guess. We open the log, we see exactly what it retrieved and what it answered, and we understand why.

And before putting anything into production, we test with real cases. Not with textbook questions, neat and predictable. With the weird, badly written, ambiguous questions your customers actually send. That is where an agent breaks. Better it breaks in your test than in front of a customer.

This is the least flashy work in the project. It is also what separates an agent that holds up from a demo that falls apart at the first real-life question.

Zero risk does not exist (and be wary of anyone who promises it)

I will be blunt. None of these levers brings the risk to zero.

A well-designed agent can still get it wrong on a rare case. The goal is not perfection. The goal is controlled risk: errors that are infrequent, mild, detectable, and an agent that knows how to say "I'm passing this to a human" when it is unsure.

If someone promises you an AI that never gets it wrong, they are selling, not building.

And here our usual reflex comes back: it depends on your use case. The level of safeguards has to match what is at stake.

An assistant answering frequent questions about your opening hours needs few safeguards. If it gets it wrong, you give the wrong hours, you correct it, you move on. An agent that touches money, personal data or decisions with consequences needs the opposite: a narrow scope, human validation, fine-grained logging, hard testing.

You do not put the same safeguards on both cases. It would be needlessly expensive in one, dangerously thin in the other. The right question is never "how do I avoid all hallucination". It is "what happens if my agent gets it wrong here, and how much does that error cost".

That answer defines how much safety engineering you need. And that is a conversation worth having before you build, not after. If you want to review your case with us, write to us. But the underlying question, the one that decides everything, is yours: in your use case, how much does an error cost?