The knowledge base entry that told the AI to lie

A platform engineer at a company with an internal assistant told me about a failure that came from a single edited document. Their assistant answered questions by retrieving from a shared knowledge base, the standard retrieval setup. The knowledge base could be edited by a fairly wide set of people, because keeping it current was everyone’s job. At some point one entry was changed to contain wrong information, whether by mistake or on purpose was never fully clear. From then on, every user who asked a question that matched that entry got a confidently wrong answer, delivered with the authority of the assistant and sourced from a document that looked as official as any other. One corrupted entry had quietly poisoned the answers for everyone, and because the assistant retrieved and presented it faithfully, nothing flagged that the source itself was bad.

The integrity of what an AI retrieves is a security property, and it is easy to overlook because retrieval feels like reading rather than trusting. A retrieval-based feature is only as trustworthy as its knowledge base, and if that knowledge base can be corrupted, then a single bad entry can corrupt the answers to everyone who touches it. The model did nothing wrong. It retrieved the best match and presented it, which is its job. The failure was upstream, in the integrity of the source, and it affected every user at once because they all drew from the same corrupted well.

A shared source is a shared point of failure

Retrieval systems concentrate trust in the knowledge base. Every answer is only as good as the documents behind it, so the knowledge base becomes a high-value target and a single point of failure at the same time. If it can be edited widely, then the trust the whole feature depends on rests on the integrity of edits made by many hands, some of which may be careless and some of which may be hostile.

This scales in a way that direct data entry does not. A wrong entry in the knowledge base does not affect one interaction. It affects every future query that retrieves it, for every user, until someone notices and fixes the source. One edit poisons the well, and the poison flows to everyone who drinks from it, wearing the same authoritative presentation as every correct answer. Users have no way to tell a poisoned answer from a good one, because both come from the assistant citing a document that looks legitimate.

The reason this hides is that the model and the retrieval are working correctly. There is no bug in the code, no exploit in the model. The system faithfully retrieves and presents what is in the knowledge base, which is exactly the problem when what is in the knowledge base is wrong. Integrity was assumed, and assumed integrity is not enforced integrity.

Protect the integrity of what you retrieve

The fix is to treat the knowledge base as a trusted asset that needs its integrity protected, the same way you protect other things the system depends on.

What that involves:

Control who can change it. Limit and track who can edit the knowledge base, because every editor is a potential source of corruption, accidental or deliberate.
Review changes to the source. Meaningful edits to what the assistant retrieves deserve a review step, so a wrong or hostile change is caught before it reaches users.
Track provenance and changes. Know where each entry came from and what changed it, so a poisoned entry can be traced and the blast radius understood.
Watch for answers that drift. A sudden change in what the assistant says about a topic can signal that the underlying source was altered, which is worth catching quickly.

The teams that get this right treat the knowledge base as part of their trust boundary, not as a convenient shared document store. They protect its integrity deliberately, because they understand that a retrieval feature hands the knowledge base’s trustworthiness directly to every user, and a single corrupted entry spends that trust for all of them.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at the integrity of what a retrieval feature draws on. Who can change the knowledge base, whether changes are reviewed, and how a corrupted entry would be caught. A poisoned source is a quiet and wide-reaching failure, because the model presents bad information as faithfully as good, and every user pays for one bad edit. We would rather protect the knowledge base’s integrity in a diagnosis than trace a wave of wrong answers back to a single entry.

A retrieval feature is only as trustworthy as its knowledge base, and a single corrupted entry poisons the answers for everyone. Protect the integrity of what you retrieve, because the model will present a bad source as confidently as a good one.