Stale embeddings: the knowledge base that answers from last quarter

The documents got updated. The embeddings did not. Your RAG system is now answering from a snapshot of the world taken the day the index was built, and it has no idea it is behind.

Built once, quietly out of date

A product manager at a data-pipeline company described the failure mode that haunts anything that moves data between systems. You build a connector, it works, and then an upstream source changes and your feed quietly stops matching reality. The pipeline did not break loudly. It just kept delivering yesterday’s shape of the data while everyone assumed it was current.

A vector index has the same disease. Someone builds it once during the pilot, embeds every document as it existed that week, and ships. Then the real work continues around it. The pricing page gets revised. A policy is replaced. A product is renamed. A section is deleted because it was wrong. The source content moves on. The embeddings sit frozen at the moment they were created, because re-embedding was never wired into the process that edits the documents.

The answer sounds current and is not

Here is what the user sees. They ask a question. The system retrieves the chunk that best matches, which is the old version, because the old version is the only version in the index. The model reads that chunk and writes a clear, grounded answer. It even cites the source. Everything about the response signals freshness and confidence, and the content is from last quarter.

That is worse than a model that admits it does not know. A blank answer makes someone go check. A confident answer from a stale embedding makes them act on information that was retired months ago. The pipeline PM’s framing fits here exactly. A feed that quietly delivers the wrong data does more damage than one that fails loudly, because nobody is looking for a problem that never announced itself.

Freshness is a property of the corpus, not the model

Teams treat the embedded corpus as a thing they built rather than a thing they maintain. Once it exists, attention moves to the model and the prompt, and the index falls off everyone’s radar. Meanwhile the gap between what the documents say and what the embeddings encode widens by a little every week. There is a rough bar worth remembering here. AI features generally need something like 85 percent accuracy or better to be dependable. A knowledge base slowly drifting away from its own source content walks that number down month after month, and no one notices because each individual answer still reads well.

The uncomfortable part is that a stale index gives you no error to catch. The retrieval succeeds. The generation succeeds. The confidence scores are high. Every automated signal says the system is healthy, because the system is faithfully answering from a corpus that stopped being true. The only way to see it is to compare a retrieved chunk against the live document and notice they no longer say the same thing.

What good teams do

The teams that avoid this treat freshness as a measured property. They track when each document was last changed and when its embedding was last computed, and they watch the gap between those two dates. When a document is edited, its embedding is regenerated as part of that edit, not left for a cleanup that never comes. The index becomes something with a known age instead of a black box someone built once and forgot.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we check whether the embedded corpus still matches the source it came from. We sample documents, compare the live version against what the index encodes, and look for how far the two have drifted apart. When we find the index answering from an old snapshot, we flag it before it grounds a wrong decision, because a confident answer from a stale embedding is the hardest kind of wrong to catch.

An index that never gets refreshed is a memory of the truth, not the truth.