The retrieved chunk that pushed your instructions out of the window
A RAG feature pulled so much text that the retrieved passages crowded the system instruction against the limit. Retrieval and instructions were fighting over one budget, and retrieval won.
The retrieved chunk that pushed your instructions out of the window
A machine learning engineer at a legal-tech firm was watching a RAG feature misbehave in a way that made no sense to him. The system prompt was solid. He had tested it in isolation and the model followed it perfectly. In production, with retrieval turned on, the same instruction seemed to lose its grip. The model got sloppy about the rules exactly when real documents were in play.
We looked at what the model actually received on a bad request. The retrieved passages were huge. His retriever was pulling many chunks, and each chunk was large, and together they nearly filled the window. The system instruction was still in there. It was just crushed into a corner by a mountain of retrieved text.
Retrieval and instructions share one budget
Here is the thing that is easy to forget. The context window is one fixed space, and everything competes for it. The instruction lives there. The conversation lives there. The retrieved documents live there too. They are not separate budgets. They are roommates in one room, and if one of them takes up most of the space, the others get squeezed.
His retriever behaved as if it had the window to itself. It grabbed generously, on the theory that more context means better answers. But every token it spent was a token the instruction and the model’s own output could not use. Past a certain point the retrieved text was not helping. It was displacing the very rules that told the model how to use it.
That is why the instruction worked alone and failed in production. Alone, it had the whole window. In production, it was a small block wedged against the limit by passages that had eaten the rest. The model was not ignoring the rules on purpose. The rules had been shoved to the margin of everything it was reading.
Cap retrieval and protect the instruction
The fix is to stop treating retrieval as if it can take whatever it wants. Retrieval gets a share of the window, and that share has a ceiling. The instruction block gets protected space that retrieval is never allowed to eat into. You decide the split on purpose instead of letting the retriever expand until something breaks.
More retrieved text is also not the same as better retrieved text. Pulling ten large chunks when two precise ones hold the answer just adds noise and burns budget. Tightening retrieval to fewer, sharper passages usually helps the answer and frees room at the same time. The goal is the right context, sized to fit, with the instruction safe.
Ways to keep retrieval in its lane:
- Cap retrieval’s share of the window. Set a hard ceiling on how much space retrieved content can take, and enforce it before you build the prompt.
- Reserve protected space for the instruction. Give the system rules a budget retrieval cannot touch, so they never get squeezed to the margin.
- Retrieve fewer, better chunks. Tune for precision over volume, since a couple of on-point passages beat a stack of loosely related ones.
- Trim or summarize oversized chunks. Shrink long passages before they go in the prompt, rather than shipping whole documents when a section would do.
- Check the assembled prompt under load. Look at the real prompt on a heavy retrieval case and confirm the instruction is still prominent, not buried.
How we approach it at Density Labs
In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500 that scopes real production work before you build, we treat the window as a shared budget and design the split before anything ships. We decide how much of it retrieval may claim, we protect the instruction block, and we test the feature on heavy cases where the retrieved text is largest, because that is where the instruction quietly gets crowded out. A system prompt that works in isolation and fails in production is one of the most common surprises in RAG, and it is far cheaper to catch it in week one than to debug a model that has been half-ignoring its rules for a month.
Your instruction and your retrieved documents are fighting over the same room. Give the instruction a seat that retrieval cannot take.