What to drop when the context window fills up

A tech lead at an insurance company came to me with a feature that failed only on the big cases. Small claims went through clean. Large claims, the ones with long histories and stacks of attached notes, came back wrong or half-formed. The pattern was strange until we looked at what happened when the prompt got long.

The window filled. And when a window fills, something has to be cut. His code did not decide what. So the framework underneath it made the call, silently, and it cut whatever fell off the end. Often that was the part that mattered most.

Something always gets truncated

Every model has a limit on how much it can read at once. Most of the time you live well under it and never think about it. Then a real case comes in with more content than the budget allows, and the system cannot send all of it. It has to leave something out.

The question is who chooses. If you have not written a rule, the default kicks in. A lot of defaults simply chop the text at the limit, keeping the front and dropping the tail, or keeping the tail and dropping the front. Neither knows which piece was load-bearing. In his case the truncation was eating the claim details while faithfully preserving boilerplate, so the model was reasoning about a claim it could no longer fully see.

The failure looked random because it depended on length. The model was not confused about insurance. It was answering with half the file missing, and nobody had chosen which half.

Decide the policy yourself

The fix is not a bigger window. A bigger window buys you time and then the exact same problem returns on a bigger case. The fix is to own the decision. Before anything gets cut, you say what is protected, what gets compressed, and what goes first when space runs out.

Think of the window as a budget you allocate on purpose. The instruction that defines the task is not negotiable, so it is protected. The critical facts for this specific case are protected too. Long supporting material can be summarized down instead of sent whole. Old, low-value filler is the first thing to drop. Once you rank things that way, a full window stops being a coin flip and becomes a plan.

A truncation policy worth having:

Protect the instruction block. The task and the hard rules never get cut, no matter how tight the budget gets.
Protect the facts this case turns on. Identify the content the answer actually depends on and put it in the keep-always tier.
Summarize before you drop. Compress long supporting material into a shorter form rather than deleting it outright, so the substance survives.
Rank what goes first. Decide the order of sacrifice ahead of time, oldest and least relevant first, so the system never improvises under pressure.
Log when you truncate. Record that a cut happened and what it removed, so a wrong answer on a big case is traceable instead of mysterious.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500 that scopes real production work before you build, a truncation policy is part of the design, not a thing you discover in production. We look at the largest realistic inputs, not the tidy demo cases, and we decide what is protected, what is summarized, and what is dropped first when the window is full. We make the framework stop choosing for you. It is a short conversation up front and a nasty class of bug to chase later, because it only shows up on the cases that are too big to ignore.

If you did not decide what to cut, you decided to let the wrong thing go. Choose first.