Summarize the history instead of shipping all of it

A backend team at a travel booking company was paying for every word twice. Their assistant handled long planning conversations, and on each new turn they resent the entire history to the model. Turn twenty carried all nineteen turns before it. The bill climbed with the length of the chat, and the answers got looser the longer a session ran.

They assumed they had to send everything. If the model does not see the whole conversation, they reasoned, it will lose the thread. So they kept stuffing the full transcript in and eating the cost.

Raw history is expensive and noisy

Sending the full history feels safe, but it works against you on two fronts. The first is money. You pay for input tokens, and a growing transcript means you re-pay for the same old turns on every single call. A long conversation becomes a slow leak.

The second is focus. Most of what happened twenty turns ago is settled. The user picked a destination, ruled out some dates, agreed on a budget. The exact wording of how they got there is noise now. When you ship all of it raw, the model has to wade through a pile of resolved back-and-forth to find the few facts that still matter. The important state is in there, buried in chatter, competing for attention with everything that is no longer relevant.

So full history costs the most and helps the least at exactly the point a conversation gets long, which is where you need it to hold together most.

A compact summary carries what matters

The team’s fix was to stop shipping the raw transcript and start shipping a rolling summary. After each turn they updated a short running note of the state that mattered: the decisions made, the constraints agreed, the open questions. That summary, plus the last few verbatim turns for immediate context, replaced the wall of history.

The results moved in the right direction on both fronts. The prompt got smaller, so the cost per turn stopped climbing with the length of the chat. And the model stayed on track, because the settled facts were now stated plainly in a few lines instead of being reconstructed from a long transcript every time. A summary is a decision about what is worth keeping. Raw history is a refusal to make that decision.

How to make this work in practice:

Keep a rolling summary of state. Maintain a short note of decisions, constraints, and open threads, and update it as the conversation moves.
Keep the last few turns verbatim. Recent messages carry the immediate context the summary is too coarse to hold, so send those in full.
Summarize the far-back turns, do not delete them. Compress old history into the summary rather than dropping it, so nothing important silently disappears.
Spend the freed budget on what is current. Use the room you saved for the retrieved facts or fresh input that actually drives the next answer.
Check the summary keeps the load-bearing facts. Review that the running note preserves the details the task depends on, since a summary that drops a key constraint is worse than none.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500 that scopes real production work before you build, we look hard at how a feature carries its history. Raw full-history prompts are one of the most common reasons an AI feature costs more than it should and drifts as sessions grow. We design the summarization strategy up front: what to keep verbatim, what to compress, and how to be sure the compression does not throw away a fact the answer needs. It is cheaper to design that in week one than to notice it on a cost report a quarter in.

You do not have to send everything to remember everything. Send a good summary and the last few turns.