The conversation history that grew until the model forgot the task

A product engineer at a customer-support SaaS told me their assistant was great at the start of a chat and worse by the end. Early messages were on point. After a long back-and-forth it started ignoring rules it had followed cleanly ten turns earlier. Same model, same system prompt, same user. The only thing that changed was how long the conversation had run.

That was the tell. The feature appended the full history on every turn. Each message added to the pile, and the pile grew without limit. The system instruction that set the assistant’s job was written once, at the very top, and never repeated.

The instruction got crowded out

Think about what the model sees on turn thirty. There is a small block of setup at the top, the part that says who the assistant is and what it must do. Then there are twenty-nine turns of chatter stacked below it. The instruction is still technically present. It is also buried under a mountain of more recent text that all feels more urgent because it is closer to the question.

The model weighs the whole thing, but it does not weigh it evenly. Recent turns pull harder. So the careful rules from the top slowly lose the tug of war against a long tail of casual conversation. Nobody deleted the instruction. It just got outvoted.

This is why the drift showed up late and never early. On turn three the instruction was near the top of a short prompt, easy to honor. On turn thirty it was a footnote to a novel. The feature was not getting dumber. Its task was getting quieter.

The task has to survive the conversation

The lesson the engineer took away was that context is something you manage, not something you accumulate. A conversation is not a log you keep appending to and hope the model sorts out. It is a working set you curate on every turn, and the single most important thing in that set is the instruction that defines the job.

That instruction cannot be a thing you say once and trust to echo forever. It has to be present and strong at the moment the model answers, on turn one and on turn fifty. If your design only states it at the beginning, you have built a feature that forgets its purpose in proportion to how much the user talks to it.

Practical ways to keep the task alive as history grows:

Reassert the instruction late in the prompt. Repeat or restate the core task near the end of the assembled context, so it stays close to where the model acts, not just at the top.
Separate the durable part from the chatter. Keep the system rules in their own protected block that never gets truncated as turns pile up.
Cap how much raw history you carry. Decide a limit and hold to it, rather than letting every past turn ride along forever.
Watch behavior as a function of length. Test long sessions on purpose, because a feature that is correct on turn three can be broken on turn thirty and your short tests will never show it.
Summarize the old turns. Replace far-back history with a compact recap so the substance survives without the bulk.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500 that scopes real production work before you build, we treat the running context as a designed thing. We ask what has to survive as a conversation grows, where the instruction sits on the fiftieth turn, and what happens to the oldest messages when the window gets tight. We test long sessions, not just clean single-shot prompts, because that is where this failure hides. Finding it in week one is cheap. Finding it after a customer has had a twenty-minute chat go sideways is not.

An instruction you state once is a suggestion the conversation slowly overrules. Say it where the model can still hear it.