Golden transcripts: the integration test for a conversation

A tech lead at a telecom company described a conversational AI feature that passed all its tests and still frustrated customers. Each test checked a single exchange, one user message in, one assistant reply out, and every reply looked correct on its own. What the tests never checked was what happened over a full conversation. The assistant would answer a question well, then contradict itself two turns later, forget a constraint the user had stated at the start, or repeat a question it had already asked. Every individual reply was fine. The conversation was a mess. The team was unit-testing the turns and had no integration test for the thing customers actually experienced.

A conversation is more than its turns

When you test one exchange at a time, you are testing the feature the way you test a pure function. Given this input, is the output correct. For a stateless request that is a reasonable model, because the output depends only on the input in front of it.

A conversation is not stateless, and that is the whole point of it. Each turn depends on everything that came before. The user says their account number in turn one, states a constraint in turn two, asks a question in turn four that only makes sense given turns one through three. The assistant’s job is to carry all of that forward, and the failures that matter are failures of carrying it forward. It forgets the account number. It ignores the constraint. It contradicts something it said earlier. None of these can be caught by a test that looks at one turn in isolation, because the bug lives in the relationship between the turns, not in any single one. Testing turns and expecting conversation quality is like testing functions and expecting the system to work. You have tested the units and skipped the integration.

The reason teams fall into this is that single-turn tests are easy and multi-turn tests are not. A single exchange is a clean input-output pair. A conversation is a sequence with state, and testing it means replaying the whole sequence and checking properties that span it. It is more work, so it gets skipped, and the feature ships with strong unit coverage and no integration coverage, which is exactly the coverage profile that lets a feature pass tests and fail users.

Test the whole conversation, not just the turns

The lesson is that a multi-turn feature needs tests that run whole conversations and check the properties that only exist across them. Teams build these as golden transcripts, saved conversations the feature must handle correctly end to end.

What a golden-transcript test checks that a single-turn test cannot.

Consistency across turns, so the assistant does not contradict something it said earlier in the same conversation.
Memory of stated facts and constraints, so information from turn one still shapes the answer in turn five.
Non-repetition, so the assistant does not re-ask what it already asked or re-explain what it already covered.
Correct handling of references, so a question that points back to an earlier turn resolves to the right thing.
Graceful recovery, so a confused or hostile turn does not derail the rest of the conversation.

Build these from real conversations, especially the ones that went wrong, and run them as integration tests. They catch the class of failure that customers feel and single-turn tests never see.

How we think about it at Density Labs

The conversational feature that passes its tests and annoys its users is one of the most common gaps we find, and it comes from testing at the wrong grain. The team built solid single-turn tests, felt covered, and shipped a feature whose real failures all live in the space between the turns, which nothing was testing. The unit tests were green. The integration was untested, and the integration was the product.

When we help a team ship a conversational feature, we make golden transcripts part of the suite, so the thing that gets tested is the conversation the customer has, not just the isolated replies. It is more work to set up, and it is the difference between a feature that passes tests and a feature that holds up.

Testing every turn tells you the turns are fine. It says nothing about whether the conversation is, and the conversation is what you shipped.