What to log now so you can evaluate the AI later

The evaluation you will want to run in three months depends on data you are either capturing today or throwing away today. Teams learn this in the wrong order. A feature misbehaves in production, someone goes to investigate, and the logs hold the output but not the input, or the answer but not the context the model saw. The question is answerable. The data to answer it is gone.

You cannot reconstruct what you did not record

A staff engineer at a networking-hardware company framed the problem the way a hardware person would. If you did not instrument the test point, you are guessing. For an AI feature the test points are specific. The exact input the user sent. The context or documents retrieved and passed to the model. The full output. The model and prompt version that produced it. What the user did next. Miss any one of these and a whole class of question becomes unanswerable after the fact.

That last item, what the user did next, is the one teams skip and later miss the most. Whether the user accepted the output, edited it, ignored it, or immediately asked again is the closest thing you have to a live quality signal. If you are not recording the user’s reaction, you are running blind on the only judge that pays you.

Logging is becoming table stakes, not a nicety

This is not a fringe practice anymore. Gartner projects that LLM observability will be part of roughly half of generative AI deployments by 2028, up from around 15 percent in early 2026. The teams building it in now are the ones who will be able to answer why a feature regressed. The teams who wait will be reading incomplete logs during an incident, which is the most expensive time to discover what you failed to capture.

A director of engineering at a messaging startup tied logging to versioning on purpose. Every output in his logs carried the model version and the prompt version that made it. When a change degraded quality, he could split the logs cleanly at the release and compare, instead of arguing about whether the feature used to be better. The version stamp turned a debate into a query.

The log lines to write before launch

The full input and the retrieved context, exactly what the model saw, not a truncated preview.
The complete output and its version stamp, the model and prompt version that produced it.
The user’s next action, accepted, edited, discarded, or retried, as your live quality signal.
A correlation id per interaction, so input, context, output, and outcome can be stitched back into one story.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement, logging is one of the design inputs we insist on before a build starts, alongside data readiness and integration. We map what has to be captured so the feature can be evaluated and debugged after launch, and we check that nothing you will need in an incident is being silently dropped today. It is unglamorous work that costs a little now and saves a painful, unanswerable investigation later.

The evaluation you run after launch is only as good as the data you started keeping before it. Decide what to log now, while it is a design choice, not during an incident, when it is a regret.