You cannot grep a prompt: logging AI calls so incidents are debuggable

I worked with a team at a large ad platform that shipped an AI feature to summarize campaign performance for advertisers. One morning a customer reported a summary that named a competitor’s brand that had never been in their account. The support ticket landed on an engineering lead. She opened the code, found the function that built the prompt, and stopped.

There was no record of the actual call. Not the prompt that went out. Not the context that got stitched in. Not the model version. Not the output the customer saw. The code that built the request was still there, but the inputs that flowed through it that morning were gone.

She could read the recipe. She could not see the meal anyone ate.

The lesson: log the full call envelope, not just the code that builds it

An AI call is a runtime event, and like any runtime event it needs a log line you can search later. The code that assembles a prompt is not the same as the prompt that was sent. Retrieved context changes per request. User input changes. The model version changes when a provider updates it under you. Sampling parameters change when someone tunes them. Reconstructing all of that from source months later is guesswork.

So log the envelope. For every AI call, capture:

The final prompt or message array as it was actually sent, not a template.
The retrieved context or documents that were injected, with their source IDs.
The model name and version string the provider returned.
The parameters (temperature, max tokens, tool definitions) for that call.
The raw output, including any tool calls or refusals.
A request ID and timestamp you can join against your other logs.

With that in hand, the ad platform incident took twenty minutes instead of two days. The log showed a retrieval step had pulled a document from the wrong tenant. The prompt was fine. The model was fine. The context was poisoned upstream, and the log made that obvious.

Handle the PII before it becomes a second incident

The reason many teams do not log the envelope is that AI prompts often carry personal data, and dumping raw prompts into a general log store is its own problem. That fear is reasonable, and it is solvable.

What careful teams do:

Route AI call logs to a store with tighter access controls than general application logs.
Redact or tokenize known PII fields on the way in, keeping a stable hash so you can still correlate.
Set a real retention window instead of keeping everything forever.
Log references to large retrieved documents (IDs and versions) rather than pasting full bodies where you can avoid it.
Write down who can read the logs and why, before the first incident, not during one.

The point is to make logging a deliberate design decision with a data-handling plan attached. Skipping the log is not a privacy strategy. It just means the next incident is unsolvable and the PII is still sitting in the provider’s request history anyway.

How we approach it at Density Labs

Most teams do not discover this gap until an AI feature already misbehaved in front of a customer. By then the calls that would have explained it are gone. The industry pattern holds here too. Roughly 95 percent of enterprise AI pilots stall on integration and operations rather than model quality, and missing observability is a big part of what “operations” means in practice.

Our AI Opportunity Assessment is a fixed two-week engagement for a fixed 2,500 dollars. We scope the real production work before you build. For a feature like this, that means looking at whether every AI call leaves a searchable record, whether the envelope is complete enough to reconstruct an incident, and whether the PII handling is deliberate rather than accidental. We write down exactly what to log, where it should live, and how long it should stay.

You cannot debug what you did not record. Log the call before you ship the feature.