The crash report that shipped a customer’s data to a third party

A reliability engineer at a SaaS company described a finding that came out of a routine vendor review. Their AI feature used a standard error monitoring service, the kind that captures the state of a request when something throws an exception, so engineers can debug it. That is genuinely useful. It also meant that whenever the feature errored, the tool grabbed the local context, which included the prompt, and the prompt was full of customer data. That data was now sitting inside a third-party monitoring vendor that had never been reviewed for holding personal information, because nobody had connected the two facts.

The exhaust paths of an AI feature leak data in ways the main data flow does not. Error reporting, performance monitoring, session replay, analytics. Each of these tools is built to capture context, and when the context is a prompt, they capture the sensitive part along with it. They do it silently, they send it somewhere outside your systems, and they were often approved for a job that predates anyone thinking about AI prompts.

Monitoring tools capture the sensitive part by design

The purpose of error monitoring is to give you enough context to reproduce a failure. That means it deliberately grabs variables, request bodies, and state at the moment things went wrong. For a normal feature, that context is mostly harmless. For an AI feature, the most important variable is often the prompt, and the prompt is exactly the thing carrying personal data.

The same is true across the other observability tools. Session replay records what the user did, which can include what they typed into an assistant. Analytics events sometimes carry payloads that include model inputs or outputs. Performance traces attach request details. All of them were set up to help you operate the product, and all of them can quietly forward sensitive AI content to third parties.

What makes this hard to catch is that these tools were approved long ago, for a purpose that seemed unrelated. The vendor review happened years before the AI feature existed. Nobody went back to ask whether the monitoring stack was now capturing prompts full of customer data, because the monitoring stack was old news.

Audit the exhaust, not just the main flow

The fix is to treat every tool that captures request context as a place your prompt data can leak, and to review it the same way you would review the model provider.

What to check:

Find what your monitoring captures. For each error, performance, replay, and analytics tool, determine whether prompts, responses, or model context can end up in what it records.
Scrub before it is captured. Configure these tools to strip prompt content and personal data from what they collect, or exclude the sensitive fields entirely.
Review the vendors for this use. A tool approved for generic error reporting may not be approved to hold customer personal data. Check whether the current use fits the agreement you have.
Include exhaust in your data map. When you inventory where sensitive data flows, the monitoring tools belong on the map, because they are outbound paths too.

The teams that handle this stop thinking of monitoring as separate from data governance. A tool that captures context is a tool that captures whatever is in the context, and for an AI feature that is the sensitive part.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we follow the exhaust paths as carefully as the main flow. We look at what the error, performance, and analytics tooling captures when the feature runs and when it fails, and whether prompt data is riding along to a third party. It is one of the leaks teams are most surprised to find, because the tool was doing exactly what it was set up to do.

Your monitoring tools capture context on purpose, and for an AI feature the context is the sensitive part. Audit the exhaust, or it will ship your customers’ data somewhere you never chose.