The analytics tool that recorded every word

A growth engineer at a consumer app company told me about a well-intentioned decision that created a quiet problem. The team wanted to understand how people used their new assistant, which is a reasonable thing to want. So they sent the conversations into their product analytics, the same place they tracked clicks and funnels, because that is where usage questions get answered. It worked. They learned a lot about behavior. They also turned their analytics warehouse, a system built for counting events, into a store of everything users typed to the assistant, including the personal things people tell an AI that they would never put in a form.

People talk to assistants differently than they interact with the rest of a product. They type in full sentences. They share context. They disclose things, sometimes sensitive things, because the interface invites it. When you pipe those conversations into product analytics, you are not just measuring usage. You are collecting a corpus of intimate, unstructured content into a system that was designed for none of the protections that content needs.

Analytics was built to count, not to hold secrets

A product analytics warehouse is optimized for a specific job. It counts events, tracks funnels, and slices behavior. It is usually broadly accessible inside the company, because the whole point is that lots of people can query it to answer questions. Its retention is generous, because historical trends are the value. Its access model is loose, because event data is not thought of as sensitive.

None of those properties are appropriate for the content of AI conversations. Broad access means many people can now read what users told the assistant. Generous retention means those conversations sit around for a long time. A loose access model means the sensitive content is protected far more weakly than the same information would be in your primary data store.

The mismatch is the whole problem. You took content that deserves the strongest handling and put it in the system with the weakest, because that system was where you go to answer usage questions. The usage questions were legitimate. The place you answered them was wrong.

Separate the metrics from the content

The fix is to get the behavioral insight you want without warehousing the raw conversations in a system built for counting.

What that looks like:

Log events, not transcripts. Capture the metrics you actually need, how often the feature is used, whether it succeeded, how long it took, without storing the words people typed.
Keep conversation content out of general analytics. If you need to review conversations for quality, do it in a controlled place with tight access and real retention limits, not the warehouse everyone can query.
Redact if content must be captured. Where some content is genuinely needed for analysis, strip personal data before it lands.
Set retention deliberately. Conversation content, if you keep any, should live under a real retention policy, not the open-ended horizon of an analytics store.

The teams that handle this keep a clean line between measuring how the feature is used and hoarding what users said while using it. You can have the first without the second, and the second is where the risk lives.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at where conversation content flows for the sake of analytics. We check whether transcripts are landing in systems built to count events, who can read them there, and how long they live. Getting usage insight is easy to do safely. Doing it by dumping conversations into the analytics warehouse is how a careful team ends up with a poorly guarded store of sensitive content.

Measuring how people use your assistant does not require keeping everything they said in your analytics warehouse. Capture the metrics, and leave the content out.