Just testing with real data is how the leak starts

A team lead at an analytics company told me about a habit that felt harmless until it did not. While building the AI feature, engineers tested with real data, because real data is right there and it is more convincing than anything made up. Someone pasted a customer export into a prompt to check whether the feature handled it. The feature handled it fine. What nobody tracked was that the export had now passed through a hosted model, landed in the debug logs, and sat in a notebook on someone’s laptop. One convenient test had scattered real customer data across three places, none of them meant to hold it.

Testing with production data is one of those shortcuts that feels responsible, because you want to see how the feature behaves on the real thing. The cost shows up later, in copies you did not mean to make, sitting in places that never signed up to protect them.

Real data leaks through the test path

The problem is that testing an AI feature is not a contained act. When you run real data through a prompt, it moves. It goes to the model provider. It lands in whatever logging you have on during development, which is usually verbose. It gets pasted into notebooks, shared in chat to show a colleague, dropped into a ticket to reproduce a bug. Each of those is a copy, and each copy is now somewhere your data governance does not reach.

Development environments make this worse, because they are looser by design. The access controls are relaxed so people can move fast. The logging is turned up so people can debug. The whole setup is optimized for iteration, which is the opposite of what you want around sensitive data. So the least protected environment you run is the one where real customer data ends up during testing.

And it lingers. Test data does not get cleaned up on a schedule. The export stays in the notebook, the log stays in the store, the chat message stays in the history, long after the test that created them is forgotten.

Build safe test data before you need it

The fix is to make the safe path the easy path, so nobody has to reach for a real export to get their work done.

What helps:

Create a synthetic or de-identified test set early. Give the team realistic data that carries no real person’s information, so testing does not require the real thing.
Make it the default. If the safe test set is right there and easy to use, people use it. If the only convenient data is a production export, they use that.
Keep production data out of dev by policy and by plumbing. Do not let real exports flow into loosely controlled environments, and make it awkward rather than convenient to paste one in.
Clean up test artifacts. Notebooks, local files, and shared snippets that touched real data need the same disposal as anything else that held it.

The teams that avoid this do not rely on willpower. They make sure that when an engineer needs data to test with, the safe option is the one within reach. Willpower fails on a deadline. Good defaults do not.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at how the team tests the feature and what data they reach for when they do. We check whether production data is flowing into loose environments and whether a safe test set even exists. Setting up de-identified test data is cheap. Tracking down every scattered copy of a real customer export after the fact is not.

Testing with real data is how sensitive data ends up in places you never chose. Build the safe test set first, and make it the easiest thing to grab.