The evaluation set full of real customer data

A machine learning engineer at a document processing company told me about a file that had quietly become a liability. Their evaluation set, the collection of examples they ran the feature against to measure quality, had been built from real production cases, because real cases are the most honest test. That was a good instinct for quality. It also meant the eval set was a curated file of real customer data, and unlike production data, it did not move. It got copied into repositories, shared with everyone working on the feature, checked into version control, and kept indefinitely, because an eval set is supposed to be stable so results stay comparable over time.

The properties that make a good eval set are the properties that make a data protection problem. It is stable, so it sticks around. It is shared, so everyone on the team has a copy. It is versioned, so old copies persist in history. When that set is built from real customer data, all of those helpful properties apply to personal information, and they apply for a very long time.

A good eval set is stable, shared, and permanent

Think about what you want from an evaluation set. You want it to stay the same, so that a score today means the same thing as a score last month. You want everyone working on the feature to run against it, so results are consistent. You want to track it over time, so you can see whether the feature is improving or regressing. Stability, breadth of access, and permanence are features of a good eval set.

Now apply those features to real customer data. Stable means the personal information does not get cleaned up on any schedule. Shared means it lives on many machines and in many repositories. Permanent means it persists in version history even after you think you removed it. The eval set becomes one of the most widely distributed and longest-lived copies of customer data you have, sitting outside your production data controls entirely.

And retention makes it sharper. Production records get deleted when a customer leaves or a policy requires it. The eval set does not follow those rules, so it can hold data about people long after every other copy is gone, in a file that half the team has cloned.

Handle eval data as the sensitive asset it is

The fix is to keep the quality value of realistic evaluation without turning the eval set into an unmanaged store of personal data.

What helps:

De-identify the eval set. Strip or replace personal data while keeping the structure and difficulty that make the examples realistic. A good eval set tests behavior, and behavior usually survives de-identification.
Control access like production. If the set must contain real data, it deserves the same access limits and encryption as the source, not a spot in a widely cloned repository.
Keep it out of version control if it holds real data. Committed data lives forever in history. An eval set of real records is exactly the thing you do not want permanently embedded in a repo.
Apply retention. The eval set has to respect the same deletion rules as the data it was drawn from, or it becomes the copy that outlives every other.

The teams that get this right treat the eval set as an asset that carries the same weight as the data it came from. The realism they want almost always survives de-identification, which means they rarely have to trade quality for safety at all.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at how the evaluation set was built and how it is stored. Whether it contains real customer data, who can reach it, whether it lives in version control, and how long it persists. A stable, shared, permanent file of personal data is easy to create by accident when you are focused on measuring quality, and easy to fix once someone names it.

The eval set you built for quality can quietly become your most widely copied and longest-lived store of customer data. De-identify it, and keep the realism without the liability.