The customer data you did not know you were sending out

A data protection lead at a healthcare-adjacent company told me about the day someone on the team printed out a full prompt, exactly as it was sent to their model provider. It was longer than anyone expected. To give the model enough context, the feature had been packing in customer records, and those records carried names, contact details, and history that nobody had decided to expose. The feature worked well, which is why no one had looked. It took reading one prompt end to end to see that personal data was leaving the building on every call.

When you use a hosted model, every prompt is an outbound transfer of whatever you put in it to a third party. That is easy to forget, because the prompt feels like an internal thing you are assembling. It is not internal once it is sent. It is data crossing your boundary to a company you do not run.

Context stuffing sends more than you think

The way these features get built almost guarantees the problem. To make the model useful, you give it context. More context tends to mean better answers, so the prompt grows. Customer records, past interactions, account details, whatever seems relevant gets pulled in. Each addition improves the demo. Each addition also increases what you are shipping to the provider.

Because it works, nobody audits it. The prompt is assembled in code, out of sight, and the feature returns good answers, so the assumption is that everything upstream is fine. Meanwhile the payload has quietly grown to include fields that would never have been approved for external transfer if someone had written them on a form and asked.

The risk is not hypothetical or purely legal. It is the plain fact that personal data is now sitting in requests to an outside service, subject to that service’s handling, retention, and terms, in ways your customers likely never agreed to.

Decide what is allowed to leave, on purpose

The fix starts with looking. You cannot govern what you have never read.

Steps that help:

Read a real prompt, in full. Take an actual production prompt and inventory every piece of data in it. This one exercise surprises almost every team.
Minimize what goes into context. Send the model what the task needs and nothing more. If a field does not change the answer, it does not belong in the prompt.
Redact or tokenize personal data before it leaves. Where the model does not need a real name or number to do its job, replace it with a placeholder and map it back on the way out.
Know your provider’s terms. Understand what the provider does with prompt data, how long it keeps it, and whether it is used for anything beyond serving your request.

The goal is to make the outbound payload a decision rather than an accident. Right now, for many teams, whatever ends up in the prompt ends up at the provider, and no one chose that. Choosing it, field by field, is most of the work.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we inventory exactly what leaves your systems on every model call. We read the real prompts, list the personal data inside them, and separate what the task needs from what got swept in. Then we design the minimization and redaction before the feature ships, not after a regulator or a customer asks what you sent.

Every prompt to a hosted model is data leaving your building. Read one all the way through, and decide on purpose what is allowed to go.