Governing the data your model sees at inference, not just at training

Ask a team how they govern their AI data and they will describe the training set. What was it trained on, was it licensed, was it cleaned. Fair questions. They almost never describe what reaches the model at runtime, and that is where the sensitive data actually moves.

Training data is the part everyone watches

Training data gets attention because it feels permanent. It went into the weights, so it feels like a decision you make once and live with. Teams write it down. They ask about provenance. They keep a record.

Then the feature ships, and a different stream of data starts flowing. A user pastes a customer record into a prompt. A retrieval step pulls three internal documents into the context window. A tool call returns a row from a production database and hands it to the model. None of that was in the training set. All of it is now data the model sees, and most governance programs have nothing to say about it.

An engineer who drew the boundary on purpose

An engineer I talked to had thought this through more carefully than most. His rule was simple. Sensitive data stays inside the company’s own systems, behind a middleware layer, with clear boundaries on exactly what gets shared outward and what never leaves.

He was not doing this only for compliance. He had found that a local, narrow model gave him more consistent results, because it learned a smaller and highly relevant dataset and hallucinated less. The boundary was a design choice with two payoffs. Sensitive records did not travel to a third party, and the model got better because the data it saw was tight and on-topic. The middleware decided what crossed the line. Everything else stayed home.

That is inference-time governance in practice. It is not a policy document. It is a layer in the architecture that knows which fields can go to the model, which must be masked first, and which never leave the building.

Where the leak actually happens

Picture the demo path. A support agent opens a ticket, clicks a button, and the feature drafts a reply. To draft it, the system retrieves the customer’s last five interactions and sends them, in full, to a hosted model. The account number was in there. So was a home address. Nobody decided that on purpose. The retrieval step grabbed the whole record because that was the easy way to build it, and no boundary said stop.

This is the gap. Training data gets reviewed by a person before it is used. Inference data flows automatically, thousands of times a day, assembled on the fly from prompts and retrieval and tool outputs. If the only governance you have sits on the training set, you are watching the door that gets opened once and ignoring the one that opens every second.

What controlling inference data looks like

The teams that handle this treat the runtime context as governed data with the same seriousness as the training corpus. A few concrete moves show up again and again.

Decide, per field, what is allowed to reach the model, and mask or drop the rest before the call.
Put a boundary in code, a middleware layer, so the decision is enforced and not left to whoever wrote the prompt.
Log what was sent, so you can answer later what data left the building and when.
Prefer a narrow local model for the sensitive paths, because keeping the data in-house and the model focused solves two problems at once.

None of this slows the feature down in a way a user notices. It moves the governance question to where the sensitive data actually travels.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we trace the full inference path, not just the training story. We map what a single request actually sends to the model: the prompt, the retrieved documents, the tool outputs. We flag the fields that should never leave your systems, and we check whether a boundary exists to stop them. Training-set governance is table stakes. The runtime path is where we usually find the exposure nobody had drawn on a diagram.

Govern the data the model sees every second, not only the data it saw once.