Treat model output as untrusted input

A backend engineer at a logistics company told me about a subtle bug that turned into a security review. They had done the responsible thing on the input side. User text was validated, escaped, checked. Then they took the model’s answer and used it to build a database operation, because the model was theirs and they trusted it. During testing, a crafted input steered the model into producing output that, when dropped into that operation, did something the team never intended. The input was clean. The output was not, and nobody was checking the output.

The lesson took a second to land, because it runs against instinct. The model feels like part of your own system, so its output feels trusted. It is not part of your system in the way that matters. It is a source that can be influenced by anyone who can reach its input, which makes its output exactly as trustworthy as the least trustworthy thing that reached it.

The model is downstream of every input it saw

Think about where model output comes from. It is shaped by your prompt, by the user’s text, by any documents or data you fed in, by anything an agent retrieved. Any of those can be hostile. So the output that comes back has, in effect, passed through untrusted hands. If you then use that output to build a query, render HTML, call an API, or write to a file, you are letting untrusted content drive a trusted action.

This is the same class of bug as the classic injection flaws that predate AI by decades. Data from an untrusted source flows into a place that interprets it as code or commands. We learned long ago to sanitize user input before it hits a database or a page. The AI feature quietly reintroduced the flaw by adding a new source, the model, and forgetting to treat it like a source.

The reason it slips through is trust by association. The model is your feature. The output is your feature talking. So it gets waved past the checks that any external input would have to pass.

Validate the output before it acts

The fix is to put the model on the untrusted side of the line and check what it produces before that output is allowed to do anything.

In practice:

Never build a query, command, or markup directly from raw model text. Parse it, validate it against what you expect, and use safe interfaces the same way you would for user input.
Constrain the output shape. Ask for structured output and validate it against a schema, so anything that does not fit the expected form is rejected before it flows onward.
Escape output at the point it is rendered. If model text reaches a browser, treat it like any untrusted string and escape it, so it cannot become active content.
Keep the sharp actions behind a check. If the output drives something destructive, validate the specific values, do not just trust that the model produced something reasonable.

The mental shift is small and it changes a lot. Stop asking whether you trust the model and start asking whether you trust everything that reached the model. Since the answer to the second question is almost always no, the output gets the same treatment as any other untrusted input, and a whole category of failure closes.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we trace what happens to model output after it is produced. Where it flows, what it drives, and whether anything validates it before it acts. Teams that carefully guard their inputs are often surprised to find the output flowing straight into a query or a render with no check at all.

The model sits downstream of every input it saw, and some of those inputs are hostile. Treat what it hands back the way you treat anything a stranger sends you.