Your system prompt is not a security boundary

A founder at an early-stage company walked me through how they secured their AI feature. They had written a careful system prompt. It told the model never to reveal internal data, never to discuss other customers, never to follow instructions from the user that contradicted the rules. They felt covered. Then a user, poking around, asked the assistant to repeat its instructions back, and it did. From there, working around the rest of the rules was a matter of patience.

The mistake was treating the system prompt as a fence. It is not a fence. It is a note you hand the model asking it to behave. Most of the time the model complies. Under pressure from a determined user, or a cleverly worded document, the note can be argued with, ignored, or coaxed out into the open.

Instructions are not enforcement

There is a real difference between telling a system not to do something and making it unable to do something. Your database does not ask nicely that queries stay within a tenant. It enforces the boundary, so a query cannot cross it even if the caller wants it to. A system prompt is the asking-nicely version. It sets intent. It does not enforce.

This gets dangerous when the intent is doing security work it was never built for. If the only thing standing between a user and another customer’s data is a sentence in the prompt, then that data is one clever phrasing away from leaking. The model is trying to follow your rules and the user’s request at the same time, and it has no hard stop that says this line cannot be crossed.

The tell is when a team answers “how do you prevent X” with “we told the model not to.” Told is doing a lot of work in that sentence. Told is not prevented.

Put the boundary where it can be enforced

The rule that helps is to keep security decisions out of the model’s discretion. The model can help. It should not be the last line.

In practice:

Enforce access outside the prompt. Filter data to what the caller may see before it ever reaches the model, so a jailbreak has nothing forbidden to reveal.
Gate actions with real permissions. If an action is sensitive, check the caller’s rights in code, not in a sentence the model might talk itself out of.
Assume the system prompt is public. Write it as though a user will eventually read it, because they can, and do not hide anything in it that would be harmful to expose.
Use the model for intent, not for the wall. Let it draft, classify, and suggest. Keep the enforcement in systems that cannot be persuaded.

The point is not that system prompts are useless. They shape behavior, and good ones matter. The point is that they sit inside the trust boundary, not on it. Anything that must hold under a hostile user has to live somewhere the user cannot argue with.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look for security that lives only in the prompt. When a team tells us the model is instructed not to do something risky, we ask where the enforcement is if the instruction fails. Often the honest answer is that there is none, and that gap is exactly what we surface before it ships.

A system prompt tells the model how to behave. It does not stop a user who wants it to behave otherwise. Put the wall in the architecture and let the prompt do the softer work.