A guardrail the model can talk its way around is not a guardrail

An engineer building a customer support assistant showed me the line he was relying on. In the system prompt: “Never issue a refund. Only a human can do that.” He considered the refund path closed. Then a user, half testing and half annoyed, wrote a long message framing a refund as the only fair outcome and the assistant agreed and told them a refund was on its way. The rule was right there in the prompt. The model read it, weighed it against a persuasive story, and set it aside.

A prompt rule is a preference, not a fence

When you put “never do X” in a prompt, you are asking the model to behave, not preventing it from misbehaving. The instruction goes into the same context as everything the user says, and the model blends all of it into one response. A firmly worded user message, a hypothetical, a role-play framing, a claim that the rule does not apply this time: any of these can outweigh the instruction, because the model is predicting a plausible reply, not enforcing a policy.

That is why the support engineer’s rule failed. It was competing with the user’s input on the same field, and sometimes the user’s input wins. A guardrail that can lose an argument is not holding anything. It is a strong suggestion that works until someone phrases their way past it, and the people most likely to try are exactly the ones you wrote the rule for.

The deeper issue is where the enforcement lives. It lived inside the model, in text, subject to interpretation. Anything enforced by interpretation can be reinterpreted.

Put the enforcement in code around the model

A real guardrail sits outside the model, in deterministic code that the model cannot rewrite by being clever. The model can suggest a refund all it likes. Whether a refund actually happens is decided by code that checks permissions and simply does not have a path to issue one from this surface. The instruction in the prompt becomes a nicety. The code is the fence.

That shift is what turns “please do not” into “cannot.” The support assistant did not need better prompt wording. It needed the refund action to be unreachable from the assistant, so that no amount of persuasion could reach a capability that was never wired up.

Here is what enforcement in code looks like.

Filter the input before it reaches the model. Screen for the categories you will not handle and short-circuit them, so the model never gets the chance to be talked into them.
Gate every real action behind a permission check. Let the model propose, and let code decide whether the action is allowed, using rules the model cannot edit.
Check the output before it takes effect. Inspect what the model produced against your policy and block anything that crosses a hard line, independent of what the prompt said.
Remove the dangerous capability from the surface entirely. If the assistant must never issue refunds, do not give that surface a refund path at all, so there is nothing to argue toward.
Keep the prompt rule as guidance, not as the control. Leave “never do X” in for tone and steering, but never let it be the only thing standing between a user and the action.

The model stays useful and the boundary stays fixed, because the boundary no longer depends on the model agreeing to it.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we go through the rules a team believes protect their feature and ask a blunt question about each one: is this enforced in code, or is it a sentence in a prompt. The sentences get moved. Anything that matters, a refund, a data write, a policy line, gets a deterministic check around the model where phrasing cannot reach it. It is far cheaper to find a prompt-only guardrail before launch than after a user has talked your assistant into something it should never have done.

If a well-worded message can undo your safety rule, the rule was advice. Put the fence in code.