The guardrail that lived in the prompt and belonged in code

An engineering lead at an insurance company described the rule that kept him up at night after it broke. Their assistant was never supposed to quote a price. Pricing was regulated, it depended on underwriting, and a wrong number in writing was a real liability. The rule lived in the system prompt: “Do not quote premiums under any circumstances.” It worked for months. Then a user asked in a way the model found compelling, and the assistant gave a number. The rule had been holding by habit, not by enforcement.

A business-critical rule was resting on a sentence

The rule was correct and the consequence of breaking it was serious. Both of those things argue for the strongest possible enforcement. Instead it was enforced the weakest possible way, by a line of text the model was free to weigh against whatever the user said. For months the model happened to keep the rule. That stretch of good behavior read like safety, and it was really just the failure not having happened yet.

What makes prompt-only enforcement risky for a rule like this is that it can fail from two directions. A user can phrase around it, the way this one did. And the prompt itself drifts. Someone edits the system message to add a feature, reorders the instructions, trims it for token cost, and the pricing rule quietly loses the weight it had. Neither the user nor the editor did anything obviously wrong. The rule was never anchored, so it moved.

For a soft preference, a prompt line is fine. For a rule where one violation is a compliance event, resting it on wording that any edit or any clever message can disturb is building on sand.

Anchor the rule in deterministic code

The rule belonged in code around the model, where it holds regardless of phrasing or prompt drift. The model can generate whatever it generates. Before that output reaches a user, code scans it for a price and blocks or strips it if one appears. Now quoting a premium is not a matter of the model choosing to obey. It is structurally prevented, and no prompt edit or user message can reach past it.

The insurance lead did not need a better-worded prohibition. He needed the constraint to live somewhere the model could not override and an editor could not accidentally weaken. Code is that place.

Here is how to anchor a business-critical rule.

Identify the rules where one violation is unacceptable. Separate the hard constraints, pricing, legal claims, protected record types, from the soft preferences, so you know what needs real enforcement.
Enforce hard rules with a deterministic check. Put code on the output that detects and blocks the forbidden thing, so the rule holds no matter what the model produced.
Remove the capability where you can. If the assistant must never touch a record type, do not give that surface access to it, so there is nothing to enforce.
Keep prompt wording as a first layer, not the only one. Leave the instruction in to steer the model, but never let it be the last line of defense on a critical rule.
Test the rule against adversarial inputs. Actively try to make the model break it, so you learn where the enforcement leaks before a user does.

The prompt can still ask nicely. The code makes sure the answer does not depend on the model agreeing.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we go looking for the rules a team is enforcing with prompt text alone, and the dangerous ones are usually business rules with real consequences: pricing, eligibility, anything regulated. We move those into deterministic checks around the model and leave the prompt to do the steering it is good at. A rule that has held for months in a prompt is not proven safe. It is one well-worded message or one careless edit away from failing, and it is far cheaper to anchor it before that happens than to explain the violation after.

If breaking the rule is a serious problem, the rule cannot live only in a sentence the model is free to ignore.