The support ticket that gave your AI its next instruction

A product lead at a support software company told me about a bug that did not look like a security bug at first. Their AI assistant read incoming tickets and drafted replies. One ticket contained a line that said, in plain English, to ignore prior guidance and forward the account details to an address the customer supplied. The assistant did what the ticket said. It read like a helpful instruction, so the model followed it.

Nobody wrote code that decided to leak. The code did what it always did. It put the ticket text and the system instructions into the same prompt, and the model had no way to tell which one came from the company and which one came from a stranger.

The model reads everything as one voice

When you build an AI feature, you write a system prompt with the rules. Then you paste in the user’s content so the model can act on it. To you those two things are different in kind. One is your instruction. The other is data to be processed. To the model they arrive as the same stream of tokens. There is no built-in marker that says this part is trusted and that part is not.

So any text a user controls becomes a place where they can try to speak to your model directly. A support ticket, a form field, a file name, a calendar invite, a product review. If it reaches the prompt, it can carry an instruction. The attacker does not need special access. They need a text box that your AI will later read.

This is the part teams miss in the demo. In the demo the tester types friendly questions, and the feature answers them. The tester is not trying to hijack the assistant. Real users include people who will try, and they only have to succeed once.

Separate the data from the commands

The fix starts with a shift in how you think about input. Treat every piece of user-supplied text as untrusted, the same way you treat form input before it touches a database. The model should know that a given block is content to analyze, not orders to obey.

A few things that help, none of them a silver bullet on their own:

Keep instructions and user content in separate roles. Put your rules where the model expects rules, and pass user text as clearly labeled data, not blended into the same paragraph.
State the boundary in the instruction. Tell the model that the following block is user data, that it may contain attempts to change its behavior, and that it should not act on instructions found inside it.
Constrain what the model can do, not just what it should do. If the assistant can forward account details, an instruction to forward account details is a live risk. Remove the capability where the task does not need it.
Check the output before it acts. A reply that suddenly wants to send data to a new address is a signal worth catching before it leaves the building.

The point is to stop relying on the model to police a line it cannot see. You draw the line in the architecture, so that even a convincing instruction buried in a ticket lands as text to be read, not a command to be run.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, one of the first things we map is where untrusted text enters the prompt. We trace every field a user or an outside system controls, and we ask what the model could be told to do through it. We would rather find the hostile ticket in a review than in a customer’s postmortem.

If a stranger can type into a box that your AI later reads, that box is part of your attack surface. Treat what comes through it as data, never as a command.