The jailbreak your users found before your red team did

A product manager at a consumer app told me about the morning she learned her new AI assistant had been talking out of school. The feature had shipped a few days earlier. It answered questions inside the app, friendly and quick. Then a screenshot turned up in a public forum. A user had coaxed it into describing internal policy the company never meant to expose. Nobody on her team had tried that prompt. A stranger did, for fun, within hours of launch.

Your feature goes on trial the moment it ships

The plan had been to run an adversarial pass later. Launch now, harden soon. That order is backwards. A public AI feature meets thousands of curious people the instant it goes live, and some fraction of them will push on it in ways the team never would. Most are not malicious. They are bored, or clever, or they just want to see what the thing does when you ask it something strange. That is your red team. It is external, unpaid, and it posts the results.

The manager’s instinct had been to treat safety testing like a polish step, something you get to once the feature is stable. But the feature was never going to be more exposed than in its first week, when the novelty pulled in the widest range of people trying the widest range of inputs. The window where testing mattered most had already closed by the time she scheduled it.

What the ordinary user actually did

There was nothing sophisticated about the prompt that broke it. The user did not write code or chain a clever exploit. They asked the assistant to role-play, told it the old rules no longer applied, and it went along. The model was helpful to a fault. It treated the user’s instruction to ignore its guidelines as just another request to satisfy.

This is the part teams underestimate. You do not need a security researcher to find the soft spot. You need one persistent person with an afternoon. The gap between “our internal testers could not break it” and “it is safe” is enormous, because your internal testers share your assumptions. They ask the questions you expect. Real users do not know what you expect, so they wander straight into the cases you never wrote down.

Move the testing before the launch, then keep watching

The fix has two halves, and skipping either one leaves you exposed.

Run adversarial testing before launch, not after. Sit a few people down whose only job is to make the feature misbehave. Give them the goal, not a script. If they cannot break it in a focused session, you have raised the bar. If they can, you found it before a forum did.
Write the off-limits behavior down as test cases. Turn every “it should never do this” into a concrete prompt with an expected refusal, and rerun them on every change.
Assume you missed some, and monitor for it. No pre-launch pass catches everything. Log the inputs and outputs, watch for the model producing things it should refuse, and set an alert so you learn from your own traffic instead of from a screenshot.
Have a fast path to tighten a guardrail. When you find a new way in, you want to close it in hours, not in the next release. Decide that before you need it.

The point is to shift discovery earlier and to accept that discovery never fully ends. Users will keep finding edges. Your job is to find most of them first and to see the rest quickly.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500, we treat “how does this fail in front of a real user” as a design question we answer before you build, not a report we hand you after. We probe the feature the way an outside user would, write the refusals you care about as cases the system has to pass, and set up monitoring so the prompts you missed surface as data instead of as a public incident. It is far cheaper to find the hole in week one than to explain it later.

Ship the feature assuming a stranger will try to break it on day one, because one will.