Attack your own AI feature before a stranger does

A head of engineering at a fintech-adjacent company admitted something to me that more teams should. Before launch, they had tested their AI feature thoroughly, in the sense that they confirmed it did the job. They asked it the questions users would ask and checked that the answers were good. What they had never done was sit down and try to break it. The first person to do that seriously was a customer, and the customer found a way to make it reveal information it should not have. The feature worked. It had just never been attacked.

Testing that a feature works and testing that it cannot be made to misbehave are two different jobs. The first uses the feature as intended. The second uses it as an adversary would. Most teams do the first and call it done, because the second one is uncomfortable and it is nobody’s default instinct to attack their own work.

Working and unbreakable are different tests

When you test that a feature works, you are cooperative. You give it reasonable inputs and you check for reasonable outputs. That tells you the happy path holds. It tells you almost nothing about what happens when someone hostile shows up.

An adversarial pass is the opposite posture. You are trying to make the feature do something it should not. You feed it inputs designed to confuse it. You plant instructions in documents it reads. You try to get it to reveal its prompt, cross a data boundary, call a tool it should not, or produce content it should refuse. You probe the seams, not the center.

The gap between these two matters because attackers only run the second test. They do not care that your feature answers questions well. They care whether it can be pushed off the rails, and if you never checked, they will be the ones who find out.

Build the adversarial pass in before launch

Red-teaming does not require a dedicated security team, though one helps. It requires deciding, before launch, to spend real effort trying to break the thing.

A starting set of attacks worth running against any AI feature:

Prompt injection through every input. Plant instructions in each field, upload, and data source the model reads, and see what follows them.
Boundary probing. Try to get the feature to reveal data from another user, another tenant, or a role above the caller’s.
Capability abuse. Try to get the model to call a tool it should not, or take an action beyond the task.
Prompt and instruction leakage. Ask it to reveal its own instructions, and assume whatever comes out is now public.
Content it should refuse. Push it toward outputs your product should never produce.

Write down what you tried and what happened. The findings become a checklist you run again after every meaningful change, because a prompt tweak or a new tool can reopen a door you closed.

The point is to move the discovery earlier. Someone is going to attack the feature. The only choice you have is whether it is you, before launch, with time to fix what you find, or a stranger, after launch, on their timeline.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, an adversarial pass is part of the work, not an add-on. We try to inject, to cross boundaries, to abuse capabilities, and to pull out things the feature should protect. We would rather hand you a list of the ways your feature can be broken than let a customer compile it for you.

A feature that works has passed the easy test. Try to break it yourself, on purpose, before the first stranger does it for you.