Security review for an AI feature: what production demands

Most AI security thinking stops at the model. Can it be tricked, can it leak, can someone jailbreak it. Those matter, but they are one component. Production security is a system, and the model is the part that gets the applause. The parts that decide whether an incident becomes a headline are the process around it, the trail it leaves, and who is accountable when an alarm fires. Skip those and a clever model is a liability with good PR.

The number that should focus a security review: breaches that involve shadow or ungoverned AI run about $670,000 higher than a standard breach. The extra cost is not the model, it is the missing system around it, the logging you did not keep and the response you did not design.

Detection is only the first part of the system

The co-founder of a physical-security company that screens for weapons walked me through how real-world security actually works, and it reframes AI security completely. Their technology detects metallic threats as people enter a building. But he was clear that detection is only part of the system. Outcomes depend on disciplined processes, well-designed secondary screening, and officers performing consistently over time. A perfect detector with a sloppy process around it keeps no one safe. He talked about necessary friction, the deliberate step that slows things down enough to stay safe without drowning staff in false alarms, and about the constant trade-off between accuracy, speed, and convenience that every deployment has to settle on purpose.

The part most relevant to AI: he described using their own AI to increase accountability. It confirms an officer is actually present, that alarms get resolved with a documented reason, and it provides audit trails and system-health monitoring. Read that again as an AI feature requirement. The security value was not just detection. It was the documented reason, the audit trail, the proof that a human did their part. That is what production demands and demos skip.

The system is what a review has to cover

His point about phones and smart devices as new contraband lands here too. The threats keep changing, so a one-time review of a fixed threat is worthless. Security is a posture you maintain, not a checkbox you tick before launch. An AI feature that passed review against last quarter’s risks is not secure, it is out of date.

So a real security review of an AI feature looks past the model at the whole system: how inputs and outputs are logged, how an incident is detected and resolved, who is accountable, and how you know the thing is still healthy in production.

What a production security review actually checks

The audit trail. Can you reconstruct what the model did, with what data, for any decision.
The response process. When something goes wrong, who is notified and what happens, designed before launch.
The necessary friction. Where a human check belongs, set deliberately, not everywhere and not nowhere.
Ongoing health. How you monitor the feature after launch, because the threats will not hold still.

How we approach it at Density Labs

In the AI Readiness Assessment, our $2,500 front door, we review the security system around the AI feature, not just the model in the middle of it. We check the audit trail, the incident response, where human friction is warranted, and how the feature is monitored once it is live. Most pilots have a strong demo and none of that system, which is precisely the gap that turns a manageable incident into the expensive kind.

The detector gets the demo. The audit trail gets you through the incident. Build both.

A model that cannot explain what it did is not a feature. It is an open question waiting for the worst possible time to be asked.