The agent that called the wrong tool for the right reason
The model was helpful. It had a tool that could delete records and a request that sounded like cleanup, so it deleted. The problem was that the tool was in reach at all.
The agent that called the wrong tool for the right reason
A platform engineer at a data company told me about an agent that did exactly what it was built to do and still caused an incident. The agent could read records, update records, and delete records, because during the build it was easier to give it the full set of tools than to figure out which ones each task needed. A user asked it to clean up some stale entries. The agent read that as permission to delete, and it deleted more than anyone intended. It was not tricked. It was helpful, and it had a dangerous tool within reach.
This is the confused-deputy shape. The agent holds more authority than the task requires, and something, a user request or an injected instruction, points that authority at the wrong target. The agent is not malicious. It is a deputy acting on power it should never have been handed.
Capability is the risk, not intent
Teams spend a lot of energy trying to make the model behave correctly. That is worth doing. It is also the wrong place to put your last line of defense, because a model that means well can still take a destructive action if the action is available to it.
The question that matters is not what the agent intends. It is what the agent can do. If an agent can delete production records, then every path that leads to a delete, a misread request, a hidden instruction in a document, a genuine bug, ends somewhere expensive. If the agent simply does not hold that tool, all of those paths dead-end harmlessly.
Most agents are over-equipped for the same reason the data-company agent was. Wiring up the full tool set is faster than scoping each capability. The blast radius of that shortcut is invisible until the day something aims the agent at the wrong thing.
Give the agent only what the task needs
The fix is to design the tool set as carefully as you would design a person’s access. Start from the task and work toward the smallest set of capabilities that gets it done.
In practice:
- Scope tools to the job. An agent that summarizes tickets does not need a delete tool. Do not hand it one because it was convenient.
- Split read from write. Reading is cheap to get wrong. Writing and deleting are not. Keep destructive actions behind their own gate, not bundled into a general toolkit.
- Require confirmation for the sharp tools. For actions that are hard to undo, put a human approval in the path rather than letting the agent proceed on its own read of the request.
- Log every tool call with its inputs. When something does go wrong, you want to see exactly which tool ran, with what arguments, on whose behalf.
The teams that avoid these incidents treat the tool list as a security decision. Every tool an agent holds is a thing that can be misdirected. Fewer sharp tools within reach means fewer ways for a helpful agent to do harm.
How we approach it at Density Labs
In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we review the full tool set an agent can call and ask, for each one, whether the task actually requires it. We flag the destructive tools that are within reach for no good reason, and we design the confirmations and scopes before the agent goes live. Trimming a tool list is a lot cheaper than restoring records after an agent misread a request.
An agent will use the tools it has. The question you answer at design time is which sharp objects to leave lying around.