The model that called a tool you never gave it

An integrations engineer described a crash that made no sense to him at first. His agent had a set of defined tools, and the code handled each one. Then it fell over on a tool call for a function nobody had written. The model had invented a plausible-sounding function name, something close to a real tool but not one that existed, and called it. His dispatch code looked up the name, found nothing, and threw. He had built branches for every tool he defined and no branch for a tool he did not.

The model can name any tool, not only yours

The assumption baked into his code was reasonable and wrong. He had defined a list of tools, so he wrote handling for that list, one branch per tool, and treated the set of possible calls as closed. But the model does not treat the tool set as closed. It generates a function name as text, and text can be anything. Most of the time it produces one of your real names. Sometimes it produces a name that sounds like it should exist, blending two of your tools or inventing one the task seemed to call for.

From the model’s side this is not malfunction, it is just generation. It predicted a token sequence that looked like a reasonable tool call. Nothing in the process guarantees that sequence matches a name you registered. The guarantee has to come from your code, and his code assumed it instead of checking it.

An unknown tool is input you did not plan for

The crash was not really about the model hallucinating a name. It was about the integration having no answer for the case. Every real system that takes structured input handles the input that does not fit the expected shape. A malformed request gets a clean rejection, not a crash. The engineer had built that discipline everywhere except at the tool-dispatch layer, where he had quietly assumed the input would always be one of the values he expected.

Once he saw it that way, the fix was obvious. The set of valid tool names is a known, small list. A call naming anything outside that list is invalid by definition, and invalid input gets rejected, not executed and not crashed on. The model inventing a name is a case to handle, the same as any other bad input, rather than an event that should be able to take the system down.

Validate the name against the registry and reject the rest

Treat the model’s chosen tool name as a claim to verify before you act on it.

Check the name against your registered set. Before dispatching, confirm the name is one you actually defined. If it is not, do not try to run it.
Reject unknown calls cleanly. An unrecognized tool name gets a defined response, not an exception. Return a clear error and keep the system standing.
Feed the error back to the model. Tell it the tool does not exist and list the tools that do. Often it will recover and call a real one on the next turn.
Log the invented names. A model repeatedly reaching for a tool you never built is a signal. Sometimes it means your real tools are named confusingly. Sometimes it means the model wants a capability worth adding.

The mindset is to never assume the model only calls tools you defined. Validate the name, and let an unknown call be a handled case rather than a surprise.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500, we check what happens at the tool-dispatch layer when the model names a tool that does not exist, because in a real system it eventually will. We make the dispatcher validate every tool name against the registered set and reject anything outside it cleanly, so a hallucinated function name becomes a graceful error and a useful log line instead of a crash. It is a small guard that turns a whole class of surprise failures into a non-event.

Never assume the model only calls the tools you defined. Check the name, and reject the ones you did not.