One benchmark gave us a quiet failure. The model wrote a tool call, and our gateway handed it back as chat text.

The setup is small. The ToolACE prompt (from the ToolACE paper) lists tools by their original names, like SleepOptimizer.configureRoom, and asks the model to write calls as text: [SleepOptimizer.configureRoom(...)]. Many tool APIs reject a dot in a function name, so we offered the tool as SleepOptimizer_configureRoom.

The model then did something reasonable. It wrote the call as text, using the original name from the prompt. Our recovery step turns a written call into a real tool call, and it did not recover the dotted name, so the call text went to the user as the answer.

The fix maps a written name to the one offered tool, and only where a dot or a space stands in for an underscore. A hyphen or a slash gets no such pass, because a-b(x=1) and a/b(x=1) read as arithmetic, and turning arithmetic into a tool call is a worse bug than the one we started with.

The parser had more traps than I expected. Review turned up four scanner traps, listed here.

  • a triple quoted string that the rewrite could change
  • a # comment that threw the scanner off by one
  • a lone carriage return, which ends a Python comment
  • a trailing comma that blocked recovery

Each one got a test, and each test was checked by breaking the code and watching it go red.

The numbers, from one internal replay of a ToolACE sample on our Base lane (our own run, which also included other tasks, so treat it as a lab number): of 1,998 answers, 337 were text calls. The old gateway recovered none of them. The new one recovered 328, and no answer outside ToolACE changed. The other 9 stay as text, because the written name contains a slash, which is also division, so recovery refuses them.

What I would take from it: a model that follows your prompt and a gateway that follows your schema can both be right and still lose the call between them. If you map names, make the map as narrow as you can defend, then try to break it with inputs that look like math.

Has anyone else seen a prompt and a tool schema disagree about a name?