100% valid tool calls is not 100% right ones: HN on constrained decoding in vLLM

Ubicloud's "Fixing Tool-Call Accuracy in vLLM" drew 172 points on Hacker News. Serving Kimi K2.6 on vLLM 0.28, the team "got the expected tool call schema only 92% of the time", or "2 out of every 25 attempted tool calls failed". Constrained decoding took Moonshot's K2VV ToolCall benchmark to 100%. Our read of the thread: the 100% is real, and it is a statement about shape. A schema check proves the call parses. It does not prove it was the right call, and the part of Ubicloud's work worth copying is the benchmark that caught the problem, not the flag that fixed it.

Why 92% was never good enough

bob1029 did the arithmetic that makes per-call accuracy matter: "With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%."

We ran the same sum for a few rates. It assumes each call fails independently, which real agents do not quite do, so treat it as the shape, not a forecast:

At 92%, a 25-step agent task finishes clean about one time in eight. That is why the fix mattered. It is also why the next question matters more: once every call parses, what fraction are correct? The benchmark's "100%" does not say. The article describes K2VV as checking "whether the generated tool calls are syntactically correct".

Syntax is not semantics

The thread kept returning to one line from NitpickLawyer: "Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file)."

Majromax explained why constraining can even hurt meaning: "forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless". They also gave the case for it with tool calls specifically: the hard choices, the tool name and its parameters, come first, so errors tend to sit in the same tokens that cause them.

hansvm gave the concrete failure. A model writing a JSON string that has already gone wrong, ending in an ellipsis, gets no way back: "With schema-constrained decoding, you must eventually terminate that broken string, and it'll typically terminate it right after the ellipses." The output validates. The content is garbage.

Constrain or retry

The practical disagreement was about what to do when a call is malformed. hansvm drew the line precisely: "Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences."

saurik pushed it further, days later: if a call needs several retries, maybe "the AI simply got locked into too difficult of a way to solve the problem", and retrying the whole response would take a different path. Constraint repairs the token. Retrying the response can repair the plan.

The two are not exclusive. Constrained decoding removes the class of failure where the call does not parse. A retry with error feedback is still the only thing that can catch a call that parses and is wrong, and only if something checks the result.

Fixed, or configured?

Aurornis argued the post overstated things: constrained decoding had been in vLLM for a while, so "really they just needed to configure it correctly". jesdo, one of the authors, replied that "we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration", and that the vLLM team later fixed it upstream.

The exchange is less interesting for who was right than for how it was settled: with a benchmark. Ubicloud found the 92% because they ran a published tool-call test against their own serving stack, and they shipped only after it read 100% "across all served model versions".

The takeaway

Three layers came out of the thread, and each catches a different failure:

1. Shape. Does the call parse against the schema? Constrained decoding settles this, cheaply.
2. Choice. Was it the right tool with the right arguments? Only an evaluation with expected calls can tell, and a schema-accuracy benchmark is not that.
3. Effect. Did the call do what the agent needed in the system it touched? Only checking the result, after the call runs, tells you.

The 100% headline covers layer one. For an agent that edits files or calls APIs, layers two and three are where tasks actually fail, and a well-formed call to the wrong place fails silently. If you serve or run open-weight models, copy Ubicloud's habit, not just their flag: keep a tool-call benchmark in CI, and add checks on what the calls did.

Caveats. One thread, read by us. The chain table assumes independent failures and is arithmetic, not a measurement. We did not re-run the K2VV benchmark.