Most of what makes an AI assistant useful isn’t prose — it’s calling tools: booking the meeting, adding the cart item, pulling the shipping estimate. When these products fail, it’s rarely bad writing. It’s the wrong function, a mangled date, or three calls when one would do.
AceBench measures exactly that: ~2,000 hand-annotated tasks drawn from ~4,500 synthetic APIs across eight domains (finance, health, travel, tech, entertainment, and more), published January 2025 and later accepted at EMNLP.
Three things make it worth a practitioner’s attention:
Grading is mechanical and brutal. Right function, right call count, right arguments — or it’s wrong. No partial credit, no AI judge. One bad field voids the form. Harsher than real life, but reproducible.
It comes in tiers. Normal (clear requests), Special (vague), Agent (multi-turn). We ran Normal, English only — this says nothing about ambiguity or agent loops.
It’s designed to be taken apart. Tasks isolate specific failure modes: value types, near-duplicate functions, mid-conversation drops. A leaderboard number says which model wins overall. A benchmark that comes apart says whether a cheap model is good enough for your work — usually the real question.
What we ran
Five models, 772 tasks each, 3,860 attempts, English Normal split:
gemini-3.6-flash,gemini-3.1-flash-lite— Google’s hosted modelsgemma-4-12B-it,gemma-4-E4B-it,gemma-4-E2B-it— open-weight, self-served (big/small/very small)
Same harness and tasks for everyone. Tool calls return a bare acknowledgement — no feedback, one shot per call.
Key point: the open model on hardware we control tied Google’s hosted model, using about an eighth of the words.
The scoreboard
The top two are a real tie: 45 tasks the hosted model got and the open one missed, 35 the other way. That’s the signature of two equally capable models, not a better and a worse one. Run it again and the order could flip. Every other gap in the table is real — but the tie at the top is between a hosted frontier model and a 12B open model on a single GPU.
What it costs
Scoring the same isn’t interesting. Scoring the same this cheaply is.
ModelWords per correct answer*:
*tokens (~¾ word each), total output
The hosted model generates ~8x more text per correct answer than the 12B model it’s tied with — almost all invisible “thinking” (655,000 tokens across the run). The other four models did none of that: read, call, stop.
And the thinking isn’t buying anything. When the hosted model got a task wrong, it generated more than twice as much text as when it got one right. Extra effort here is a sign of being stuck, not a way out. On a benchmark where the winning move is two steps, there’s not much to think about.
Generated tokens are the expensive half of any pricing page, and what determines latency. So: same accuracy, a fraction of the tokens, and a deployment you own instead of rent.
The catch
Cheap on tokens ≠ cheap in practice:
The 12B model that ties Google’s best was 4x slower per answer than the API it ties, despite generating far less text. The smaller E4B, on the same GPU, hit 5.6s at ~91% of the top score. Note: these numbers are default vLLM, full BF16, on a single L40S — plenty of room to push both further with better hardware.
The smallest model is better than its score
gemma-4-E2B-it came last at 67%. Look at how it failed and half the gap disappears.
Its top mistake wasn’t picking the wrong tool — it was making the right call plus extra, already-completed ones from earlier in the conversation:
Called: ask about gift etiquette → send the gift → schedule the meeting Wanted: schedule the meeting
It knew the current turn. It just also re-did two things already done. Graded: zero.
72 of its 255 failures end with exactly the right call, buried under repeated history. Score only the current turn and it jumps from 67% to 76% — close to Google’s smaller model. That’s a prompt fix, not a bigger model — and it barely moves the other four (0-3 failures each of this kind).
Where bigger models still earn their keep
Three places:
Nested arguments. A tool wanting {"journey": {"from": "Shanghai", "to": "Hangzhou", "times": {...}}} trips up everyone — best models barely clear 60%, versus 85-96% for flatter arguments.
Several calls at once. Same tool, three times, different details: the hosted model pulls ahead ~5 points. Coordinating calls is harder than making one.
Long conversations — the sharpest split. Four of five models improve turn over turn as context narrows the options. The smallest model collapses:
That last column is what matters in production: a conversation only works if every turn lands. 87% per-turn becomes 74% overall.
Two models beat one
Since every model saw the same tasks, we can check what a pair covers:
Pairing the hosted model with the open one adds 4.5 points; pairing it with Google’s other model adds just 2.1. The two hosted models fail on the same tasks — the open one fails on different ones. If you can check an answer and retry, the self-served model is the better (and cheaper) second opinion.
So what do you do with this
If the task is “here’s what I want, here are the tools, go” — use a small self-hosted model. There’s no ambiguity for extra reasoning to resolve. A 12B model matches the frontier here; even 4B gets within ~90%.
If the work needs complex structures, coordinated multi-calls, or long conversations, the gap reopens — fastest for the smallest models.
Check what a scoreboard actually measures before trusting it: one category here was scoring a missing input, not the model, so we dropped it. Another score was understated nine points over a technicality about which calls “count” for a turn. Neither shows up in a leaderboard number — and both change what you’d buy.