What we ran
Three models, two tool interfaces, 340 CRM tasks each — 2,040 agent rollouts in total.
The benchmark is CRMArena, run through the Harbor/dockworker harness. Every task asks a question about a read-only Salesforce org (”in May 2021, which state had the quickest case closures?”, “which knowledge article does this quote violate?”) and grades the submitted answer by exact match. Seventeen task categories, split evenly across a B2B and a B2C org.
The interesting design choice is that the same 340 questions are posed twice, behind two different tool surfaces:
Identical questions, identical ground truth — we verified this holds on every task where both suites produced a graded answer. So the two suites isolate one variable: does the model do better writing SQL, or picking the right pre-built tool and filling in its arguments?
The three models: gemini-3.6-flash and gemini-3.1-flash-lite (hosted), and gemma-4-E4B-it (open weights, served locally on vLLM). One is a small local model; two are hosted. Every trial used the same noshell agent scaffold, whose only way to answer is to call a terminal submit_answer tool.
The scoreboard
Read it and you’d conclude the ordering is obvious and the small model isn’t close: 39.7% against 67.6% is a 28-point gap. Every pairwise difference here is statistically significant (p ≤ 0.009).
Then you look at how the failures happen, and the picture inverts.
Half of the small model’s “failures” contain the right answer
Not every zero is a wrong answer. A trial scores zero if the answer file was never written at all — and the scaffold only writes it when the model calls submit_answer. Classifying all 2,040 trials by how they failed:
More than half of gemma’s SQL trials never submitted anything. And they didn’t crash or time out — in 171 of them the model finished its work, wrote the answer out in prose, and simply never called the tool. Something like:
The
case_metricscall returned an average closure time of 4.2 days for CA, the lowest of any state. The answer is CA.
Graded: zero.
So we tested the obvious question — were those prose answers right? Ground truth isn’t recorded for ungraded trials, but because the same 340 tasks appear in both suites, we could recover the expected answer for every one of them and grep the final message for it.
Of gemma’s 250 prose non-submissions, 44–52% contained the correct answer. The range is the strict and loose reading of the same check: the loose count is any trial whose prose contains the expected value; the strict count additionally requires that the model wasn’t hedging across a list of candidates (≤1 other record ID mentioned). Both bound the same conclusion.
That reframes the scoreboard as a lower bound. Fixing one scaffold behaviour — get the model to call the tool — moves gemma to:
A model that looked like it scored 26% on SQL was doing work worth about 50%. Its measured number was roughly half its actual competence, and every point of that gap is a formatting bug.
The same correction barely moves the hosted models — gemini-3.6-flash has exactly zero prose non-submissions, and gemini-3.1-flash-lite has one. They always call the tool. What we were measuring, for a third of the benchmark, was instruction-following on the harness contract, not CRM reasoning.
There’s a cleaner way to see it. Restrict to trials that submitted anything, and the reasoning quality behind the scoreboard separates from the plumbing:
On answers it actually submits, the small local model is within two points of the hosted flash-lite model — a difference well inside the noise at this sample size. The 11-point headline gap between them is almost entirely tool-calling discipline.
What it costs
This is where the small model stops being a curiosity. Token totals are the whole run; the per-win column divides by correct answers, which is the number that matters if you’re paying for throughput.
Gemma on the tool API buys a correct answer for 45.4k tokens. gemini-3.6-flash needs 196k for the same thing — 4.3× more. On SQL it needs 598k, or 13× gemma’s best configuration.
The driver is visible in the reasoning-token column. gemini-3.6-flash spent 1.36M reasoning tokens on the API suite and 2.59M on SQL. The other two models spent none. That’s what the extra accuracy is bought with: 3.6-flash’s median trial emits 3,978 output tokens on the API suite against gemma’s 501 — an 8× difference in generated text per attempt.
And time to completion — median wall time of the agent execution phase alone, excluding container build and verification:
Here the small model does not win, and it’s worth being precise about why. gemma is 12.6s per attempt against 3.6-flash’s 30.0s — faster per attempt — but because it converts fewer attempts into correct answers, it lands at 73.8s per win versus 51.8s. And this column mixes model speed with serving infrastructure: a local vLLM instance against Google’s production endpoints. It is not a property of the models, and it’s the one metric here we’d throw out of a purchasing decision. Tokens are the honest efficiency measure; wall-clock is an artifact of where each model happened to be running.
One more pattern worth noting: every model spends more tokens on the trials it gets wrong — 1.5× to 2.1× its passing median. Failure is not cheap. Effort is a symptom of being lost, not a route out of it, which means the token cost of a wrong answer exceeds the token cost of a right one across the board.
Where each model actually fails
The three models have almost nothing in common in their failure profiles.
gemma-4-E4B-it — a formatting problem wearing a capability problem’s clothes. 250 of its failures are prose-instead-of-tool-call; roughly half contain the right answer. Its second tendency is over-caution: 50 trials answered None (”nothing matches”) when a real record existed. It abstains too readily and it won’t call the terminal tool. Both are addressable without touching the model.
gemini-3.1-flash-lite — quietly broken generations. 34 API trials (10%) ended with an empty completion: a single EOS token, no text and no tool call. Not prose, not a wrong answer — nothing at all. That signature points at serving or sampling rather than the prompt, and it’s the cheapest 10% anyone in this comparison could recover. Beyond that its losses are ordinary wrong answers (21–23%), the highest wrong-answer rate of the three.
gemini-3.6-flash — runs out of budget, and won’t say “none.” It never fails to submit when it finishes, but it frequently doesn’t finish: 60 SQL trials (18%) and 17 API trials (5%) hit the 25-step cap mid-work. Its failing trials burn a median 9,747 output tokens against 6,452 when passing — it iterates on queries that never land. Its other weakness is the mirror image of gemma’s: 63 trials invented a value where None was correct. The strongest model is the one most likely to manufacture an answer rather than concede there isn’t one.
That last contrast is the most useful thing here for anyone building on these models. The two abstention errors point in opposite directions — gemma says “none” when an answer exists, 3.6-flash asserts an answer when none does — so there is no single prompt that fixes both. It’s a calibration problem per model, not a benchmark-wide one.
The interface matters, and not the way you’d guess
Because the same questions appear behind both tool surfaces, we can ask whether SQL or semantic tools suit each model better as a paired comparison — the same task, two interfaces — using an exact McNemar test on the tasks where the two disagree.
For gemini-3.6-flash, the only model whose two runs had matched step budgets, the tool API wins clearly: +8.5 points, p < 0.001, with 47 tasks solved only through the API against 18 solved only through SQL.
But the aggregate hides something better. Across the models, between 65 and 133 of the 340 tasks flip outcome between the two interfaces. For gemini-3.1-flash-lite the headline rates are nearly identical (49.1% vs 50.9%) while 94 tasks flip — 44 solved only via SQL, 50 only via the API, with just 123 solved by both. The interface doesn’t change how many tasks it gets right; it changes which third of the benchmark it gets right. A model that looks interface-indifferent on the scoreboard is nothing of the kind, and running both interfaces and taking either success would score far above either alone.
Category detail shows where this bites. gemini-3.6-flash gets sales-amount-understanding right 60% of the time through semantic tools and 5% through SQL — but that collapse is not an inability to write the query. 18 of its 20 SQL attempts hit the step cap mid-work. Where a semantic tool answers “total order amount by owner in this window” in one call, SQL requires discovering the schema, joining line items to orders, and windowing on the right date column — and it ran out of budget getting there. lead-routing, by contrast, it solves 100% either way.
Not all 17 categories are the same benchmark
Treating CRMArena as one number hides the most actionable result in the run. Here is every category, worst-first, across all six runs (36 = gemini-3.6-flash, 31 = gemini-3.1-flash-lite, gm = gemma-4-E4B-it):
Group the 17 categories by what they actually ask for, and a pattern appears that the headline rates completely obscure:
On policy judgement, model scale buys almost nothing. Reading a chat transcript and deciding whether an agent breached policy, or whether a lead is qualified: gemini-3.6-flash manages 37.5%, gemma manages 32.5%. The small model is at 87% of the best model’s performance. Compare that to aggregate metrics, where the same small model reaches only 57% of the frontier score and the spread across runs hits 85 points on monthly-trend-analysis.
The reason is visible in the failure modes. On policy-violation-identification, half of gemini-3.6-flash’s API failures are missed_abstention — 8 trials asserting a violation where the correct answer was “none” — against 8 passes. gemma’s failures on the same category split 4 missed_abstention and 4 false_abstention. This category isn’t primarily testing reasoning; it’s testing whether a model will decline to answer. That’s a calibration property, and calibration doesn’t scale with capability the way multi-step numeric work does.
So the practical read is: the categories where a small model is nearly as good are the judgement ones, and the categories where it falls off a cliff are the multi-step aggregations. If your workload is “read this conversation and classify it,” a 4B local model is a serious candidate. If it’s “compute this metric across three joins and a date window,” it is not.
Two more things the table says:
invalid-config is the one genuine capability cliff. gemini-3.6-flash gets 50% on both interfaces; every other run scores 5–10%. A 45-point gap that survives both tool surfaces is the clearest evidence in this run of something the smaller models simply cannot do — matching a quote’s configuration against the knowledge article it violates. Notably it’s not a formatting artifact: gemma’s failures here are 8 wrong answers alongside 7 prose non-submissions, so even the recovered ceiling stays low.
quote-approval defeats everything. 8% pooled, topping out at 20%, and gemini-3.6-flash scores 5% and 0%. No model, no interface, no step budget helps. When the strongest model in a comparison scores 5% on a category, that’s usually a signal about the task or the grader rather than the models — and it’s the first thing we’d re-examine before drawing conclusions about the remaining headroom.
And where does the small model actually beat a hosted model head-to-head? On the tool API, gemma matches or beats gemini-3.1-flash-lite in 5 of 17 categories: policy-violation-identification (35% vs 15%, +20 points), wrong-stage-rectification (40% vs 35%), invalid-config (10% vs 5%), and ties on case-routing and lead-routing. Four of those five are judgement or routing tasks, not aggregations — the same pattern again.
What we’d take away
The headline number on an agent benchmark is a joint measurement of the model and the scaffold around it, and for small models the scaffold dominates. gemma-4-E4B-it looked like a 26% model on SQL. It was doing work worth about 50%, and the missing half was one unmade tool call. Anyone comparing a small open model against a hosted frontier model on a leaderboard number, without looking at the failure modes underneath it, is partly measuring which model was better at following the harness’s calling convention.
Small models win on the axis that gets left off the leaderboard. At 45.4k tokens per correct answer against 196k, gemma is 4.3× cheaper per unit of useful output than the model that beats it by 28 points — and it gets there with zero reasoning tokens against 1.36M. If your workload tolerates 50% task accuracy, or you can put a verifier behind it and retry, the small model is the better economic choice by a wide margin.
But the honest version of “small models can win” is narrower than the slogan. gemma did not beat gemini-3.6-flash in a single one of the 17 categories, on either interface. What it did was reach parity with a hosted flash-lite model on reasoning quality — 56.0% against 57.9% on submitted answers — at a fraction of the token cost, while running locally. That’s the claim the data supports: not that small models beat frontier models, but that the gap to the tier below frontier is mostly plumbing, and the plumbing is cheap to fix.
The category breakdown sharpens that into a selection rule. The gap is not uniform across work: on policy judgement the small model is at 87% of the frontier model, on multi-step aggregation it is at 57%, and on one category (invalid-config) it is nowhere near. “Can a small model do this?” has no general answer, but it has a reliable one per task family — and the families where the answer is yes are the judgement-shaped ones, which is the opposite of where most people assume a small model will struggle.
The fixes are unglamorous and they’re all in the harness: make the terminal tool call unmissable for models that write prose, investigate the empty completions, raise the step cap for models that iterate, and calibrate abstention per model rather than globally. None of them require a bigger model.