AI has moved from the playground to production, and the bills moved with it. Teams now spend thousands of dollars sending every query to a massive frontier model, including the queries that are simple.

Most developers know this is wasteful. You want frontier-level reasoning for the complex edge cases. For the repetitive tasks that make up most of your traffic, you want the speed and cost of a specialized small language model (SLM).

The hard part is knowing which is which. Deciding when a smaller model is good enough takes constant benchmarking, testing and custom routing logic. That is a full-time job, and it isn’t the product you set out to build.

Taskrouter does that job for you. It isn’t another API aggregator. It’s a router that learns your specific tasks and optimizes them for quality, speed and cost.

How Taskrouter works: from frontier to specialized

Integration is one API key. You point your application at Taskrouter the same way you would at OpenRouter or OpenAI.

From there, four things happen:

1. Start strong. Your initial workloads go to top-tier frontier models. That sets a quality baseline and captures gold-standard outputs for each task.

2. Test continuously. Behind the scenes, Taskrouter runs those same tasks against a fleet of smaller, specialized and open-source models.

3. Hand off when the data says so. Once a cheaper, faster SLM is shown to match or beat the frontier model on your specific task, Taskrouter routes that traffic to it automatically.

4. Keep the safety net. Automatic failover to frontier models stays in place for tasks that require deep reasoning.

The result is lower cost and lower latency at the same quality. Our first Fortune 500 customer to deploy Taskrouter in production saved millions of dollars, over 90% of their inference costs, while seeing their latency drop by 200ms on average and accuracy improving 5 percentage points.

The elephant in the room: privacy, IP and your data

When enterprise developers hear “learns from your data,” alarm bells ring. You picture your proprietary data being fed into a massive public model. That is a reasonable fear, and we built Taskrouter around it.

We come from enterprise. Taskrouter was built by the team that built my first company. We ran backup for Fortune 500 firms and had to pass rigorous security audits. In our current form we have been building SLMs for enterprise customers for over a year, and have active Fortune 500 customers on those SLMs. We know what it takes to pass a corporate infosec review.

Here is the Taskrouter privacy guarantee:

• It’s your IP. We don’t train public models on your data. Period. We only use it to route to the right models, and to fine tune models that are just yours, specialized to your unique workflows. You own those models.

• Optimization is isolated. The routing intelligence and the specialized models we spin up for your workloads are entirely isolated. They learn for you, not for us, and certainly not for your competitors.

• You control the data. We act as a strict pass-through with enterprise-grade data retention policies. When we use your data to benchmark a smaller model, it stays inside a secure, tenant-isolated environment.

Measured models, clearer choices

Routing decisions shouldn’t be a black box, so we show our work.

See the comparisons. Our Task Explorer page shows hundreds of comparisons of various models on common enterprise workloads. You can check how each model performs before you route a single request.

Keep control. Taskrouter can handle routing automatically, but you keep ultimate control over which models are used and where your data goes.

Prove the ROI. It’s not enough to be doing AI; you have to show the business value. Taskrouter’s analytics show exactly how much money and time efficient routing saves you.

That is the whole idea: real tasks, measured models, and clearer choices.

Try it, and tell us where it breaks

Sign up by October 10th and get $25 in free inference credits, good on any model. taskrouter.com

We also want your feedback. Throw your hardest tasks at Taskrouter and tell us where it breaks.

Stop paying frontier prices for everyday tasks. Let Taskrouter learn your workloads and optimize your AI infrastructure automatically.

Most AI teams run inference as income. A request comes in, you call a model, you get an answer, the answer is consumed. The next request costs the same, uses the same model, and produces the same quality. You’ve spent the token. Nothing accrued.

Tobi Lütke posted a chart on September 1 that shows what reinvestment looks like. Shopify’s ML team fine-tuned a 0.8B-parameter model for one job, building buyer profiles, and it now beats GPT-5.6-sol at its highest reasoning setting on that task: 84.6 versus 83.0 on their judge. Tobi’s take was that tiny models for special-purpose tasks work “incredibly well” when you have “a great self improving recursive flywheel.” Read that carefully. The result isn’t the small model. The result is the machine that produced it.

What the chart actually shows

Three training runs in one week. July 23, 29k samples, score 75.3. July 27, 42k samples, 78.1. July 30, 54k samples, 84.6. The gains got bigger with each run, not smaller. That’s the shape of compounding, and it came from three loops running simultaneously.

The quality loop. The frontier model is the teacher. Its outputs get graded, and the ones that pass become training data for the student. Every expensive API call is doing two jobs: serving the request and depositing a labeled example into your dataset.

The cost loop. The system prompt went from 9.1K tokens written out in full to 1.1K tokens “gisted,” an 8x cut. Once the model has learned the instructions through training, you stop paying to re-explain them on every call. Smaller model, shorter prompt, cheaper request.

The volume loop. Throughput went from 2M profiles per day on the prior 2B production model to 72M per day on the 0.8B model across 100 H100s. A 36x increase. More volume means more graded samples, which means better training data, which means the next run is cheaper and better still. The savings from cheap inference fund the inference that generates the next round of training data.

Each loop feeds the other two. That’s the reinvestment.

What it takes to build this

Anyone can fine-tune Qwen. That isn’t the hard part and it isn’t the moat. The Shopify team went from behind the frontier to ahead of it in seven days because the surrounding infrastructure already existed. Four pieces matter.

A task-level definition of quality. Not “is the model good” but “is this specific output a good buyer profile.” You need a judge that scores individual outputs on this task, calibrated well enough that you’d trust it to decide what goes into the training set. Without this, you have no way to know whether run three is better than run two, and no way to filter the teacher’s outputs into clean data. Everything downstream depends on it.

A pipeline from production traffic to training samples. The data isn’t in a lab. It’s in the requests you’re already serving. If your inference logs are write-only, you’re paying the teacher and throwing the lesson away. The plumbing that turns yesterday’s traffic into today’s graded dataset is the actual asset.

A cadence. Three runs in a week. Not a quarterly retraining project with a Jira epic. The flywheel compounds at the frequency you turn it, and a team that ships a new checkpoint every few days will lap a team that does it twice a year, even if the second team’s individual runs are better.

Clear routing between teacher and student. Early on, the frontier model handles everything and the student learns. As the student closes the gap, traffic shifts. You need to know, per task, which model is winning right now, and the answer changes weekly. Guessing here either wastes money on the teacher or ships bad outputs from the student.

The mindset shift

Stop thinking about your API bill as rent. Think of it as tuition. The frontier model’s job on any narrow, high-volume task is to make itself unnecessary for that task. If you’re six months into production and still calling the same frontier model with the same 9K-token prompt for the same job, you’ve been paying tuition and skipping class.

The teams that internalize this won’t look dramatically different next quarter. Compounding never does at first. They’ll look untouchable in two years, with dozens of tasks running on models that cost a fraction of frontier pricing and outperform it, and it’ll seem sudden to everyone who was paying simple interest the whole time.

If you want a system that makes it easy to compound your IP into AI models, that’s what we do here at Neurometric. Contact us if you would like to chat.

1. Document Structured Extraction --Published

Endpoint: neurometric/document-structured-extraction

Use this to extract selected fields from document text. The caller supplies the document and an output schema defining the requested fields and types. Missing fields are returned as
null.

Example input:

{”document”:”Invoice 1042. Vendor: Acme Corporation. Date: 2026-09-01. Total: USD 250.00.”,”output_schema”:
{”invoice_id”:”string”,”vendor”:”string”,”date”:”string”,”total”:”number”,”currency”:”string”,”purchase_order”:”string|null”}}

Expected output:

{”invoice_id”:”1042”,”vendor”:”Acme Corporation”,”date”:”2026-09-01”,”total”:250,”currency”:”USD”,”purchase_order”:null}

It currently expects extracted text rather than a raw PDF.

2. Grounded Document QA

Endpoint: neurometric/grounded-document-qa

Use this to answer questions using only supplied document chunks. The response includes the answer and supporting source IDs. If the documents do not contain the answer, the endpoint abstains.

Example input:

{”chunks”:[{”id”:”POLICY-1”,”status”:”authoritative”,”text”:”Customers may return unused products within 30 days of purchase with a receipt.”},{”id”:”POLICY-
2”,”status”:”authoritative”,”text”:”Refunds are issued to the original payment method.”}],”question”:”How long do customers have to return an unused product?”}

Expected output:

{”answer”:”30 days”,”citations”:[”POLICY-1”]}

Example when the answer is unavailable:

{”answer”:null,”citations”:[]}

3. Conversation Summary

Endpoint: neurometric/conversation-summary

Use this to extract the latest decision, current status, outstanding actions, owners, deadlines, and material risks from a business conversation. Completed and superseded information
is excluded.

Example input:

{”conversation_id”:”THREAD-001”,”messages”:[{”id”:”MSG-001”,”body”:”FINAL DECISION: Launch the new customer portal”},{”id”:”MSG-002”,”body”:”CURRENT STATUS: Waiting for security
approval”},{”id”:”MSG-003”,”body”:”OPEN ACTION: Send the approval packet. OWNER: Amina. DUE: 2026-09-10.”},{”id”:”MSG-004”,”body”:”MATERIAL RISK: Security review may delay the
launch”}]}

Expected output:

{”decision”:”Launch the new customer portal”,”current_status”:”Waiting for security approval”,”open_items”:[{”action”:”Send the approval packet”,”owner”:”Amina”,”due_date”:”2026-09-
10”}],”risks”:[”Security review may delay the launch”]}

4. Classification Router
Endpoint: neurometric/classification-router

Use this to classify requests using a taxonomy supplied at request time. The caller provides labels, their meanings, a fallback label, and the requests to classify.

Example input:

{”labels”:{”Billing”:”charges or refunds”,”Technical”:”errors or outages”,”Other”:”anything else”},”fallback_label”:”Other”,”requests”:[{”id”:”REQ-1”,”text”:”Please refund this
duplicate charge.”},{”id”:”REQ-2”,”text”:”The API returns a 502 error.”},{”id”:”REQ-3”,”text”:”What is your office address?”}]}

Expected output:

{”REQ-1”:”Billing”,”REQ-2”:”Technical”,”REQ-3”:”Other”}

Most teams building agents are paying frontier-model prices for a task that does not need a frontier model. Tool selection is a narrow, highly structured problem. Treating it as one changes the economics of the whole system.

What you get

Cost reduction on the largest line item. In a running agent, the expensive part is not reasoning. It’s context accumulation: re-sending tool definitions, past tool outputs, and environment state on every single turn. That accounts for 80–90% of agent spend in most architectures. Moving tool selection onto a small, cheap model drops cost per turn by 70–90%, and the effect compounds with every additional turn in a loop.

Lower latency end to end. Small models deliver much faster time to first token and higher generation throughput. In multi-step agents, 30–40% of wall-clock time disappears into inter-call orchestration overhead rather than useful work. Shortening each hop shortens the entire chain. Users experience this as an agent that feels responsive instead of one that appears to stall between steps.

Higher schema reliability. A model fine-tuned exclusively on JSON schema adherence produces fewer syntax errors, fewer missing required arguments, and fewer hallucinated parameters than a general-purpose model juggling reasoning, tone, and format compliance in a single pass. Specialization beats breadth here. Fewer malformed calls also means fewer retries, which is a second-order cost and latency win.

Tighter control over what leaves your perimeter. Tool schemas are a description of your internal systems: database fields, endpoint names, argument semantics. Isolating schema binding in a dedicated small model lets you keep those definitions out of the context you send to a general-purpose provider, and gives you a deployment story that fits inside a VPC or on-prem environment when that matters.

How to use it

Four patterns cover most of what we’ve seen work.

Intent router (fast path). Put the model at the entry point of your pipeline. Standard single-turn requests such as “look up user ID 1234” or “fetch local weather” go straight to it, which generates the API payload and executes. Your reasoning model never sees the request. In most production agents, a large majority of turns look like this.

Planner/executor split. Use a large reasoning model strictly to decompose a complex problem into natural-language sub-goals. Hand each sub-goal to the small model, which converts the instruction into valid arguments and calls the tool. The expensive model does planning, which is what it’s good at. The cheap model does binding, which is what it’s good at.

Structured output compiler. Let your reasoning model emit its decision as lightweight text rather than strict JSON. Feed that text to the small model to map intent onto schema-bound parameters. This removes a real failure mode, where a reasoning model degrades its own reasoning because it is simultaneously trying to satisfy a format constraint.

Speculative tool generation. Fire the small model in parallel to predict likely tool calls from the raw user input while your primary model is still drafting its strategy. When the prediction matches, you’ve already paid the latency. At these prices, speculating and discarding is cheap enough to be a rounding error.

Getting started

The model is live on TrustedRouter now. If you’re already routing through TrustedRouter, you can point tool-calling traffic at it and compare against your current path directly, on your own workloads.

AceBench measures exactly that: ~2,000 hand-annotated tasks drawn from ~4,500 synthetic APIs across eight domains (finance, health, travel, tech, entertainment, and more), published January 2025 and later accepted at EMNLP.

Three things make it worth a practitioner’s attention:

Grading is mechanical and brutal. Right function, right call count, right arguments — or it’s wrong. No partial credit, no AI judge. One bad field voids the form. Harsher than real life, but reproducible.

It comes in tiers. Normal (clear requests), Special (vague), Agent (multi-turn). We ran Normal, English only — this says nothing about ambiguity or agent loops.

It’s designed to be taken apart. Tasks isolate specific failure modes: value types, near-duplicate functions, mid-conversation drops. A leaderboard number says which model wins overall. A benchmark that comes apart says whether a cheap model is good enough for your work — usually the real question.

What we ran

Five models, 772 tasks each, 3,860 attempts, English Normal split:

• gemini-3.6-flash, gemini-3.1-flash-lite — Google’s hosted models

• gemma-4-12B-it, gemma-4-E4B-it, gemma-4-E2B-it — open-weight, self-served (big/small/very small)

Same harness and tasks for everyone. Tool calls return a bare acknowledgement — no feedback, one shot per call.

Key point: the open model on hardware we control tied Google’s hosted model, using about an eighth of the words.

The scoreboard

The top two are a real tie: 45 tasks the hosted model got and the open one missed, 35 the other way. That’s the signature of two equally capable models, not a better and a worse one. Run it again and the order could flip. Every other gap in the table is real — but the tie at the top is between a hosted frontier model and a 12B open model on a single GPU.

What it costs

Scoring the same isn’t interesting. Scoring the same this cheaply is.

ModelWords per correct answer*:

*tokens (~¾ word each), total output

The hosted model generates ~8x more text per correct answer than the 12B model it’s tied with — almost all invisible “thinking” (655,000 tokens across the run). The other four models did none of that: read, call, stop.

And the thinking isn’t buying anything. When the hosted model got a task wrong, it generated more than twice as much text as when it got one right. Extra effort here is a sign of being stuck, not a way out. On a benchmark where the winning move is two steps, there’s not much to think about.

Generated tokens are the expensive half of any pricing page, and what determines latency. So: same accuracy, a fraction of the tokens, and a deployment you own instead of rent.

The catch

Cheap on tokens ≠ cheap in practice:

The 12B model that ties Google’s best was 4x slower per answer than the API it ties, despite generating far less text. The smaller E4B, on the same GPU, hit 5.6s at ~91% of the top score. Note: these numbers are default vLLM, full BF16, on a single L40S — plenty of room to push both further with better hardware.

The smallest model is better than its score

gemma-4-E2B-it came last at 67%. Look at how it failed and half the gap disappears.

Its top mistake wasn’t picking the wrong tool — it was making the right call plus extra, already-completed ones from earlier in the conversation:

Called: ask about gift etiquette → send the gift → schedule the meeting Wanted: schedule the meeting

It knew the current turn. It just also re-did two things already done. Graded: zero.

72 of its 255 failures end with exactly the right call, buried under repeated history. Score only the current turn and it jumps from 67% to 76% — close to Google’s smaller model. That’s a prompt fix, not a bigger model — and it barely moves the other four (0-3 failures each of this kind).

Where bigger models still earn their keep

Three places:

Nested arguments. A tool wanting {"journey": {"from": "Shanghai", "to": "Hangzhou", "times": {...}}} trips up everyone — best models barely clear 60%, versus 85-96% for flatter arguments.

Several calls at once. Same tool, three times, different details: the hosted model pulls ahead ~5 points. Coordinating calls is harder than making one.

Long conversations — the sharpest split. Four of five models improve turn over turn as context narrows the options. The smallest model collapses:

That last column is what matters in production: a conversation only works if every turn lands. 87% per-turn becomes 74% overall.

Two models beat one

Since every model saw the same tasks, we can check what a pair covers:

Pairing the hosted model with the open one adds 4.5 points; pairing it with Google’s other model adds just 2.1. The two hosted models fail on the same tasks — the open one fails on different ones. If you can check an answer and retry, the self-served model is the better (and cheaper) second opinion.

So what do you do with this

If the task is “here’s what I want, here are the tools, go” — use a small self-hosted model. There’s no ambiguity for extra reasoning to resolve. A 12B model matches the frontier here; even 4B gets within ~90%.

If the work needs complex structures, coordinated multi-calls, or long conversations, the gap reopens — fastest for the smallest models.

Check what a scoreboard actually measures before trusting it: one category here was scoring a missing input, not the model, so we dropped it. Another score was understated nine points over a technicality about which calls “count” for a turn. Neither shows up in a leaderboard number — and both change what you’d buy.

The paper focuses on terminal agents — models given nothing but command-line access as a tool, which turns out to be enough to tackle an enormous range of tasks, from software engineering to security work to scheduling. Using this recipe, AI2 boosted a Qwen model’s Terminal-Bench score from roughly 25% to 31%, closing meaningful ground on Claude Haiku’s roughly 33%.

What made this one worth an episode wasn’t just the score bump. It was the design choices behind it.

Watch on YouTube: https://youtu.be/4U0gqv9sSn4?si=ejv04gktGbTOMJM8

Listen: https://tokenengineering.podbean.com

Paper: TMax: A recipe for terminal agents — https://arxiv.org/abs/2606.23321

Three models, two tool interfaces, 340 CRM tasks each — 2,040 agent rollouts in total.

The benchmark is CRMArena, run through the Harbor/dockworker harness. Every task asks a question about a read-only Salesforce org (”in May 2021, which state had the quickest case closures?”, “which knowledge article does this quote violate?”) and grades the submitted answer by exact match. Seventeen task categories, split evenly across a B2B and a B2C org.

The interesting design choice is that the same 340 questions are posed twice, behind two different tool surfaces:

Identical questions, identical ground truth — we verified this holds on every task where both suites produced a graded answer. So the two suites isolate one variable: does the model do better writing SQL, or picking the right pre-built tool and filling in its arguments?

The three models: gemini-3.6-flash and gemini-3.1-flash-lite (hosted), and gemma-4-E4B-it (open weights, served locally on vLLM). One is a small local model; two are hosted. Every trial used the same noshell agent scaffold, whose only way to answer is to call a terminal submit_answer tool.

The scoreboard

Read it and you’d conclude the ordering is obvious and the small model isn’t close: 39.7% against 67.6% is a 28-point gap. Every pairwise difference here is statistically significant (p ≤ 0.009).

Then you look at how the failures happen, and the picture inverts.

Half of the small model’s “failures” contain the right answer

Not every zero is a wrong answer. A trial scores zero if the answer file was never written at all — and the scaffold only writes it when the model calls submit_answer. Classifying all 2,040 trials by how they failed:

More than half of gemma’s SQL trials never submitted anything. And they didn’t crash or time out — in 171 of them the model finished its work, wrote the answer out in prose, and simply never called the tool. Something like:

The case_metrics call returned an average closure time of 4.2 days for CA, the lowest of any state. The answer is CA.

Graded: zero.

So we tested the obvious question — were those prose answers right? Ground truth isn’t recorded for ungraded trials, but because the same 340 tasks appear in both suites, we could recover the expected answer for every one of them and grep the final message for it.

Of gemma’s 250 prose non-submissions, 44–52% contained the correct answer. The range is the strict and loose reading of the same check: the loose count is any trial whose prose contains the expected value; the strict count additionally requires that the model wasn’t hedging across a list of candidates (≤1 other record ID mentioned). Both bound the same conclusion.

That reframes the scoreboard as a lower bound. Fixing one scaffold behaviour — get the model to call the tool — moves gemma to:

A model that looked like it scored 26% on SQL was doing work worth about 50%. Its measured number was roughly half its actual competence, and every point of that gap is a formatting bug.

The same correction barely moves the hosted models — gemini-3.6-flash has exactly zero prose non-submissions, and gemini-3.1-flash-lite has one. They always call the tool. What we were measuring, for a third of the benchmark, was instruction-following on the harness contract, not CRM reasoning.

There’s a cleaner way to see it. Restrict to trials that submitted anything, and the reasoning quality behind the scoreboard separates from the plumbing:

On answers it actually submits, the small local model is within two points of the hosted flash-lite model — a difference well inside the noise at this sample size. The 11-point headline gap between them is almost entirely tool-calling discipline.

What it costs

This is where the small model stops being a curiosity. Token totals are the whole run; the per-win column divides by correct answers, which is the number that matters if you’re paying for throughput.

Gemma on the tool API buys a correct answer for 45.4k tokens. gemini-3.6-flash needs 196k for the same thing — 4.3× more. On SQL it needs 598k, or 13× gemma’s best configuration.

The driver is visible in the reasoning-token column. gemini-3.6-flash spent 1.36M reasoning tokens on the API suite and 2.59M on SQL. The other two models spent none. That’s what the extra accuracy is bought with: 3.6-flash’s median trial emits 3,978 output tokens on the API suite against gemma’s 501 — an 8× difference in generated text per attempt.

And time to completion — median wall time of the agent execution phase alone, excluding container build and verification:

Here the small model does not win, and it’s worth being precise about why. gemma is 12.6s per attempt against 3.6-flash’s 30.0s — faster per attempt — but because it converts fewer attempts into correct answers, it lands at 73.8s per win versus 51.8s. And this column mixes model speed with serving infrastructure: a local vLLM instance against Google’s production endpoints. It is not a property of the models, and it’s the one metric here we’d throw out of a purchasing decision. Tokens are the honest efficiency measure; wall-clock is an artifact of where each model happened to be running.

One more pattern worth noting: every model spends more tokens on the trials it gets wrong — 1.5× to 2.1× its passing median. Failure is not cheap. Effort is a symptom of being lost, not a route out of it, which means the token cost of a wrong answer exceeds the token cost of a right one across the board.

Where each model actually fails

The three models have almost nothing in common in their failure profiles.

gemma-4-E4B-it — a formatting problem wearing a capability problem’s clothes. 250 of its failures are prose-instead-of-tool-call; roughly half contain the right answer. Its second tendency is over-caution: 50 trials answered None (”nothing matches”) when a real record existed. It abstains too readily and it won’t call the terminal tool. Both are addressable without touching the model.

gemini-3.1-flash-lite — quietly broken generations. 34 API trials (10%) ended with an empty completion: a single EOS token, no text and no tool call. Not prose, not a wrong answer — nothing at all. That signature points at serving or sampling rather than the prompt, and it’s the cheapest 10% anyone in this comparison could recover. Beyond that its losses are ordinary wrong answers (21–23%), the highest wrong-answer rate of the three.

gemini-3.6-flash — runs out of budget, and won’t say “none.” It never fails to submit when it finishes, but it frequently doesn’t finish: 60 SQL trials (18%) and 17 API trials (5%) hit the 25-step cap mid-work. Its failing trials burn a median 9,747 output tokens against 6,452 when passing — it iterates on queries that never land. Its other weakness is the mirror image of gemma’s: 63 trials invented a value where None was correct. The strongest model is the one most likely to manufacture an answer rather than concede there isn’t one.

That last contrast is the most useful thing here for anyone building on these models. The two abstention errors point in opposite directions — gemma says “none” when an answer exists, 3.6-flash asserts an answer when none does — so there is no single prompt that fixes both. It’s a calibration problem per model, not a benchmark-wide one.

The interface matters, and not the way you’d guess

Because the same questions appear behind both tool surfaces, we can ask whether SQL or semantic tools suit each model better as a paired comparison — the same task, two interfaces — using an exact McNemar test on the tasks where the two disagree.

For gemini-3.6-flash, the only model whose two runs had matched step budgets, the tool API wins clearly: +8.5 points, p < 0.001, with 47 tasks solved only through the API against 18 solved only through SQL.

But the aggregate hides something better. Across the models, between 65 and 133 of the 340 tasks flip outcome between the two interfaces. For gemini-3.1-flash-lite the headline rates are nearly identical (49.1% vs 50.9%) while 94 tasks flip — 44 solved only via SQL, 50 only via the API, with just 123 solved by both. The interface doesn’t change how many tasks it gets right; it changes which third of the benchmark it gets right. A model that looks interface-indifferent on the scoreboard is nothing of the kind, and running both interfaces and taking either success would score far above either alone.

Category detail shows where this bites. gemini-3.6-flash gets sales-amount-understanding right 60% of the time through semantic tools and 5% through SQL — but that collapse is not an inability to write the query. 18 of its 20 SQL attempts hit the step cap mid-work. Where a semantic tool answers “total order amount by owner in this window” in one call, SQL requires discovering the schema, joining line items to orders, and windowing on the right date column — and it ran out of budget getting there. lead-routing, by contrast, it solves 100% either way.

Not all 17 categories are the same benchmark

Treating CRMArena as one number hides the most actionable result in the run. Here is every category, worst-first, across all six runs (36 = gemini-3.6-flash, 31 = gemini-3.1-flash-lite, gm = gemma-4-E4B-it):

Group the 17 categories by what they actually ask for, and a pattern appears that the headline rates completely obscure:

On policy judgement, model scale buys almost nothing. Reading a chat transcript and deciding whether an agent breached policy, or whether a lead is qualified: gemini-3.6-flash manages 37.5%, gemma manages 32.5%. The small model is at 87% of the best model’s performance. Compare that to aggregate metrics, where the same small model reaches only 57% of the frontier score and the spread across runs hits 85 points on monthly-trend-analysis.

The reason is visible in the failure modes. On policy-violation-identification, half of gemini-3.6-flash’s API failures are missed_abstention — 8 trials asserting a violation where the correct answer was “none” — against 8 passes. gemma’s failures on the same category split 4 missed_abstention and 4 false_abstention. This category isn’t primarily testing reasoning; it’s testing whether a model will decline to answer. That’s a calibration property, and calibration doesn’t scale with capability the way multi-step numeric work does.

So the practical read is: the categories where a small model is nearly as good are the judgement ones, and the categories where it falls off a cliff are the multi-step aggregations. If your workload is “read this conversation and classify it,” a 4B local model is a serious candidate. If it’s “compute this metric across three joins and a date window,” it is not.

Two more things the table says:

invalid-config is the one genuine capability cliff. gemini-3.6-flash gets 50% on both interfaces; every other run scores 5–10%. A 45-point gap that survives both tool surfaces is the clearest evidence in this run of something the smaller models simply cannot do — matching a quote’s configuration against the knowledge article it violates. Notably it’s not a formatting artifact: gemma’s failures here are 8 wrong answers alongside 7 prose non-submissions, so even the recovered ceiling stays low.

quote-approval defeats everything. 8% pooled, topping out at 20%, and gemini-3.6-flash scores 5% and 0%. No model, no interface, no step budget helps. When the strongest model in a comparison scores 5% on a category, that’s usually a signal about the task or the grader rather than the models — and it’s the first thing we’d re-examine before drawing conclusions about the remaining headroom.

And where does the small model actually beat a hosted model head-to-head? On the tool API, gemma matches or beats gemini-3.1-flash-lite in 5 of 17 categories: policy-violation-identification (35% vs 15%, +20 points), wrong-stage-rectification (40% vs 35%), invalid-config (10% vs 5%), and ties on case-routing and lead-routing. Four of those five are judgement or routing tasks, not aggregations — the same pattern again.

What we’d take away

The headline number on an agent benchmark is a joint measurement of the model and the scaffold around it, and for small models the scaffold dominates. gemma-4-E4B-it looked like a 26% model on SQL. It was doing work worth about 50%, and the missing half was one unmade tool call. Anyone comparing a small open model against a hosted frontier model on a leaderboard number, without looking at the failure modes underneath it, is partly measuring which model was better at following the harness’s calling convention.

Small models win on the axis that gets left off the leaderboard. At 45.4k tokens per correct answer against 196k, gemma is 4.3× cheaper per unit of useful output than the model that beats it by 28 points — and it gets there with zero reasoning tokens against 1.36M. If your workload tolerates 50% task accuracy, or you can put a verifier behind it and retry, the small model is the better economic choice by a wide margin.

But the honest version of “small models can win” is narrower than the slogan. gemma did not beat gemini-3.6-flash in a single one of the 17 categories, on either interface. What it did was reach parity with a hosted flash-lite model on reasoning quality — 56.0% against 57.9% on submitted answers — at a fraction of the token cost, while running locally. That’s the claim the data supports: not that small models beat frontier models, but that the gap to the tier below frontier is mostly plumbing, and the plumbing is cheap to fix.

The category breakdown sharpens that into a selection rule. The gap is not uniform across work: on policy judgement the small model is at 87% of the frontier model, on multi-step aggregation it is at 57%, and on one category (invalid-config) it is nowhere near. “Can a small model do this?” has no general answer, but it has a reliable one per task family — and the families where the answer is yes are the judgement-shaped ones, which is the opposite of where most people assume a small model will struggle.

The fixes are unglamorous and they’re all in the harness: make the terminal tool call unmissable for models that write prose, investigate the empty completions, raise the step cap for models that iterate, and calibrate abstention per model rather than globally. None of them require a bigger model.

What Is TrustedRouter?

TrustedRouter is an inference gateway in the same family as OpenRouter: one OpenAI-compatible API, one base URL, and hundreds of models and routes behind it. You point your existing client at the gateway and swap model strings instead of rewriting integrations every time you want to try a different provider.

What makes it interesting is the trust posture. TrustedRouter keeps no record of what you route — no prompt or output logs, by design, and the gateway is attested and fails closed rather than silently degrading. It publishes a live status page and a continuously sampled performance leaderboard, so latency and uptime claims are measured rather than asserted. There’s also a documented migration path from OpenRouter for teams that want to move without a rewrite.

For anyone who has spent time explaining to a security team why prompts containing customer data are being logged by a third party, that combination is worth something.

What We’re Serving Today

We’re live with three small models on prepaid routes:

• ibm-granite/granite-4.1-8b — 32K context, $0.0525/1M prompt and $0.105/1M completion

• qwen/qwen3-vl-8b-instruct — 262K context, vision-capable

• qwen/qwen3-vl-8b-thinking — 32K context, reasoning traces

Early measured numbers on our routes: 863 ms p50 time-to-first-token on Granite and 100% uptime across the sampling window. Our listed policy note is accurate about what we do and don’t guarantee — upstream trace logging is disabled, prompts and completions are not written to observability or object storage, and we retain only aggregate request counts and token totals. That’s a no-store posture, not contractual ZDR, and we’d rather say so plainly than let people assume more than we’ve committed to.

Task-Specific Models Are Coming Next

Small general-purpose models are the entry point, not the thesis. The reason we build the way we do is that most production workloads aren’t “chat” — they’re a narrow, repetitive task where a tuned 8B model matches or beats a frontier model at a fraction of the cost.

Three task-specific models from our model marketplace we expect to list on TrustedRouter in the coming weeks:

1. Text-to-SQL — natural language to correct, schema-aware queries against a known database, tuned for join accuracy rather than conversational fluency.

2. Structured extraction — pulling typed JSON out of invoices, contracts, and claims documents against a supplied schema, with predictable failure behavior on missing fields.

3. Support intent classification — routing inbound tickets to the right queue and priority, the kind of high-volume, low-token task where per-call cost dominates everything else.

Each is graded against our proprietary task benchmark corpus, so you can see how a specialist performs on your task class before you route production traffic to it.

If you want to try the current models, grab a key at TrustedRouter and use neurometric as the provider. If you have a task class you’d like us to build for, tell us — that’s how the roadmap gets set.

To cut through the hype, organizations must fundamentally rethink their deployment strategies. In our experience, the most critical decision a technical leader makes isn’t which specific foundational model to choose, but rather the strategic posture they take toward the underlying business task. We have observed that enterprise AI strategy ultimately boils down to a fundamental set of decisions. There are usually four options on the table, and they require a level of candor that is often missing from vendor pitches and internal strategy meetings.

The Four Options

When evaluating a new AI capability or use case, technology leaders must choose between four distinct paths: Buy, Build, Route, or Wait.

Buy is the right answer much more often than technical leaders want it to be. Engineers are naturally wired to create, and the allure of constructing a bespoke AI system is incredibly strong. However, if a vendor has already solved a generic enterprise problem—like drafting marketing copy, summarizing meeting notes, or triaging customer support tickets—purchasing that solution is almost always the superior economic choice.

Build is the right answer only when the task at hand is the core business. If the AI system is going to directly drive your competitive advantage in the marketplace, you cannot outsource it.

Route is the correct posture when the task is high-volume and the models are fungible. If you are processing millions of identical queries where the subtle nuances of a massive frontier model are unnecessary, routing queries to the most efficient model available is the only way to scale without destroying your margins.

Finally, there is Wait. Wait is the honest default. It is the right decision more often than anyone wants to admit, and yet it is almost never proposed in a strategy meeting. There is immense career risk in telling a CEO to wait on AI. But for highly volatile use cases, or problems where the underlying foundational models are currently struggling but rapidly improving, waiting six months for the ecosystem to mature is often vastly superior to burning capital on a brittle, premature V1.

The “Build Test”

If you are leaning toward building a custom AI solution, you must subject your proposal to the “Build Test.” Building bespoke AI is an expensive, resource-intensive endeavor that requires long-term commitment. At Neurometric, we advise clients to build only if they can definitively answer “yes” to all three of the following criteria:

First, is the task core to your differentiation? The AI must do something that separates you from your competitors. If it is merely an operational efficiency that every other company in your sector will eventually adopt, it fails this test.

Second, do you possess proprietary data or a unique workflow that a vendor structurally cannot access? You need an unfair advantage. If you are building a model using the exact same public datasets and standard enterprise tools as the major SaaS vendors, they will eventually commoditize your creation. You must have a moat built on data or processes that are uniquely yours.

Third, can you staff the maintenance of this system for the next three years? Building the model is merely the starting line. Models drift, APIs change, underlying data distributions shift, and security vulnerabilities emerge. You are not just funding a build phase; you are funding a permanent product team.

If you meet two out of these three criteria, it is not a “maybe.” Two out of three means you default to Buy. The economics of maintaining a sub-scale, non-differentiated AI system will slowly drain your engineering resources.

Model Selection: The Frontier Premium

Once you have decided how to acquire the capability, you must choose the right engine. The market is currently bifurcated between massive “frontier” models and smaller, specialized, or distilled models.

Frontier models are the bleeding-edge giants of the industry. They possess incredible reasoning capabilities and vast world knowledge. You should reserve these models strictly for tasks requiring deep judgment, handling high ambiguity, and managing low-volume, high-stakes scenarios. If an AI is reviewing a complex legal contract for a multi-million dollar merger, you want the frontier model.

Conversely, small, specialized, or distilled models are the workhorses of the modern enterprise. These should be deployed for high-volume, well-specified, and latency-sensitive tasks. When you are processing tens of thousands of basic data extraction requests per hour, you do not need an AI that can write a sonnet or pass the bar exam.

The divergence in economics here is huge, often separated by one to two orders of magnitude. A distilled model can easily be 10x to 100x cheaper per token than a frontier model, while returning responses in a fraction of the time. Yet, we routinely see organizations using frontier models for absolutely everything simply because it requires only one API integration. They are paying a 30× premium for the privilege of architectural laziness.

Routing as an Operating Discipline

To capture the economic benefits of smaller models without sacrificing quality, organizations must adopt routing as a core operating discipline.

The biggest mistake teams make is routing at the application level—deciding that an entire application will use just one model. Instead, you must route per task. A single customer service application might use a cheap, fast model to identify the language of an incoming ticket, a specialized model to extract the customer’s account number, and only invoke a frontier model if the ticket requires a complex, nuanced apology for a service failure.

Implementing this requires establishing a strict quality floor for every specific task. You determine the minimum acceptable accuracy, and then you dynamically route the workload to the absolute cheapest model that clears that floor. Because the open-source and proprietary model landscapes are evolving at a breakneck pace, this is not a set-it-and-forget-it architecture. You must re-benchmark your routing logic monthly. The model that was the most cost-effective in January might be entirely obsolete by April.

Vendor Risk and AI Sovereignty

Finally, organizations must wake up to the reality of vendor risk in the AI space. It is real, it is severe, and it is currently vastly underweighted in enterprise risk assessments.

When you build a product highly dependent on a third-party API, you are at the mercy of their roadmap. Deprecation schedules are often brutally short, forcing expensive emergency migrations when a vendor decides to sunset a specific model version. Price changes can occur overnight, destroying the unit economics of your application. Rate limits can be abruptly tightened, throttling your application’s ability to scale during a surge in user demand.

Furthermore, data privacy terms are a constantly shifting target. An update to a vendor’s terms of service could suddenly allow them to use your enterprise data to train their next generation of models, violating your internal compliance policies.

Most insidiously, a model can change its behavior right underneath you without any version bump. Vendors continuously tweak, align, and “improve” their models behind the scenes. A prompt that reliably output perfect JSON formatting on Tuesday might suddenly start wrapping its output in conversational pleasantries on Thursday, breaking your entire data pipeline.

To mitigate this, technology leaders must push back during procurement. Do not accept standard click-wrap agreements for critical AI infrastructure. You must explicitly contract for extended notice periods regarding deprecations, strict rate limit guarantees, immutable data terms, and rigid version control to ensure the model you test is the exact model you run in production.

The AI ecosystem is moving incredibly fast, but the fundamental laws of enterprise software engineering and business economics have not been suspended. By rigorously applying the Build Test, rightsizing your model selection, establishing a disciplined routing architecture, and aggressively managing vendor risk, you can navigate this landscape successfully. The goal isn’t to deploy AI the fastest; the goal is to deploy it in a way that actually works for your business.

If you want to shield yourself from the chaos of the model selection and routing ecosystem, Neurometric’s core platform evaluates, chooses, and routes models automatically. Try it for free.

This should feel familiar. When companies moved to the cloud, an entire layer of cost management and optimization tooling emerged around it — because once compute became a metered utility, optimizing that meter became a discipline with real dollars attached. AI inference is following the same path, except the stakes are arriving faster. A production AI workload at enterprise scale — say, a million model calls a day — can swing by millions of dollars a year based on decisions most teams made once, early, and never revisited.

Token engineering is the discipline that fixes this. A token engineering platform is the system that operationalizes it.

What Is Token Engineering?

Token engineering is the practice of treating tokens as an engineered resource: measured, benchmarked, routed, and continuously optimized. It’s a systems discipline, not a procurement exercise.

The common misconception is that token engineering means “use a cheaper model.” It doesn’t. It means optimizing every AI workload across three dimensions simultaneously: cost, speed, and reliability. Sometimes the right answer is a smaller, cheaper model. Sometimes it’s a faster one. Sometimes it’s the frontier model, but with a compressed prompt and an aggressive caching layer in front of it. The point is that the answer is different for every task, and it changes constantly.

Three forces make this urgent right now. First, model proliferation: frontier LLMs, open-weight models, and small language models (SLMs) now number in the hundreds, with meaningful new releases every month. Second, price variance: the cost of completing the same task can vary by 100x or more depending on which model, technique, and hardware you choose. Third, the capability crossover: for a growing share of enterprise tasks, purpose-built SLMs now match or beat frontier models at a fraction of the cost.

Meanwhile, most teams pick a model at the start of a project, hardcode it, and move on. Every month that decision goes unexamined, the gap between what they pay and what they should pay gets wider.

The Core Components of a Token Engineering Platform

Managing this problem manually doesn’t scale. A token engineering platform automates it. Here’s what a complete platform looks like — and how the Neurometric platform implements each piece.

1. Workload Evaluation and Benchmarking

You cannot optimize what you cannot measure, and public leaderboards won’t save you. Generic benchmarks tell you how models perform on academic tasks, not on your workloads — your customer service intents, your document extraction formats, your code review standards.

A token engineering platform continuously evaluates models and token engineering techniques against your actual tasks. When a new model ships, you shouldn’t have to wonder whether it’s better for your use case; the platform should tell you, with graded evidence. Neurometric’s Harbor evaluation infrastructure does exactly this, with more than 15,000 graded tasks and thousands of benchmark runs powering every recommendation the platform makes.

2. Task-Level Routing

Evaluation tells you which model is best for which task. Routing acts on it — automatically, per request, in production.

This is the decision engine at the heart of the platform. Most applications don’t have one workload; they have dozens of distinct tasks hiding inside a single product, each with different cost, latency, and quality requirements. Routing at the application level means paying frontier prices for tasks a model one-tenth the cost handles perfectly. Routing at the task level means every request goes to the cheapest model that meets its quality bar. Our Task Level Router keeps this workload-to-model mapping current as models, prices, and your traffic all change — so the routing decision you’d make today doesn’t quietly decay into the wrong decision six months from now.

3. Automated SLM Creation and Fine-Tuning

Sometimes no existing model sits at the right point on the cost/quality frontier for your task. The frontier model is overkill and overpriced; the small models miss your quality bar. Historically, the answer was a fine-tuning project: hire ML engineers, build a data pipeline, spend a quarter.

A token engineering platform makes this a feature, not a project. When evaluation data shows that a task is a candidate for a purpose-built model, the platform can distill and fine-tune an SLM automatically, validate it against your benchmarks, and slot it into the routing layer. The economics are hard to ignore: a task-specific SLM can run at 1/50th the cost of a frontier model while matching its accuracy on that narrow task. Our Auto-SLM Creator turns what used to be a specialized data science effort into a platform capability.

4. Hardware and Deployment Optimization

Which model you run is only half the equation. Where you run it matters just as much.

The same open-weight model has wildly different economics depending on whether you consume it through an API, self-host it on dedicated GPUs, run it quantized on cheaper hardware, or batch it for throughput over latency. At scale, the API-versus-self-hosting decision alone can be worth seven figures annually — and the right answer flips as your volume grows and hardware prices move. A token engineering platform models these deployment economics continuously and recommends the optimal placement for each model in your stack, so the decision gets revisited by software instead of forgotten by people.

5. Prompt Compression, Rewriting, and Caching

The cheapest token is the one you never send.

Before a request ever reaches a model, there are three opportunities to shrink it: compress the prompt to strip redundancy, rewrite it for token efficiency without losing intent, and cache aggressively — both exact-match and semantic — so repeated or near-repeated requests never hit the model at all. Individually these look like small percentage gains. At enterprise volume they compound into real money: at a million calls a day, a 20% reduction in tokens per call is not a rounding error, it’s a budget line. The platform applies these techniques automatically and only where evaluation shows they don’t degrade quality.

6. Cost Attribution

None of the above works as a one-time exercise. Optimization needs a feedback loop, and that loop starts with knowing exactly where your tokens go.

A token engineering platform attributes token spend at the level your business actually thinks in: per task, per team, per product feature, per customer. This is the FinOps layer for AI. It’s what lets you answer questions like “what does our IVR summarization actually cost per call?” or “which feature’s AI spend grew 40% last quarter, and was that traffic or inefficiency?” Without attribution, every optimization is a guess. With it, the platform can show you — in dollars — what each routing decision, SLM deployment, and caching policy is saving.

7. Governance, Budgets, and Reliability Controls

Finally, the layer that makes all of this safe to run in production: spend limits per team or application, fallback chains when a provider degrades, SLA enforcement on latency and quality, and defined degradation policies for when things go wrong.

This is the difference between a developer tool and an enterprise platform. Your finance team gets budget enforcement. Your platform team gets reliability guarantees. Your compliance team gets an audit trail of which model handled which request and why.

How It All Fits Together

These aren’t seven point solutions bolted together. They’re a flywheel: benchmark → route → optimize → attribute → re-benchmark.

Evaluation data drives routing decisions. Routing data reveals which tasks are candidates for purpose-built SLMs. Deployment optimization changes the economics that feed back into routing. Cost attribution measures the impact of all of it and surfaces the next opportunity. Every component makes the others smarter, and the loop runs continuously — which matters, because the model landscape changes monthly and your traffic changes daily.

This is also the honest answer to the build-versus-buy question. Any strong engineering team can build one of these components. Very few can build all seven, and almost none can afford to maintain all seven against a model market that reprices and re-ranks itself every few weeks. Nobody builds their own cloud cost management platform anymore. The same logic is arriving for tokens.

In Summary

AI inference is becoming one of the largest new line items in enterprise technology budgets, and it’s currently one of the least managed. Token engineering is the discipline that closes that gap. A token engineering platform — evaluation, routing, automated SLM creation, deployment optimization, prompt optimization, cost attribution, and governance — is how you run it at scale.

If you’re spending real money on model calls and can’t say with confidence that each task is running on the right model, at the right price, on the right hardware, that’s the gap Neurometric was built to close. [Get in touch to benchmark your workloads -sales@neurometric.ai]

• The tokenminning.com origin story — (As seen in the NYT) The Tokenminning Manifesto by Neurometric is gaining traction, with coverage in the NYT. As enterprises prove out AI use cases while blowing through their token budgets, token engineering is becoming a first-order concern.

• Specialized models vs. frontier models — Small, task-specific models can almost always beat a frontier model on a given task, even at a fraction of the parameter count. The catch: that specialized model can only do that one thing well. This is a feature, not a bug.

• The new Token Engineering Platform — Neurometric's answer to a fragmented tooling landscape. It brings SLM fine-tuning, distillation, and token caching into a single system so teams get one view across their entire model stack.

•  COGS vs. OPEX — how Neurometric segments its customer base, and why companies whose AI spend hits gross margin (not just headcount-adjacent budgets) are the ones moving fastest toward token efficiency.

Listen to the full episode: https://tokenengineering.podbean.com/

Watch on YouTube: https://youtu.be/JHFeraXq3RU?si=fzzRcEBuPo3vbF6A

**Resources mentioned:**

• Tokenminning Manifesto: tokenminning.com

• Neurometric AI: neurometric.ai

• NYT Article: https://www.nytimes.com/2026/06/18/technology/ai-token-minimizing.html

They all started with one large frontier model, and as their inference costs grew, they started to peel off their high volume simpler workloads and set up “task specific endpoints.” Workloads like customer sentiment analysis of support tickets, named entity extraction from documents, email summarization, these don’t need frontier intelligence. By setting up an endpoint with a small model for just those workflows, they saw lower costs and faster latency.

The problem is, there are thousands of models out there, and hundreds of techniques to optimize them - from where you run the model to prompting and token management techniques. And that is all on top of the complicated issue of which model works best for your use case in the first place.

Today we are announcing that Neurometric has pulled all of these tools together in a single platform that makes it easy to make token engineering decisions. With our platform you can:

• Monitor and measure AI workloads

• Optimize workloads for cost or latency

• Build custom SLMs for workloads that benefit from that approach

• Evaluate and test many models, techniques, and hosting platforms

Our customers typically see an 80% drop in inference charges and a 4x improvement in latency using the Neurometric platform.

If you want take your AI optimization to the next level, hire a token engineer, and use the Neurometric platform. Reach out to us at sales@neurometric.ai if you want to learn more.

Rob May, our CEO, told the Times: CEOs who couldn’t measure AI savviness defaulted to “who’s using the most tokens.” Volume over efficiency was never going to hold once the bills landed.

Uber blew through its full-year AI budget in four months. Meta is capping usage after an “exponential increase” in costs. AT&T’s shows companies can save up to 90% by using less powerful models for most tasks.

This is the insight we built Neurometric on. 

Read the full article: https://www.nytimes.com/2026/06/18/technology/ai-token-minimizing.html

That’s the central argument of the Pioneer Agent paper from Fastino Labs — and it maps pretty cleanly onto what we’ve been building at Neurometric.

In this episode of Inference Time Tactics, Director of AI Research Yash Sharma breaks it down with co-founder Calvin Cooper: what Pioneer Agent actually is, how it uses Claude Sonnet as an ML engineer in a box, and what the results tell us about where SLMs go next.

Key topics:

— Cold start data curation with zero customer traces

— Why naive retraining fails with noisy production data (and how an agentic loop fixes it)

— 83-point benchmark improvements — and why the 1.6-point cases matter just as much

— Regression rollback, hyperparameter search, and the heuristics baked into the system

— Where we think SLMs can go beyond “simple tasks”

Watch the full episode on YouTube: https://youtu.be/PiIrywMGsAA?si=gs_3nDwgS7QbvwQG

Or listen in on any platform: https://inferencetimetactics.podstream.com

Subscribe

Currently, enterprise infrastructure spend is run with zero discipline. We regularly see frontier models costing $5 to $25 per million tokens assigned to commodity work that $0.10 to $0.50 small models handle easily on benchmarks. This creates a 50x to 250x price spread, paid on every request, every single day. To survive this efficiency gap, a new engineering discipline has emerged: Token Engineering. It is the systematic optimization of which model runs which task—at what size, with what prompt structure, and at what cost.

The Agentic Scale Problem

The root of this cost crisis is that our foundational software design patterns have completely changed. Over 70% of routed inference traffic now comes from autonomous agents and CLI tools—not human chat interfaces. While a standard chat turn consumes just a few thousand tokens, a single agentic task regularly chews through 100K to 1M tokens as it loops, reasons, and self-corrects.

This volume shift has caused platform-wide token volume to explode by more than 10x in roughly a year, with 16 to 18 trillion tokens per week routed on OpenRouter alone. When workloads scale to this magnitude, token waste becomes an engineering failure, not a model failure.

The Five Pillars of Token Waste

When auditing modern agent pipelines, token waste typically boils down to five core engineering oversights:

• Oversized Models: Allocating expensive frontier models to simple extraction, classification, and formatting tasks that small models win on benchmarks.

• Prompt Bloat: Deploying unversioned, unmeasured prompts that carry thousands of redundant tokens into every single call.

• No Caching Strategy: Re-sending massive chunks of static context on every request instead of caching it at 10% to 20% of the standard price.

• Sequential Sprawl: Running agent steps serially with full context when steps could be decomposed, parallelized, and right-sized across lean endpoints.

• Blind Retries: Retrying unexpected failures on the same expensive model with no confidence scoring or cheaper fallback path.

Why Human Optimization Fails

Fixing these leaks manually is a noble goal, but token engineering simply does not scale as a human job. First, we face continuous Model Churn. Major model releases land weekly. Look at Gemma 4: it went from non-existent to routing 240 billion tokens per week—roughly one-third of all small-model traffic on OpenRouter—in a mere 70 days. No human team can re-benchmark thousands of model-by-task combinations on that clock.

Second, we hit Price Churn. Providers reprice continuously; caching, batching, and changing infrastructure economics shift the optimal choice even when models remain static. The half-life of an optimization is measured in weeks; a hand-tuned pipeline is stale before the next sprint ends. The engineer has to be automated.

The FinOps Parallel

Every era of computing waste eventually forces the transition from a heroic manual task into an automated platform practice. When web applications struggled with uptime, we turned manual on-call duties into Site Reliability Engineering (SRE). When cloud spend spiraled out of control, cloud waste created FinOps platforms that natively paid for themselves. Today, token spend is the fastest-growing cost line in software, and it demands its own dedicated engineering platform.

As LLM infrastructure commoditizes, value is rapidly migrating away from foundational providers and straight to the orchestration layer—the core intelligence about which intelligence to use. By decoupling your software workflows from rigid API keys and transitioning to dynamic task endpoints, you can let automation drive delivery costs down via leaner prompts, continuous model re-matching, and purpose-built small models. In this new landscape, remember: every standard inference vendor makes more money when your system wastes tokens. True operational maturity belongs to the engineering teams who build systems that actively eliminate the bleed.

If you want help, Neurometric offers a platform ideal for token engineering. Contact us to chat more about it.

The Numbers

5,791 benchmark runs. 15,750 individual graded tasks.

That’s a meaningful dataset. But the headline isn’t a model score — it’s that the framework couldn’t reliably finish.

53% of runs errored out. An errored run produces nothing: zero tasks completed, no usable result. More than half our compute returned empty-handed, not because the models failed, but because the harness crashed.

This matters beyond the obvious waste. When your error rate crosses 50%, you’re no longer sampling performance — you’re sampling infrastructure luck.

The Grading Problem

The errors weren’t just crashes. They were silent distortions inside the results that did come back.

In 765 cases, the agent produced the exactly correct answer — and Harbor logged it as an error anyway. That’s one in five of all correct results misclassified as failures. A grading system that can’t recognize its own correct answers isn’t grading. It’s noise with a spreadsheet attached.

The implication: any model rankings produced under these conditions would systematically understate performance, with the degree of understatement varying arbitrarily across runs. You can’t normalize your way out of that.

Dataset Coverage

Harbor ships with 80 datasets. We got 42 of them to run at all — just over half. Of those 42, 17 never produced a single scored result. That’s 17 datasets that opened, ran, and returned nothing actionable.

Effective coverage: roughly 31% of the catalog produced data you could learn from. You cannot benchmark across a catalog when two-thirds of it is structurally inoperable.

The Hello-World Test

The clearest evidence is the simplest task.

Harbor’s hello-world benchmark asks the agent to create a file containing "Hello, world!" That’s it. No reasoning. No retrieval. No multi-step planning. Just: write a file.

We ran it across 1,526 agent+model combinations. 645 of them — 42% — never completed it cleanly even once.

In one case, GPT-5.4 wrote the file perfectly. The run still errored.

The intelligence was never the bottleneck. The harness was.

What This Means for AI Benchmarking

Benchmark infrastructure is load-bearing. It doesn’t just measure performance — it defines what counts as performance. When the harness fails silently, misclassifies correct answers, and drops the majority of its own datasets, the output isn’t a measurement. It’s a corrupted signal that looks like data.

The field has spent enormous energy debating which benchmarks best capture model capability. That’s the right debate to have — once the execution layer can be trusted. Our results suggest the execution layer deserves a lot more scrutiny than it typically gets.

A model that writes the correct answer deserves to have that answer counted. That’s table stakes for any evaluation system. When it isn’t met, the benchmark isn’t measuring models. It’s measuring the framework.

The OpenClaw cost problem is not a secret. Users are posting about it openly. One tech blogger documented $3,600 in a single month. Others report $200 days from runaway automation loops. Multi-agent setups on premium models routinely hit $600/month before anyone audits the config. The culprit is almost always the same: every sub-task — a heartbeat check, a JSON format, a ticket classification — is hitting a frontier model at full price.

The problem got more complicated in April. Anthropic blocked Claude Pro and Max subscribers from using their flat-rate plans with third-party tools like OpenClaw. OpenAI went the other direction. Sam Altman posted at 2am on May 2: “you can sign in to openclaw with your chatgpt account now and use your subscription there.” ChatGPT Plus and Pro now cover OpenClaw usage at a flat monthly rate via Codex OAuth — no per-token billing.

That is good news for a lot of users. But subscriptions have limits. ChatGPT Plus carries a 5-hour weekly usage quota. Pro users hit ceilings too on heavy workflows. The moment you are running serious automation — multi-agent pipelines, scheduled tasks, anything that fires dozens of requests an hour — you are back against a wall regardless of which subscription you are on.

The underlying issue is not which provider you use. It is that the wrong model is handling the wrong work.

wrong work.

Why the Bill Gets Out of Control

OpenClaw routes every task to whatever model you have set as default. That model does not know the difference between a request that requires genuine reasoning and one that is just formatting a JSON object or summarizing a paragraph. It handles both the same way: full context load, full inference, full cost.

Most OpenClaw workflows are 70-80% routine. Heartbeats, memory housekeeping, classification, extraction, formatting, cron jobs. None of it requires a frontier model. But if your default is Claude Sonnet or GPT-5.4, every one of those calls is priced like it does.

That is the frontier tax. You are paying for capability you are not using.

What We Built and Why It Works

Neurometric builds task-specific Small Language Models — purpose-built for narrow jobs. A model fine-tuned to classify support tickets does not need 1.7 trillion parameters to do that job well. A 7B model trained on thousands of legal extraction examples outperforms a frontier model on that specific task and runs at a fraction of the cost.

This is not a discount. It is architecture. Small models doing specific things are cheaper to run because they are smaller and more efficient at their job. The economics are structural, not promotional. That is why we can offer 100 million tokens per month for free and sustain unlimited token plans at prices that make sense.

The frontier model handles what it is actually good at: complex reasoning, multi-step planning, creative synthesis. Everything else routes to a specialist.

ClawPack is how this plugs into OpenClaw. It sits alongside your existing model as a standard provider. One model ID. Automatic routing. Your frontier model — or your ChatGPT subscription — stays in place for the work that needs it. ClawPack handles the rest.

The result is the same OpenClaw experience, with 60-90% fewer frontier model calls. If you are on a ChatGPT subscription, your quota goes further. If you are on API billing, your bill drops. Either way, you are not burning Opus-level compute to check whether your inbox has anything urgent.

The Stack We Recommend

We are partnering with hosting providers like LumaDock to make this easy for OpenClaw users to set up end to end.

LumaDock offers a purpose-built OpenClaw VPS template — you pick a plan, deploy a server, and OpenClaw is already installed and running when you SSH in. Starts at $1.99/month. Their complete OpenClaw guide and FAQ cover everything from first setup to production configuration.

Once OpenClaw is running, adding ClawPack takes two steps. Go to marketplace.neurometric.ai/clawpack, get your free API key, and copy the pre-populated install command the dashboard generates. Paste it into your server terminal. Then:

openclaw models set neurometric/clawpack

That’s it. ClawPack is live. For complex reasoning tasks, configure a fallback to your frontier model or ChatGPT subscription and OpenClaw escalates automatically when the task warrants it.

Free tier covers 100M tokens/month. No credit card. Unlimited plans available through your Neurometric account.

The Bigger Picture

The era of subsidized frontier inference for agentic workflows is coming to an end. Anthropic made that clear in April. OpenAI’s subscription path is a better deal for many users, but it is still a ceiling, not a solution.

The solution is not finding a cheaper frontier model. It is using the right model for the right task. Specialized intelligence delivered at the compute cost it actually requires. That is what we are building at Neurometric, and it is why the economics hold regardless of what the big providers do next.

Get started: LumaDock OpenClaw x ClawPack

Multiply that across 10,000 daily requests. Congratulations—you’ve built an inference money pit.

The AI engineering community collectively discovered that “just throw it at a frontier model” works great in demos and collapses in production. Agents enter retry death spirals. Context windows bloat with irrelevant RAG results. Sequential LLM calls stack latency until users abandon the workflow. The tools are extraordinarily powerful, and we are using them with the efficiency of a factory floor that nobody has ever walked with a stopwatch.

Lean Manufacturing fixed this problem for physical production 40 years ago. It’s time to apply the same discipline to inference.

Lean Inference Workflows are the systematic application of Lean/TPS (Toyota Production System) principles to the design of LLM-powered agent architectures. Not as metaphor—as engineering discipline.

The 7 Wastes of LLM Inference

Taiichi Ohno’s muda framework identified seven categories of waste in manufacturing. Each maps cleanly onto the failure modes we build into agents every day.

1. Overproduction — The Frontier Model Default

The most expensive waste is calling a 70B+ frontier model for tasks that don’t need it. Routing a support ticket to the right queue? That’s an 8B classification task. Extracting structured fields from a form submission? That’s a fine-tuned 3B model with a JSON schema. Summarizing a 500-word support thread? You don’t need GPT-4o.

The cost asymmetry is staggering. claude-sonnet runs ~3x the cost of haiku per token. GPT-4o runs ~10x the cost of GPT-4o-mini. When you reflexively reach for the frontier model on every step of a 15-step agent loop, you’re not just overspending—you’re adding latency at every node. If your task is a common one, you can even move to SLMs which are faster and two orders of magnitude cheaper.

Treat your agent’s model selection the same way a traffic engineer treats routing decisions—based on payload size, complexity score, and confidence threshold, not habit.

2. Inventory — RAG Bloat

Your vector database returns the top-20 chunks, and you shove all 20 into the context window “just in case.” That’s inventory waste: stockpiling inputs you probably won’t use, forcing the model to process them, inflating your input token count, and degrading retrieval precision in the process. More context isn’t better—it’s a longer assembly line with more defect opportunities.

Controlled inventory means retrieving fewer, better chunks via re-ranking (a cross-encoder pass over your top-k candidates), then truncating aggressively before injection.

3. Waiting — Sequential Blocking

Tool calls that could run in parallel are running in series. You need to fetch a user’s account history, check their subscription tier, and retrieve their recent support tickets. Instead of three parallel async calls, you have three sequential blocking calls: 300ms + 280ms + 310ms = 890ms of pure waiting.

async/await + parallel execution is the asyncio.gather or Promise.all call you should have made. In a multi-step agent DAG, every synchronous bottleneck is a latency tax.

4. Defects — Malformed Outputs and Retry Loops

An agent asks for a JSON tool call. The model returns Markdown-wrapped JSON with an extra trailing comma. Your parser throws. The orchestrator retries. The model hallucinates a different schema on the retry. You’re now three LLM calls deep on a task that should have been one.

Defects in inference are uniquely expensive because retries aren’t cheap reruns—they’re full-price LLM calls on an already-failed path. Structured outputs (OpenAI’s response_format, Anthropic’s tool use schemas, the instructor library for Python) eliminate this entirely by constraining output at the token-probability level.

5. Over-Processing — Unnecessary Chain-of-Thought

CoT is a forcing function for reasoning. It is not a default that belongs in every prompt. A routing classifier does not need to explain its reasoning to itself before assigning a ticket category. A field extractor does not need <thinking> tokens. Stripping CoT from non-reasoning tasks can cut your output token count by 40–60% on those steps—with zero quality loss.

Core Principles of Lean Inference

Just-In-Time Context: The Pull System

In Lean manufacturing, a pull system means downstream demand triggers upstream production—nothing gets built until it’s needed. JIT Context means your agent fetches context exactly when a step requires it, scoped precisely to what that step needs.

The anti-pattern is the “God Context”: a single massive system prompt that pre-loads everything the agent might need across all possible execution paths. You pay the full token tax on every call, even when 80% of that context is never accessed.

The Lean pattern:

• Semantic caching at the retrieval layer: if a semantically similar query was answered 30 seconds ago, return the cached embedding result, not a fresh DB round-trip.

• Re-ranking before injection: Run a cross-encoder (a fast, cheap model like ms-marco-MiniLM-L-6-v2) over your retrieved chunks before injecting them into the LLM context. Top-3 precision beats top-20 recall for most tasks.

• Step-scoped context: Each node in your agent DAG gets only the context its specific tool call requires. The summarization node doesn’t need the tool definitions. The routing node doesn’t need the document corpus.

Standardized Work: Deterministic Guardrails

Lean’s standardized work principle says that defined, repeatable processes reduce variation and defects. In agent architecture, this translates to: make your LLM do as little undirected reasoning as possible.

Logic that can be encoded deterministically should be. Your state machine transitions, routing rules, retry budgets, and tool call sequencing should live in code—not in a prompt asking the model to figure it out.

Tools like LangGraph let you encode agent control flow as an explicit graph: nodes are LLM calls or tool invocations, edges are conditional transitions, and the state machine is a first-class object you can inspect, test, and version-control. This is categorically different from a single ReAct loop where the model decides everything.

Structured outputs (via instructor, OpenAI’s strict: true JSON mode, or Anthropic’s tool schemas) are the manufacturing equivalent of a jig: they physically constrain the output to the valid shape, making defects structurally impossible rather than probabilistically unlikely.

Takt Time: The Latency Budget

Takt time in manufacturing is the maximum allowable time per unit to meet customer demand. In agent design, every workflow should have an explicit latency budget per step and per full execution path.

Define your takt time first. If your end-to-end SLA is 2 seconds and you have 6 agent steps, your average per-step budget is ~333ms. That budget forces architectural decisions:

• Can this step use a smaller model to hit the latency target?

• Should this step be parallelized?

• Does this step even need an LLM, or is a heuristic or cached result sufficient?

DAG decomposition is your primary tool here. A complex task that looks like a single LLM call is often a DAG of 4–6 smaller model calls that can execute in parallel, each with faster TTFT on a smaller model, combining to lower overall latency than the single big call.

Prompt Caching as Kanban

Anthropic’s prompt caching and OpenAI’s equivalent cache system are Kanban cards for inference: reusable, pre-positioned work items that don’t need to be re-manufactured from scratch.

Your system prompt, tool definitions, and static knowledge base content are the same across thousands of requests. Cache them. On Anthropic’s API, a cache hit on a 10,000-token system prompt costs 10% of the base input token price. Over millions of calls, this is not a micro-optimization—it’s a cost structure change.

Design your prompts with cache-friendly prefix ordering: static system prompt first, static tool definitions second, dynamic context last. Anything that changes per-request must come after anything that doesn’t.

Before and After: Repo Analysis Agent

Before (Naive Architecture)

A single ReAct loop. One GPT-4o call per step. Full repository context dumped into the window on every iteration. Tool definitions re-sent each time. Sequential file reads. No output validation. Average: 14 seconds, ~85,000 tokens, ~$1.20 per run.

After (Lean Architecture)

• A small router model (8B, fine-tuned) classifies the task type and selects the appropriate specialist pipeline — adds 80ms, saves 60% of downstream model costs

• Prompt caching on tool definitions and system context — 90% cache hit rate after warmup

• Parallel tool execution for file reads — 4 simultaneous reads instead of sequential

• Structured output enforcement via instructor — zero retry loops in 500-run benchmark

• Strict step budget: 6 steps max, with a fallback to human handoff at budget exhaustion

Result: 4.2 seconds average, ~18,000 tokens, ~$0.09 per run. Same output quality score on eval suite. 13x cost reduction. 3.3x latency improvement.

Conclusion

The next frontier of AI engineering isn’t a bigger context window or a more capable base model. It’s the discipline to use what we already have without waste.

Every unnecessary frontier model call, every bloated RAG context, every sequential blocking operation, every retry loop from malformed output—these are engineering failures, not model failures. We built them. We can fix them.

Lean Inference isn’t a philosophy—it’s a set of concrete architectural decisions you can make this sprint. Audit your agent’s token burn by step. Map your sequential calls. Add structured outputs. Right-size your models. Cache your static prompts.

Build leaner. Run faster. Spend less. Ship better agents.

Built on 5 Small Models

When we say Small Language Model, we mean models under 20B parameters. Some are under 3B. They’re small enough to run on a single GPU, fast enough to return structured output in seconds, and cheap enough that running five of them in parallel against thousands of files is a rounding error, not a line item.

The catch — and it’s a real one — is that they don’t know everything. A 3B model is not going to write your novel, debate Kant, or replace Opus. But it doesn’t need to. It needs to find SQL injections. Or flag a deprecated dependency. Or notice that a function has crept past 400 lines.

Narrow the task, and small wins.

The SLMs That Compose Owlpack

Owlpack runs five agents every night, each focused on a different domain:

• Hunter scans for security vulnerabilities — SQLi, SSRF, auth bypasses, leaked secrets, cross-referenced against live CVE feeds.

• Tracker hunts bugs — null references, race conditions, off-by-one errors, suspicious test coverage gaps.

• Keeper audits dependencies — outdated packages, breaking changes, deprecation notices, abandoned libraries.

• Mason identifies refactoring opportunities — duplication, coupling hotspots, long functions, outdated patterns.

• Scribe analyzes the codebase itself — churn, review latency, complexity drift, module ownership.

Each one is a specialist. Each one runs in parallel. Each one returns structured findings that get deduplicated, diffed against history, and ranked by severity before they land in your inbox by morning.

Try doing that with a single frontier model. The math falls apart. A nightly full-repo scan, across thousands of users, calling GPT-class inference five times per repo, would cost more than most teams pay for their entire dev tooling stack. That’s why no one was offering this service. The unit economics didn’t work.

With SLMs, they do.

The case for specialization

There’s a deeper reason we built it this way, and it goes beyond cost.

When you fine-tune a small model on a specific task — say, identifying CVE patterns in JavaScript — it gets better at that task than a frontier model trained to do everything. Specialists beat generalists when the task has a defined shape. Code review is a defined shape. Dependency auditing is a defined shape. Detecting a leaked AWS key is a very defined shape.

A frontier model brings a trillion-plus parameters of knowledge about Roman history and SQL injection patterns. Hunter brings the SQL injection patterns. For this job, that’s the better tool.

It also means the agents don’t drift. They don’t get creative. They don’t hallucinate a function name that doesn’t exist in your repo because they read about it in a blog post once. Narrowness is a feature.

Privacy as a byproduct

There’s a third benefit that falls out of using SLMs: a smaller blast radius. Owlpack clones your repository at scan time, runs the agents, and deletes the clone. We don’t need a 1.5T-parameter foundation model to read your code. We need five tight, task-specific models that do their job and forget. That architecture is easier to audit, easier to contain, and easier to deploy in environments where data exfiltration risk actually matters.

What this looks like in practice

You install the GitHub App. You pick which repos to scan. We clone them at your chosen time, run all five agents in parallel, delete the clone, and deliver a structured briefing by morning. If there’s nothing new, we don’t email you. If there’s a CVE in a transitive dependency, you’ll know before standup.

The whole pipeline costs a fraction of what a single frontier call would. That’s what passes through to pricing. That’s what makes the service viable.

The big-model era taught the industry to reach for the biggest hammer in the room. The next era is about picking the right one. Five small, sharp tools, running every night — that’s what Owlpack is. SLMs are why it works.

Try Owlpack Free For 7 Days. Or email us sales@neurometric.ai if you want to try a team plan.

But somewhere along the way, a quieter revolution started inside the enterprise. Companies stopped asking “what’s the most capable model?” and started asking “what’s the most useful one?”

Small Language Models — generally defined as models with fewer than 10 billion parameters — have emerged as the answer. They’re cheaper to run, fast enough for real-time workflows, and small enough to deploy on-prem or even on a laptop. That changes the economics and the privacy posture of AI in ways the headline-grabbing frontier models simply can’t match.

At Neurometric, we just crossed 3,000 active SLM users. You can download a fine tuned SLM from us without becoming a customer so, only a little over 2,200 of those users have applied for a key and are using Neurometric to host the SLM. But we’ve seen which pre-fine-tuned models are most popular from our SLM Marketplace, and we’ve talked to some enterprises directly who have asked for our help with larger scale SLM deployments.

If you’ve ever wondered what people use SLMs for in the real world - here are the top 5 things we’ve seen.

1. Summarization: Gist Generation at Scale

The single biggest workload across our user base is document summarization. Legal briefs, customer transcripts, research reports, internal wikis.

Why an SLM wins here: most summarization doesn’t need creative prose. It needs the gist — accurate, fast, and cheap enough to run across thousands of documents a day. Pushing a 50-page PDF through a frontier model with a giant context window costs real money. An SLM does the same job at a fraction of the latency and a tiny fraction of the cost. When you’re processing 10,000 documents a week, that math becomes existential.

2. Resume Screening: Extraction Without Hallucination

HR teams were one of the fastest verticals to adopt. The job isn’t to “write a beautiful candidate evaluation” — it’s to pull skills, years of experience, certifications, and seniority into a structured format.

That’s an extraction task, not a creativity task. And ironically, smaller, fine-tuned models are less prone to embellishment than their bigger cousins. Fewer parameters means fewer paths for the model to “fill in the blank” with something that wasn’t on the resume. For HR — where a hallucinated qualification is a compliance problem — the precision of an SLM is a feature, not a limitation.

3. Code Refactoring: Local, Low-Latency Suggestions

Developers using SLMs for code refactoring are the most popular group who download and use it themselves, rather than have us host it. The two reasons they seem to like the SLM approach:

• Latency. An autocomplete suggestion that arrives 800ms after you stop typing is useless. Local SLMs respond in tens of milliseconds.

• Security. Proprietary code never leaves the machine. For regulated industries and any company with a defensible codebase, that’s non-negotiable.

The frontier model can still review the architecture. The SLM handles the thousand small edits in between.

4. CRM Summary: Killing the Sales Busy Work

Sales reps don’t write call notes; they scribble them. The result is a CRM full of half-finished entries that nobody trusts.

Our users are using SLMs to convert messy voice memos and chat transcripts into structured CRM fields — next steps, objections, deal stage, sentiment. This is a textbook SLM workload: repetitive, schema-bound, and high-volume. A rep doing eight calls a day generates 40 summaries a week. Multiply by a 200-person sales org and you can see why running this on a frontier model is a non-starter.

5. Meeting Prep: Briefing Sheets Before the Call

The fifth pattern is what users are calling “pre-meeting intelligence.” Before a customer call, the SLM ingests prior emails, past call notes, account history, and recent product activity — then generates a one-page briefing sheet.

Speed is the metric here. The briefing needs to be ready in the 90 seconds between the previous meeting ending and the next one starting. Frontier models can’t hit that latency reliably. SLMs can.

So What Are People Actually Using SLMs For?

They’re not writing novels. They’re not passing the bar exam. They’re not solving open-ended research problems.

They’re doing the most common work tasks.

SLMs are workhorse models. Across our thousands of users, the pattern is consistent: repetitive, high-volume, structured tasks where you need roughly 90% accuracy at roughly 5% of the cost of a frontier LLM. That’s not a consolation prize — that’s the entire enterprise AI opportunity. Most business value isn’t locked behind PhD-level reasoning. It’s locked behind tasks too small and too numerous to justify a $0.10-per-call inference bill.

The Shift: From General Purpose to Specialized Agents

What our users are really previewing is the next phase of enterprise AI: a shift away from one giant general-purpose model handling everything, toward fleets of specialized agents each doing one thing exceptionally well.

The frontier model becomes the orchestrator. The SLMs do the work.

If you want to find where SLMs fit in your stack, don’t start with your hardest problems. Start with your most repetitive ones. Find the tasks your team does a thousand times a week that need to be correct, not clever. That’s the frontier now.

Bigger isn’t better. Specialized is.