AceBench measures exactly that: ~2,000 hand-annotated tasks drawn from ~4,500 synthetic APIs across eight domains (finance, health, travel, tech, entertainment, and more), published January 2025 and later accepted at EMNLP.

Three things make it worth a practitioner’s attention:

Grading is mechanical and brutal. Right function, right call count, right arguments — or it’s wrong. No partial credit, no AI judge. One bad field voids the form. Harsher than real life, but reproducible.

It comes in tiers. Normal (clear requests), Special (vague), Agent (multi-turn). We ran Normal, English only — this says nothing about ambiguity or agent loops.

It’s designed to be taken apart. Tasks isolate specific failure modes: value types, near-duplicate functions, mid-conversation drops. A leaderboard number says which model wins overall. A benchmark that comes apart says whether a cheap model is good enough for your work — usually the real question.

What we ran

Five models, 772 tasks each, 3,860 attempts, English Normal split:

• gemini-3.6-flash, gemini-3.1-flash-lite — Google’s hosted models

• gemma-4-12B-it, gemma-4-E4B-it, gemma-4-E2B-it — open-weight, self-served (big/small/very small)

Same harness and tasks for everyone. Tool calls return a bare acknowledgement — no feedback, one shot per call.

Key point: the open model on hardware we control tied Google’s hosted model, using about an eighth of the words.

The scoreboard

The top two are a real tie: 45 tasks the hosted model got and the open one missed, 35 the other way. That’s the signature of two equally capable models, not a better and a worse one. Run it again and the order could flip. Every other gap in the table is real — but the tie at the top is between a hosted frontier model and a 12B open model on a single GPU.

What it costs

Scoring the same isn’t interesting. Scoring the same this cheaply is.

ModelWords per correct answer*:

*tokens (~¾ word each), total output

The hosted model generates ~8x more text per correct answer than the 12B model it’s tied with — almost all invisible “thinking” (655,000 tokens across the run). The other four models did none of that: read, call, stop.

And the thinking isn’t buying anything. When the hosted model got a task wrong, it generated more than twice as much text as when it got one right. Extra effort here is a sign of being stuck, not a way out. On a benchmark where the winning move is two steps, there’s not much to think about.

Generated tokens are the expensive half of any pricing page, and what determines latency. So: same accuracy, a fraction of the tokens, and a deployment you own instead of rent.

The catch

Cheap on tokens ≠ cheap in practice:

The 12B model that ties Google’s best was 4x slower per answer than the API it ties, despite generating far less text. The smaller E4B, on the same GPU, hit 5.6s at ~91% of the top score. Note: these numbers are default vLLM, full BF16, on a single L40S — plenty of room to push both further with better hardware.

The smallest model is better than its score

gemma-4-E2B-it came last at 67%. Look at how it failed and half the gap disappears.

Its top mistake wasn’t picking the wrong tool — it was making the right call plus extra, already-completed ones from earlier in the conversation:

Called: ask about gift etiquette → send the gift → schedule the meeting Wanted: schedule the meeting

It knew the current turn. It just also re-did two things already done. Graded: zero.

72 of its 255 failures end with exactly the right call, buried under repeated history. Score only the current turn and it jumps from 67% to 76% — close to Google’s smaller model. That’s a prompt fix, not a bigger model — and it barely moves the other four (0-3 failures each of this kind).

Where bigger models still earn their keep

Three places:

Nested arguments. A tool wanting {"journey": {"from": "Shanghai", "to": "Hangzhou", "times": {...}}} trips up everyone — best models barely clear 60%, versus 85-96% for flatter arguments.

Several calls at once. Same tool, three times, different details: the hosted model pulls ahead ~5 points. Coordinating calls is harder than making one.

Long conversations — the sharpest split. Four of five models improve turn over turn as context narrows the options. The smallest model collapses:

That last column is what matters in production: a conversation only works if every turn lands. 87% per-turn becomes 74% overall.

Two models beat one

Since every model saw the same tasks, we can check what a pair covers:

Pairing the hosted model with the open one adds 4.5 points; pairing it with Google’s other model adds just 2.1. The two hosted models fail on the same tasks — the open one fails on different ones. If you can check an answer and retry, the self-served model is the better (and cheaper) second opinion.

So what do you do with this

If the task is “here’s what I want, here are the tools, go” — use a small self-hosted model. There’s no ambiguity for extra reasoning to resolve. A 12B model matches the frontier here; even 4B gets within ~90%.

If the work needs complex structures, coordinated multi-calls, or long conversations, the gap reopens — fastest for the smallest models.

Check what a scoreboard actually measures before trusting it: one category here was scoring a missing input, not the model, so we dropped it. Another score was understated nine points over a technicality about which calls “count” for a turn. Neither shows up in a leaderboard number — and both change what you’d buy.

The paper focuses on terminal agents — models given nothing but command-line access as a tool, which turns out to be enough to tackle an enormous range of tasks, from software engineering to security work to scheduling. Using this recipe, AI2 boosted a Qwen model’s Terminal-Bench score from roughly 25% to 31%, closing meaningful ground on Claude Haiku’s roughly 33%.

What made this one worth an episode wasn’t just the score bump. It was the design choices behind it.

Watch on YouTube: https://youtu.be/4U0gqv9sSn4?si=ejv04gktGbTOMJM8

Listen: https://tokenengineering.podbean.com

Paper: TMax: A recipe for terminal agents — https://arxiv.org/abs/2606.23321

Three models, two tool interfaces, 340 CRM tasks each — 2,040 agent rollouts in total.

The benchmark is CRMArena, run through the Harbor/dockworker harness. Every task asks a question about a read-only Salesforce org (”in May 2021, which state had the quickest case closures?”, “which knowledge article does this quote violate?”) and grades the submitted answer by exact match. Seventeen task categories, split evenly across a B2B and a B2C org.

The interesting design choice is that the same 340 questions are posed twice, behind two different tool surfaces:

Identical questions, identical ground truth — we verified this holds on every task where both suites produced a graded answer. So the two suites isolate one variable: does the model do better writing SQL, or picking the right pre-built tool and filling in its arguments?

The three models: gemini-3.6-flash and gemini-3.1-flash-lite (hosted), and gemma-4-E4B-it (open weights, served locally on vLLM). One is a small local model; two are hosted. Every trial used the same noshell agent scaffold, whose only way to answer is to call a terminal submit_answer tool.

The scoreboard

Read it and you’d conclude the ordering is obvious and the small model isn’t close: 39.7% against 67.6% is a 28-point gap. Every pairwise difference here is statistically significant (p ≤ 0.009).

Then you look at how the failures happen, and the picture inverts.

Half of the small model’s “failures” contain the right answer

Not every zero is a wrong answer. A trial scores zero if the answer file was never written at all — and the scaffold only writes it when the model calls submit_answer. Classifying all 2,040 trials by how they failed:

More than half of gemma’s SQL trials never submitted anything. And they didn’t crash or time out — in 171 of them the model finished its work, wrote the answer out in prose, and simply never called the tool. Something like:

The case_metrics call returned an average closure time of 4.2 days for CA, the lowest of any state. The answer is CA.

Graded: zero.

So we tested the obvious question — were those prose answers right? Ground truth isn’t recorded for ungraded trials, but because the same 340 tasks appear in both suites, we could recover the expected answer for every one of them and grep the final message for it.

Of gemma’s 250 prose non-submissions, 44–52% contained the correct answer. The range is the strict and loose reading of the same check: the loose count is any trial whose prose contains the expected value; the strict count additionally requires that the model wasn’t hedging across a list of candidates (≤1 other record ID mentioned). Both bound the same conclusion.

That reframes the scoreboard as a lower bound. Fixing one scaffold behaviour — get the model to call the tool — moves gemma to:

A model that looked like it scored 26% on SQL was doing work worth about 50%. Its measured number was roughly half its actual competence, and every point of that gap is a formatting bug.

The same correction barely moves the hosted models — gemini-3.6-flash has exactly zero prose non-submissions, and gemini-3.1-flash-lite has one. They always call the tool. What we were measuring, for a third of the benchmark, was instruction-following on the harness contract, not CRM reasoning.

There’s a cleaner way to see it. Restrict to trials that submitted anything, and the reasoning quality behind the scoreboard separates from the plumbing:

On answers it actually submits, the small local model is within two points of the hosted flash-lite model — a difference well inside the noise at this sample size. The 11-point headline gap between them is almost entirely tool-calling discipline.

What it costs

This is where the small model stops being a curiosity. Token totals are the whole run; the per-win column divides by correct answers, which is the number that matters if you’re paying for throughput.

Gemma on the tool API buys a correct answer for 45.4k tokens. gemini-3.6-flash needs 196k for the same thing — 4.3× more. On SQL it needs 598k, or 13× gemma’s best configuration.

The driver is visible in the reasoning-token column. gemini-3.6-flash spent 1.36M reasoning tokens on the API suite and 2.59M on SQL. The other two models spent none. That’s what the extra accuracy is bought with: 3.6-flash’s median trial emits 3,978 output tokens on the API suite against gemma’s 501 — an 8× difference in generated text per attempt.

And time to completion — median wall time of the agent execution phase alone, excluding container build and verification:

Here the small model does not win, and it’s worth being precise about why. gemma is 12.6s per attempt against 3.6-flash’s 30.0s — faster per attempt — but because it converts fewer attempts into correct answers, it lands at 73.8s per win versus 51.8s. And this column mixes model speed with serving infrastructure: a local vLLM instance against Google’s production endpoints. It is not a property of the models, and it’s the one metric here we’d throw out of a purchasing decision. Tokens are the honest efficiency measure; wall-clock is an artifact of where each model happened to be running.

One more pattern worth noting: every model spends more tokens on the trials it gets wrong — 1.5× to 2.1× its passing median. Failure is not cheap. Effort is a symptom of being lost, not a route out of it, which means the token cost of a wrong answer exceeds the token cost of a right one across the board.

Where each model actually fails

The three models have almost nothing in common in their failure profiles.

gemma-4-E4B-it — a formatting problem wearing a capability problem’s clothes. 250 of its failures are prose-instead-of-tool-call; roughly half contain the right answer. Its second tendency is over-caution: 50 trials answered None (”nothing matches”) when a real record existed. It abstains too readily and it won’t call the terminal tool. Both are addressable without touching the model.

gemini-3.1-flash-lite — quietly broken generations. 34 API trials (10%) ended with an empty completion: a single EOS token, no text and no tool call. Not prose, not a wrong answer — nothing at all. That signature points at serving or sampling rather than the prompt, and it’s the cheapest 10% anyone in this comparison could recover. Beyond that its losses are ordinary wrong answers (21–23%), the highest wrong-answer rate of the three.

gemini-3.6-flash — runs out of budget, and won’t say “none.” It never fails to submit when it finishes, but it frequently doesn’t finish: 60 SQL trials (18%) and 17 API trials (5%) hit the 25-step cap mid-work. Its failing trials burn a median 9,747 output tokens against 6,452 when passing — it iterates on queries that never land. Its other weakness is the mirror image of gemma’s: 63 trials invented a value where None was correct. The strongest model is the one most likely to manufacture an answer rather than concede there isn’t one.

That last contrast is the most useful thing here for anyone building on these models. The two abstention errors point in opposite directions — gemma says “none” when an answer exists, 3.6-flash asserts an answer when none does — so there is no single prompt that fixes both. It’s a calibration problem per model, not a benchmark-wide one.

The interface matters, and not the way you’d guess

Because the same questions appear behind both tool surfaces, we can ask whether SQL or semantic tools suit each model better as a paired comparison — the same task, two interfaces — using an exact McNemar test on the tasks where the two disagree.

For gemini-3.6-flash, the only model whose two runs had matched step budgets, the tool API wins clearly: +8.5 points, p < 0.001, with 47 tasks solved only through the API against 18 solved only through SQL.

But the aggregate hides something better. Across the models, between 65 and 133 of the 340 tasks flip outcome between the two interfaces. For gemini-3.1-flash-lite the headline rates are nearly identical (49.1% vs 50.9%) while 94 tasks flip — 44 solved only via SQL, 50 only via the API, with just 123 solved by both. The interface doesn’t change how many tasks it gets right; it changes which third of the benchmark it gets right. A model that looks interface-indifferent on the scoreboard is nothing of the kind, and running both interfaces and taking either success would score far above either alone.

Category detail shows where this bites. gemini-3.6-flash gets sales-amount-understanding right 60% of the time through semantic tools and 5% through SQL — but that collapse is not an inability to write the query. 18 of its 20 SQL attempts hit the step cap mid-work. Where a semantic tool answers “total order amount by owner in this window” in one call, SQL requires discovering the schema, joining line items to orders, and windowing on the right date column — and it ran out of budget getting there. lead-routing, by contrast, it solves 100% either way.

Not all 17 categories are the same benchmark

Treating CRMArena as one number hides the most actionable result in the run. Here is every category, worst-first, across all six runs (36 = gemini-3.6-flash, 31 = gemini-3.1-flash-lite, gm = gemma-4-E4B-it):

Group the 17 categories by what they actually ask for, and a pattern appears that the headline rates completely obscure:

On policy judgement, model scale buys almost nothing. Reading a chat transcript and deciding whether an agent breached policy, or whether a lead is qualified: gemini-3.6-flash manages 37.5%, gemma manages 32.5%. The small model is at 87% of the best model’s performance. Compare that to aggregate metrics, where the same small model reaches only 57% of the frontier score and the spread across runs hits 85 points on monthly-trend-analysis.

The reason is visible in the failure modes. On policy-violation-identification, half of gemini-3.6-flash’s API failures are missed_abstention — 8 trials asserting a violation where the correct answer was “none” — against 8 passes. gemma’s failures on the same category split 4 missed_abstention and 4 false_abstention. This category isn’t primarily testing reasoning; it’s testing whether a model will decline to answer. That’s a calibration property, and calibration doesn’t scale with capability the way multi-step numeric work does.

So the practical read is: the categories where a small model is nearly as good are the judgement ones, and the categories where it falls off a cliff are the multi-step aggregations. If your workload is “read this conversation and classify it,” a 4B local model is a serious candidate. If it’s “compute this metric across three joins and a date window,” it is not.

Two more things the table says:

invalid-config is the one genuine capability cliff. gemini-3.6-flash gets 50% on both interfaces; every other run scores 5–10%. A 45-point gap that survives both tool surfaces is the clearest evidence in this run of something the smaller models simply cannot do — matching a quote’s configuration against the knowledge article it violates. Notably it’s not a formatting artifact: gemma’s failures here are 8 wrong answers alongside 7 prose non-submissions, so even the recovered ceiling stays low.

quote-approval defeats everything. 8% pooled, topping out at 20%, and gemini-3.6-flash scores 5% and 0%. No model, no interface, no step budget helps. When the strongest model in a comparison scores 5% on a category, that’s usually a signal about the task or the grader rather than the models — and it’s the first thing we’d re-examine before drawing conclusions about the remaining headroom.

And where does the small model actually beat a hosted model head-to-head? On the tool API, gemma matches or beats gemini-3.1-flash-lite in 5 of 17 categories: policy-violation-identification (35% vs 15%, +20 points), wrong-stage-rectification (40% vs 35%), invalid-config (10% vs 5%), and ties on case-routing and lead-routing. Four of those five are judgement or routing tasks, not aggregations — the same pattern again.

What we’d take away

The headline number on an agent benchmark is a joint measurement of the model and the scaffold around it, and for small models the scaffold dominates. gemma-4-E4B-it looked like a 26% model on SQL. It was doing work worth about 50%, and the missing half was one unmade tool call. Anyone comparing a small open model against a hosted frontier model on a leaderboard number, without looking at the failure modes underneath it, is partly measuring which model was better at following the harness’s calling convention.

Small models win on the axis that gets left off the leaderboard. At 45.4k tokens per correct answer against 196k, gemma is 4.3× cheaper per unit of useful output than the model that beats it by 28 points — and it gets there with zero reasoning tokens against 1.36M. If your workload tolerates 50% task accuracy, or you can put a verifier behind it and retry, the small model is the better economic choice by a wide margin.

But the honest version of “small models can win” is narrower than the slogan. gemma did not beat gemini-3.6-flash in a single one of the 17 categories, on either interface. What it did was reach parity with a hosted flash-lite model on reasoning quality — 56.0% against 57.9% on submitted answers — at a fraction of the token cost, while running locally. That’s the claim the data supports: not that small models beat frontier models, but that the gap to the tier below frontier is mostly plumbing, and the plumbing is cheap to fix.

The category breakdown sharpens that into a selection rule. The gap is not uniform across work: on policy judgement the small model is at 87% of the frontier model, on multi-step aggregation it is at 57%, and on one category (invalid-config) it is nowhere near. “Can a small model do this?” has no general answer, but it has a reliable one per task family — and the families where the answer is yes are the judgement-shaped ones, which is the opposite of where most people assume a small model will struggle.

The fixes are unglamorous and they’re all in the harness: make the terminal tool call unmissable for models that write prose, investigate the empty completions, raise the step cap for models that iterate, and calibrate abstention per model rather than globally. None of them require a bigger model.

What Is TrustedRouter?

TrustedRouter is an inference gateway in the same family as OpenRouter: one OpenAI-compatible API, one base URL, and hundreds of models and routes behind it. You point your existing client at the gateway and swap model strings instead of rewriting integrations every time you want to try a different provider.

What makes it interesting is the trust posture. TrustedRouter keeps no record of what you route — no prompt or output logs, by design, and the gateway is attested and fails closed rather than silently degrading. It publishes a live status page and a continuously sampled performance leaderboard, so latency and uptime claims are measured rather than asserted. There’s also a documented migration path from OpenRouter for teams that want to move without a rewrite.

For anyone who has spent time explaining to a security team why prompts containing customer data are being logged by a third party, that combination is worth something.

What We’re Serving Today

We’re live with three small models on prepaid routes:

• ibm-granite/granite-4.1-8b — 32K context, $0.0525/1M prompt and $0.105/1M completion

• qwen/qwen3-vl-8b-instruct — 262K context, vision-capable

• qwen/qwen3-vl-8b-thinking — 32K context, reasoning traces

Early measured numbers on our routes: 863 ms p50 time-to-first-token on Granite and 100% uptime across the sampling window. Our listed policy note is accurate about what we do and don’t guarantee — upstream trace logging is disabled, prompts and completions are not written to observability or object storage, and we retain only aggregate request counts and token totals. That’s a no-store posture, not contractual ZDR, and we’d rather say so plainly than let people assume more than we’ve committed to.

Task-Specific Models Are Coming Next

Small general-purpose models are the entry point, not the thesis. The reason we build the way we do is that most production workloads aren’t “chat” — they’re a narrow, repetitive task where a tuned 8B model matches or beats a frontier model at a fraction of the cost.

Three task-specific models from our model marketplace we expect to list on TrustedRouter in the coming weeks:

1. Text-to-SQL — natural language to correct, schema-aware queries against a known database, tuned for join accuracy rather than conversational fluency.

2. Structured extraction — pulling typed JSON out of invoices, contracts, and claims documents against a supplied schema, with predictable failure behavior on missing fields.

3. Support intent classification — routing inbound tickets to the right queue and priority, the kind of high-volume, low-token task where per-call cost dominates everything else.

Each is graded against our proprietary task benchmark corpus, so you can see how a specialist performs on your task class before you route production traffic to it.

If you want to try the current models, grab a key at TrustedRouter and use neurometric as the provider. If you have a task class you’d like us to build for, tell us — that’s how the roadmap gets set.

To cut through the hype, organizations must fundamentally rethink their deployment strategies. In our experience, the most critical decision a technical leader makes isn’t which specific foundational model to choose, but rather the strategic posture they take toward the underlying business task. We have observed that enterprise AI strategy ultimately boils down to a fundamental set of decisions. There are usually four options on the table, and they require a level of candor that is often missing from vendor pitches and internal strategy meetings.

The Four Options

When evaluating a new AI capability or use case, technology leaders must choose between four distinct paths: Buy, Build, Route, or Wait.

Buy is the right answer much more often than technical leaders want it to be. Engineers are naturally wired to create, and the allure of constructing a bespoke AI system is incredibly strong. However, if a vendor has already solved a generic enterprise problem—like drafting marketing copy, summarizing meeting notes, or triaging customer support tickets—purchasing that solution is almost always the superior economic choice.

Build is the right answer only when the task at hand is the core business. If the AI system is going to directly drive your competitive advantage in the marketplace, you cannot outsource it.

Route is the correct posture when the task is high-volume and the models are fungible. If you are processing millions of identical queries where the subtle nuances of a massive frontier model are unnecessary, routing queries to the most efficient model available is the only way to scale without destroying your margins.

Finally, there is Wait. Wait is the honest default. It is the right decision more often than anyone wants to admit, and yet it is almost never proposed in a strategy meeting. There is immense career risk in telling a CEO to wait on AI. But for highly volatile use cases, or problems where the underlying foundational models are currently struggling but rapidly improving, waiting six months for the ecosystem to mature is often vastly superior to burning capital on a brittle, premature V1.

The “Build Test”

If you are leaning toward building a custom AI solution, you must subject your proposal to the “Build Test.” Building bespoke AI is an expensive, resource-intensive endeavor that requires long-term commitment. At Neurometric, we advise clients to build only if they can definitively answer “yes” to all three of the following criteria:

First, is the task core to your differentiation? The AI must do something that separates you from your competitors. If it is merely an operational efficiency that every other company in your sector will eventually adopt, it fails this test.

Second, do you possess proprietary data or a unique workflow that a vendor structurally cannot access? You need an unfair advantage. If you are building a model using the exact same public datasets and standard enterprise tools as the major SaaS vendors, they will eventually commoditize your creation. You must have a moat built on data or processes that are uniquely yours.

Third, can you staff the maintenance of this system for the next three years? Building the model is merely the starting line. Models drift, APIs change, underlying data distributions shift, and security vulnerabilities emerge. You are not just funding a build phase; you are funding a permanent product team.

If you meet two out of these three criteria, it is not a “maybe.” Two out of three means you default to Buy. The economics of maintaining a sub-scale, non-differentiated AI system will slowly drain your engineering resources.

Model Selection: The Frontier Premium

Once you have decided how to acquire the capability, you must choose the right engine. The market is currently bifurcated between massive “frontier” models and smaller, specialized, or distilled models.

Frontier models are the bleeding-edge giants of the industry. They possess incredible reasoning capabilities and vast world knowledge. You should reserve these models strictly for tasks requiring deep judgment, handling high ambiguity, and managing low-volume, high-stakes scenarios. If an AI is reviewing a complex legal contract for a multi-million dollar merger, you want the frontier model.

Conversely, small, specialized, or distilled models are the workhorses of the modern enterprise. These should be deployed for high-volume, well-specified, and latency-sensitive tasks. When you are processing tens of thousands of basic data extraction requests per hour, you do not need an AI that can write a sonnet or pass the bar exam.

The divergence in economics here is huge, often separated by one to two orders of magnitude. A distilled model can easily be 10x to 100x cheaper per token than a frontier model, while returning responses in a fraction of the time. Yet, we routinely see organizations using frontier models for absolutely everything simply because it requires only one API integration. They are paying a 30× premium for the privilege of architectural laziness.

Routing as an Operating Discipline

To capture the economic benefits of smaller models without sacrificing quality, organizations must adopt routing as a core operating discipline.

The biggest mistake teams make is routing at the application level—deciding that an entire application will use just one model. Instead, you must route per task. A single customer service application might use a cheap, fast model to identify the language of an incoming ticket, a specialized model to extract the customer’s account number, and only invoke a frontier model if the ticket requires a complex, nuanced apology for a service failure.

Implementing this requires establishing a strict quality floor for every specific task. You determine the minimum acceptable accuracy, and then you dynamically route the workload to the absolute cheapest model that clears that floor. Because the open-source and proprietary model landscapes are evolving at a breakneck pace, this is not a set-it-and-forget-it architecture. You must re-benchmark your routing logic monthly. The model that was the most cost-effective in January might be entirely obsolete by April.

Vendor Risk and AI Sovereignty

Finally, organizations must wake up to the reality of vendor risk in the AI space. It is real, it is severe, and it is currently vastly underweighted in enterprise risk assessments.

When you build a product highly dependent on a third-party API, you are at the mercy of their roadmap. Deprecation schedules are often brutally short, forcing expensive emergency migrations when a vendor decides to sunset a specific model version. Price changes can occur overnight, destroying the unit economics of your application. Rate limits can be abruptly tightened, throttling your application’s ability to scale during a surge in user demand.

Furthermore, data privacy terms are a constantly shifting target. An update to a vendor’s terms of service could suddenly allow them to use your enterprise data to train their next generation of models, violating your internal compliance policies.

Most insidiously, a model can change its behavior right underneath you without any version bump. Vendors continuously tweak, align, and “improve” their models behind the scenes. A prompt that reliably output perfect JSON formatting on Tuesday might suddenly start wrapping its output in conversational pleasantries on Thursday, breaking your entire data pipeline.

To mitigate this, technology leaders must push back during procurement. Do not accept standard click-wrap agreements for critical AI infrastructure. You must explicitly contract for extended notice periods regarding deprecations, strict rate limit guarantees, immutable data terms, and rigid version control to ensure the model you test is the exact model you run in production.

The AI ecosystem is moving incredibly fast, but the fundamental laws of enterprise software engineering and business economics have not been suspended. By rigorously applying the Build Test, rightsizing your model selection, establishing a disciplined routing architecture, and aggressively managing vendor risk, you can navigate this landscape successfully. The goal isn’t to deploy AI the fastest; the goal is to deploy it in a way that actually works for your business.

If you want to shield yourself from the chaos of the model selection and routing ecosystem, Neurometric’s core platform evaluates, chooses, and routes models automatically. Try it for free.

This should feel familiar. When companies moved to the cloud, an entire layer of cost management and optimization tooling emerged around it — because once compute became a metered utility, optimizing that meter became a discipline with real dollars attached. AI inference is following the same path, except the stakes are arriving faster. A production AI workload at enterprise scale — say, a million model calls a day — can swing by millions of dollars a year based on decisions most teams made once, early, and never revisited.

Token engineering is the discipline that fixes this. A token engineering platform is the system that operationalizes it.

What Is Token Engineering?

Token engineering is the practice of treating tokens as an engineered resource: measured, benchmarked, routed, and continuously optimized. It’s a systems discipline, not a procurement exercise.

The common misconception is that token engineering means “use a cheaper model.” It doesn’t. It means optimizing every AI workload across three dimensions simultaneously: cost, speed, and reliability. Sometimes the right answer is a smaller, cheaper model. Sometimes it’s a faster one. Sometimes it’s the frontier model, but with a compressed prompt and an aggressive caching layer in front of it. The point is that the answer is different for every task, and it changes constantly.

Three forces make this urgent right now. First, model proliferation: frontier LLMs, open-weight models, and small language models (SLMs) now number in the hundreds, with meaningful new releases every month. Second, price variance: the cost of completing the same task can vary by 100x or more depending on which model, technique, and hardware you choose. Third, the capability crossover: for a growing share of enterprise tasks, purpose-built SLMs now match or beat frontier models at a fraction of the cost.

Meanwhile, most teams pick a model at the start of a project, hardcode it, and move on. Every month that decision goes unexamined, the gap between what they pay and what they should pay gets wider.

The Core Components of a Token Engineering Platform

Managing this problem manually doesn’t scale. A token engineering platform automates it. Here’s what a complete platform looks like — and how the Neurometric platform implements each piece.

1. Workload Evaluation and Benchmarking

You cannot optimize what you cannot measure, and public leaderboards won’t save you. Generic benchmarks tell you how models perform on academic tasks, not on your workloads — your customer service intents, your document extraction formats, your code review standards.

A token engineering platform continuously evaluates models and token engineering techniques against your actual tasks. When a new model ships, you shouldn’t have to wonder whether it’s better for your use case; the platform should tell you, with graded evidence. Neurometric’s Harbor evaluation infrastructure does exactly this, with more than 15,000 graded tasks and thousands of benchmark runs powering every recommendation the platform makes.

2. Task-Level Routing

Evaluation tells you which model is best for which task. Routing acts on it — automatically, per request, in production.

This is the decision engine at the heart of the platform. Most applications don’t have one workload; they have dozens of distinct tasks hiding inside a single product, each with different cost, latency, and quality requirements. Routing at the application level means paying frontier prices for tasks a model one-tenth the cost handles perfectly. Routing at the task level means every request goes to the cheapest model that meets its quality bar. Our Task Level Router keeps this workload-to-model mapping current as models, prices, and your traffic all change — so the routing decision you’d make today doesn’t quietly decay into the wrong decision six months from now.

3. Automated SLM Creation and Fine-Tuning

Sometimes no existing model sits at the right point on the cost/quality frontier for your task. The frontier model is overkill and overpriced; the small models miss your quality bar. Historically, the answer was a fine-tuning project: hire ML engineers, build a data pipeline, spend a quarter.

A token engineering platform makes this a feature, not a project. When evaluation data shows that a task is a candidate for a purpose-built model, the platform can distill and fine-tune an SLM automatically, validate it against your benchmarks, and slot it into the routing layer. The economics are hard to ignore: a task-specific SLM can run at 1/50th the cost of a frontier model while matching its accuracy on that narrow task. Our Auto-SLM Creator turns what used to be a specialized data science effort into a platform capability.

4. Hardware and Deployment Optimization

Which model you run is only half the equation. Where you run it matters just as much.

The same open-weight model has wildly different economics depending on whether you consume it through an API, self-host it on dedicated GPUs, run it quantized on cheaper hardware, or batch it for throughput over latency. At scale, the API-versus-self-hosting decision alone can be worth seven figures annually — and the right answer flips as your volume grows and hardware prices move. A token engineering platform models these deployment economics continuously and recommends the optimal placement for each model in your stack, so the decision gets revisited by software instead of forgotten by people.

5. Prompt Compression, Rewriting, and Caching

The cheapest token is the one you never send.

Before a request ever reaches a model, there are three opportunities to shrink it: compress the prompt to strip redundancy, rewrite it for token efficiency without losing intent, and cache aggressively — both exact-match and semantic — so repeated or near-repeated requests never hit the model at all. Individually these look like small percentage gains. At enterprise volume they compound into real money: at a million calls a day, a 20% reduction in tokens per call is not a rounding error, it’s a budget line. The platform applies these techniques automatically and only where evaluation shows they don’t degrade quality.

6. Cost Attribution

None of the above works as a one-time exercise. Optimization needs a feedback loop, and that loop starts with knowing exactly where your tokens go.

A token engineering platform attributes token spend at the level your business actually thinks in: per task, per team, per product feature, per customer. This is the FinOps layer for AI. It’s what lets you answer questions like “what does our IVR summarization actually cost per call?” or “which feature’s AI spend grew 40% last quarter, and was that traffic or inefficiency?” Without attribution, every optimization is a guess. With it, the platform can show you — in dollars — what each routing decision, SLM deployment, and caching policy is saving.

7. Governance, Budgets, and Reliability Controls

Finally, the layer that makes all of this safe to run in production: spend limits per team or application, fallback chains when a provider degrades, SLA enforcement on latency and quality, and defined degradation policies for when things go wrong.

This is the difference between a developer tool and an enterprise platform. Your finance team gets budget enforcement. Your platform team gets reliability guarantees. Your compliance team gets an audit trail of which model handled which request and why.

How It All Fits Together

These aren’t seven point solutions bolted together. They’re a flywheel: benchmark → route → optimize → attribute → re-benchmark.

Evaluation data drives routing decisions. Routing data reveals which tasks are candidates for purpose-built SLMs. Deployment optimization changes the economics that feed back into routing. Cost attribution measures the impact of all of it and surfaces the next opportunity. Every component makes the others smarter, and the loop runs continuously — which matters, because the model landscape changes monthly and your traffic changes daily.

This is also the honest answer to the build-versus-buy question. Any strong engineering team can build one of these components. Very few can build all seven, and almost none can afford to maintain all seven against a model market that reprices and re-ranks itself every few weeks. Nobody builds their own cloud cost management platform anymore. The same logic is arriving for tokens.

In Summary

AI inference is becoming one of the largest new line items in enterprise technology budgets, and it’s currently one of the least managed. Token engineering is the discipline that closes that gap. A token engineering platform — evaluation, routing, automated SLM creation, deployment optimization, prompt optimization, cost attribution, and governance — is how you run it at scale.

If you’re spending real money on model calls and can’t say with confidence that each task is running on the right model, at the right price, on the right hardware, that’s the gap Neurometric was built to close. [Get in touch to benchmark your workloads -sales@neurometric.ai]

• The tokenminning.com origin story — (As seen in the NYT) The Tokenminning Manifesto by Neurometric is gaining traction, with coverage in the NYT. As enterprises prove out AI use cases while blowing through their token budgets, token engineering is becoming a first-order concern.

• Specialized models vs. frontier models — Small, task-specific models can almost always beat a frontier model on a given task, even at a fraction of the parameter count. The catch: that specialized model can only do that one thing well. This is a feature, not a bug.

• The new Token Engineering Platform — Neurometric's answer to a fragmented tooling landscape. It brings SLM fine-tuning, distillation, and token caching into a single system so teams get one view across their entire model stack.

•  COGS vs. OPEX — how Neurometric segments its customer base, and why companies whose AI spend hits gross margin (not just headcount-adjacent budgets) are the ones moving fastest toward token efficiency.

Listen to the full episode: https://tokenengineering.podbean.com/

Watch on YouTube: https://youtu.be/JHFeraXq3RU?si=fzzRcEBuPo3vbF6A

**Resources mentioned:**

• Tokenminning Manifesto: tokenminning.com

• Neurometric AI: neurometric.ai

• NYT Article: https://www.nytimes.com/2026/06/18/technology/ai-token-minimizing.html

They all started with one large frontier model, and as their inference costs grew, they started to peel off their high volume simpler workloads and set up “task specific endpoints.” Workloads like customer sentiment analysis of support tickets, named entity extraction from documents, email summarization, these don’t need frontier intelligence. By setting up an endpoint with a small model for just those workflows, they saw lower costs and faster latency.

The problem is, there are thousands of models out there, and hundreds of techniques to optimize them - from where you run the model to prompting and token management techniques. And that is all on top of the complicated issue of which model works best for your use case in the first place.

Today we are announcing that Neurometric has pulled all of these tools together in a single platform that makes it easy to make token engineering decisions. With our platform you can:

• Monitor and measure AI workloads

• Optimize workloads for cost or latency

• Build custom SLMs for workloads that benefit from that approach

• Evaluate and test many models, techniques, and hosting platforms

Our customers typically see an 80% drop in inference charges and a 4x improvement in latency using the Neurometric platform.

If you want take your AI optimization to the next level, hire a token engineer, and use the Neurometric platform. Reach out to us at sales@neurometric.ai if you want to learn more.

Rob May, our CEO, told the Times: CEOs who couldn’t measure AI savviness defaulted to “who’s using the most tokens.” Volume over efficiency was never going to hold once the bills landed.

Uber blew through its full-year AI budget in four months. Meta is capping usage after an “exponential increase” in costs. AT&T’s shows companies can save up to 90% by using less powerful models for most tasks.

This is the insight we built Neurometric on. 

Read the full article: https://www.nytimes.com/2026/06/18/technology/ai-token-minimizing.html

That’s the central argument of the Pioneer Agent paper from Fastino Labs — and it maps pretty cleanly onto what we’ve been building at Neurometric.

In this episode of Inference Time Tactics, Director of AI Research Yash Sharma breaks it down with co-founder Calvin Cooper: what Pioneer Agent actually is, how it uses Claude Sonnet as an ML engineer in a box, and what the results tell us about where SLMs go next.

Key topics:

— Cold start data curation with zero customer traces

— Why naive retraining fails with noisy production data (and how an agentic loop fixes it)

— 83-point benchmark improvements — and why the 1.6-point cases matter just as much

— Regression rollback, hyperparameter search, and the heuristics baked into the system

— Where we think SLMs can go beyond “simple tasks”

Watch the full episode on YouTube: https://youtu.be/PiIrywMGsAA?si=gs_3nDwgS7QbvwQG

Or listen in on any platform: https://inferencetimetactics.podstream.com

Subscribe

Currently, enterprise infrastructure spend is run with zero discipline. We regularly see frontier models costing $5 to $25 per million tokens assigned to commodity work that $0.10 to $0.50 small models handle easily on benchmarks. This creates a 50x to 250x price spread, paid on every request, every single day. To survive this efficiency gap, a new engineering discipline has emerged: Token Engineering. It is the systematic optimization of which model runs which task—at what size, with what prompt structure, and at what cost.

The Agentic Scale Problem

The root of this cost crisis is that our foundational software design patterns have completely changed. Over 70% of routed inference traffic now comes from autonomous agents and CLI tools—not human chat interfaces. While a standard chat turn consumes just a few thousand tokens, a single agentic task regularly chews through 100K to 1M tokens as it loops, reasons, and self-corrects.

This volume shift has caused platform-wide token volume to explode by more than 10x in roughly a year, with 16 to 18 trillion tokens per week routed on OpenRouter alone. When workloads scale to this magnitude, token waste becomes an engineering failure, not a model failure.

The Five Pillars of Token Waste

When auditing modern agent pipelines, token waste typically boils down to five core engineering oversights:

• Oversized Models: Allocating expensive frontier models to simple extraction, classification, and formatting tasks that small models win on benchmarks.

• Prompt Bloat: Deploying unversioned, unmeasured prompts that carry thousands of redundant tokens into every single call.

• No Caching Strategy: Re-sending massive chunks of static context on every request instead of caching it at 10% to 20% of the standard price.

• Sequential Sprawl: Running agent steps serially with full context when steps could be decomposed, parallelized, and right-sized across lean endpoints.

• Blind Retries: Retrying unexpected failures on the same expensive model with no confidence scoring or cheaper fallback path.

Why Human Optimization Fails

Fixing these leaks manually is a noble goal, but token engineering simply does not scale as a human job. First, we face continuous Model Churn. Major model releases land weekly. Look at Gemma 4: it went from non-existent to routing 240 billion tokens per week—roughly one-third of all small-model traffic on OpenRouter—in a mere 70 days. No human team can re-benchmark thousands of model-by-task combinations on that clock.

Second, we hit Price Churn. Providers reprice continuously; caching, batching, and changing infrastructure economics shift the optimal choice even when models remain static. The half-life of an optimization is measured in weeks; a hand-tuned pipeline is stale before the next sprint ends. The engineer has to be automated.

The FinOps Parallel

Every era of computing waste eventually forces the transition from a heroic manual task into an automated platform practice. When web applications struggled with uptime, we turned manual on-call duties into Site Reliability Engineering (SRE). When cloud spend spiraled out of control, cloud waste created FinOps platforms that natively paid for themselves. Today, token spend is the fastest-growing cost line in software, and it demands its own dedicated engineering platform.

As LLM infrastructure commoditizes, value is rapidly migrating away from foundational providers and straight to the orchestration layer—the core intelligence about which intelligence to use. By decoupling your software workflows from rigid API keys and transitioning to dynamic task endpoints, you can let automation drive delivery costs down via leaner prompts, continuous model re-matching, and purpose-built small models. In this new landscape, remember: every standard inference vendor makes more money when your system wastes tokens. True operational maturity belongs to the engineering teams who build systems that actively eliminate the bleed.

If you want help, Neurometric offers a platform ideal for token engineering. Contact us to chat more about it.

The Numbers

5,791 benchmark runs. 15,750 individual graded tasks.

That’s a meaningful dataset. But the headline isn’t a model score — it’s that the framework couldn’t reliably finish.

53% of runs errored out. An errored run produces nothing: zero tasks completed, no usable result. More than half our compute returned empty-handed, not because the models failed, but because the harness crashed.

This matters beyond the obvious waste. When your error rate crosses 50%, you’re no longer sampling performance — you’re sampling infrastructure luck.

The Grading Problem

The errors weren’t just crashes. They were silent distortions inside the results that did come back.

In 765 cases, the agent produced the exactly correct answer — and Harbor logged it as an error anyway. That’s one in five of all correct results misclassified as failures. A grading system that can’t recognize its own correct answers isn’t grading. It’s noise with a spreadsheet attached.

The implication: any model rankings produced under these conditions would systematically understate performance, with the degree of understatement varying arbitrarily across runs. You can’t normalize your way out of that.

Dataset Coverage

Harbor ships with 80 datasets. We got 42 of them to run at all — just over half. Of those 42, 17 never produced a single scored result. That’s 17 datasets that opened, ran, and returned nothing actionable.

Effective coverage: roughly 31% of the catalog produced data you could learn from. You cannot benchmark across a catalog when two-thirds of it is structurally inoperable.

The Hello-World Test

The clearest evidence is the simplest task.

Harbor’s hello-world benchmark asks the agent to create a file containing "Hello, world!" That’s it. No reasoning. No retrieval. No multi-step planning. Just: write a file.

We ran it across 1,526 agent+model combinations. 645 of them — 42% — never completed it cleanly even once.

In one case, GPT-5.4 wrote the file perfectly. The run still errored.

The intelligence was never the bottleneck. The harness was.

What This Means for AI Benchmarking

Benchmark infrastructure is load-bearing. It doesn’t just measure performance — it defines what counts as performance. When the harness fails silently, misclassifies correct answers, and drops the majority of its own datasets, the output isn’t a measurement. It’s a corrupted signal that looks like data.

The field has spent enormous energy debating which benchmarks best capture model capability. That’s the right debate to have — once the execution layer can be trusted. Our results suggest the execution layer deserves a lot more scrutiny than it typically gets.

A model that writes the correct answer deserves to have that answer counted. That’s table stakes for any evaluation system. When it isn’t met, the benchmark isn’t measuring models. It’s measuring the framework.

The OpenClaw cost problem is not a secret. Users are posting about it openly. One tech blogger documented $3,600 in a single month. Others report $200 days from runaway automation loops. Multi-agent setups on premium models routinely hit $600/month before anyone audits the config. The culprit is almost always the same: every sub-task — a heartbeat check, a JSON format, a ticket classification — is hitting a frontier model at full price.

The problem got more complicated in April. Anthropic blocked Claude Pro and Max subscribers from using their flat-rate plans with third-party tools like OpenClaw. OpenAI went the other direction. Sam Altman posted at 2am on May 2: “you can sign in to openclaw with your chatgpt account now and use your subscription there.” ChatGPT Plus and Pro now cover OpenClaw usage at a flat monthly rate via Codex OAuth — no per-token billing.

That is good news for a lot of users. But subscriptions have limits. ChatGPT Plus carries a 5-hour weekly usage quota. Pro users hit ceilings too on heavy workflows. The moment you are running serious automation — multi-agent pipelines, scheduled tasks, anything that fires dozens of requests an hour — you are back against a wall regardless of which subscription you are on.

The underlying issue is not which provider you use. It is that the wrong model is handling the wrong work.

wrong work.

Why the Bill Gets Out of Control

OpenClaw routes every task to whatever model you have set as default. That model does not know the difference between a request that requires genuine reasoning and one that is just formatting a JSON object or summarizing a paragraph. It handles both the same way: full context load, full inference, full cost.

Most OpenClaw workflows are 70-80% routine. Heartbeats, memory housekeeping, classification, extraction, formatting, cron jobs. None of it requires a frontier model. But if your default is Claude Sonnet or GPT-5.4, every one of those calls is priced like it does.

That is the frontier tax. You are paying for capability you are not using.

What We Built and Why It Works

Neurometric builds task-specific Small Language Models — purpose-built for narrow jobs. A model fine-tuned to classify support tickets does not need 1.7 trillion parameters to do that job well. A 7B model trained on thousands of legal extraction examples outperforms a frontier model on that specific task and runs at a fraction of the cost.

This is not a discount. It is architecture. Small models doing specific things are cheaper to run because they are smaller and more efficient at their job. The economics are structural, not promotional. That is why we can offer 100 million tokens per month for free and sustain unlimited token plans at prices that make sense.

The frontier model handles what it is actually good at: complex reasoning, multi-step planning, creative synthesis. Everything else routes to a specialist.

ClawPack is how this plugs into OpenClaw. It sits alongside your existing model as a standard provider. One model ID. Automatic routing. Your frontier model — or your ChatGPT subscription — stays in place for the work that needs it. ClawPack handles the rest.

The result is the same OpenClaw experience, with 60-90% fewer frontier model calls. If you are on a ChatGPT subscription, your quota goes further. If you are on API billing, your bill drops. Either way, you are not burning Opus-level compute to check whether your inbox has anything urgent.

The Stack We Recommend

We are partnering with hosting providers like LumaDock to make this easy for OpenClaw users to set up end to end.

LumaDock offers a purpose-built OpenClaw VPS template — you pick a plan, deploy a server, and OpenClaw is already installed and running when you SSH in. Starts at $1.99/month. Their complete OpenClaw guide and FAQ cover everything from first setup to production configuration.

Once OpenClaw is running, adding ClawPack takes two steps. Go to marketplace.neurometric.ai/clawpack, get your free API key, and copy the pre-populated install command the dashboard generates. Paste it into your server terminal. Then:

openclaw models set neurometric/clawpack

That’s it. ClawPack is live. For complex reasoning tasks, configure a fallback to your frontier model or ChatGPT subscription and OpenClaw escalates automatically when the task warrants it.

Free tier covers 100M tokens/month. No credit card. Unlimited plans available through your Neurometric account.

The Bigger Picture

The era of subsidized frontier inference for agentic workflows is coming to an end. Anthropic made that clear in April. OpenAI’s subscription path is a better deal for many users, but it is still a ceiling, not a solution.

The solution is not finding a cheaper frontier model. It is using the right model for the right task. Specialized intelligence delivered at the compute cost it actually requires. That is what we are building at Neurometric, and it is why the economics hold regardless of what the big providers do next.

Get started: LumaDock OpenClaw x ClawPack

Multiply that across 10,000 daily requests. Congratulations—you’ve built an inference money pit.

The AI engineering community collectively discovered that “just throw it at a frontier model” works great in demos and collapses in production. Agents enter retry death spirals. Context windows bloat with irrelevant RAG results. Sequential LLM calls stack latency until users abandon the workflow. The tools are extraordinarily powerful, and we are using them with the efficiency of a factory floor that nobody has ever walked with a stopwatch.

Lean Manufacturing fixed this problem for physical production 40 years ago. It’s time to apply the same discipline to inference.

Lean Inference Workflows are the systematic application of Lean/TPS (Toyota Production System) principles to the design of LLM-powered agent architectures. Not as metaphor—as engineering discipline.

The 7 Wastes of LLM Inference

Taiichi Ohno’s muda framework identified seven categories of waste in manufacturing. Each maps cleanly onto the failure modes we build into agents every day.

1. Overproduction — The Frontier Model Default

The most expensive waste is calling a 70B+ frontier model for tasks that don’t need it. Routing a support ticket to the right queue? That’s an 8B classification task. Extracting structured fields from a form submission? That’s a fine-tuned 3B model with a JSON schema. Summarizing a 500-word support thread? You don’t need GPT-4o.

The cost asymmetry is staggering. claude-sonnet runs ~3x the cost of haiku per token. GPT-4o runs ~10x the cost of GPT-4o-mini. When you reflexively reach for the frontier model on every step of a 15-step agent loop, you’re not just overspending—you’re adding latency at every node. If your task is a common one, you can even move to SLMs which are faster and two orders of magnitude cheaper.

Treat your agent’s model selection the same way a traffic engineer treats routing decisions—based on payload size, complexity score, and confidence threshold, not habit.

2. Inventory — RAG Bloat

Your vector database returns the top-20 chunks, and you shove all 20 into the context window “just in case.” That’s inventory waste: stockpiling inputs you probably won’t use, forcing the model to process them, inflating your input token count, and degrading retrieval precision in the process. More context isn’t better—it’s a longer assembly line with more defect opportunities.

Controlled inventory means retrieving fewer, better chunks via re-ranking (a cross-encoder pass over your top-k candidates), then truncating aggressively before injection.

3. Waiting — Sequential Blocking

Tool calls that could run in parallel are running in series. You need to fetch a user’s account history, check their subscription tier, and retrieve their recent support tickets. Instead of three parallel async calls, you have three sequential blocking calls: 300ms + 280ms + 310ms = 890ms of pure waiting.

async/await + parallel execution is the asyncio.gather or Promise.all call you should have made. In a multi-step agent DAG, every synchronous bottleneck is a latency tax.

4. Defects — Malformed Outputs and Retry Loops

An agent asks for a JSON tool call. The model returns Markdown-wrapped JSON with an extra trailing comma. Your parser throws. The orchestrator retries. The model hallucinates a different schema on the retry. You’re now three LLM calls deep on a task that should have been one.

Defects in inference are uniquely expensive because retries aren’t cheap reruns—they’re full-price LLM calls on an already-failed path. Structured outputs (OpenAI’s response_format, Anthropic’s tool use schemas, the instructor library for Python) eliminate this entirely by constraining output at the token-probability level.

5. Over-Processing — Unnecessary Chain-of-Thought

CoT is a forcing function for reasoning. It is not a default that belongs in every prompt. A routing classifier does not need to explain its reasoning to itself before assigning a ticket category. A field extractor does not need <thinking> tokens. Stripping CoT from non-reasoning tasks can cut your output token count by 40–60% on those steps—with zero quality loss.

Core Principles of Lean Inference

Just-In-Time Context: The Pull System

In Lean manufacturing, a pull system means downstream demand triggers upstream production—nothing gets built until it’s needed. JIT Context means your agent fetches context exactly when a step requires it, scoped precisely to what that step needs.

The anti-pattern is the “God Context”: a single massive system prompt that pre-loads everything the agent might need across all possible execution paths. You pay the full token tax on every call, even when 80% of that context is never accessed.

The Lean pattern:

• Semantic caching at the retrieval layer: if a semantically similar query was answered 30 seconds ago, return the cached embedding result, not a fresh DB round-trip.

• Re-ranking before injection: Run a cross-encoder (a fast, cheap model like ms-marco-MiniLM-L-6-v2) over your retrieved chunks before injecting them into the LLM context. Top-3 precision beats top-20 recall for most tasks.

• Step-scoped context: Each node in your agent DAG gets only the context its specific tool call requires. The summarization node doesn’t need the tool definitions. The routing node doesn’t need the document corpus.

Standardized Work: Deterministic Guardrails

Lean’s standardized work principle says that defined, repeatable processes reduce variation and defects. In agent architecture, this translates to: make your LLM do as little undirected reasoning as possible.

Logic that can be encoded deterministically should be. Your state machine transitions, routing rules, retry budgets, and tool call sequencing should live in code—not in a prompt asking the model to figure it out.

Tools like LangGraph let you encode agent control flow as an explicit graph: nodes are LLM calls or tool invocations, edges are conditional transitions, and the state machine is a first-class object you can inspect, test, and version-control. This is categorically different from a single ReAct loop where the model decides everything.

Structured outputs (via instructor, OpenAI’s strict: true JSON mode, or Anthropic’s tool schemas) are the manufacturing equivalent of a jig: they physically constrain the output to the valid shape, making defects structurally impossible rather than probabilistically unlikely.

Takt Time: The Latency Budget

Takt time in manufacturing is the maximum allowable time per unit to meet customer demand. In agent design, every workflow should have an explicit latency budget per step and per full execution path.

Define your takt time first. If your end-to-end SLA is 2 seconds and you have 6 agent steps, your average per-step budget is ~333ms. That budget forces architectural decisions:

• Can this step use a smaller model to hit the latency target?

• Should this step be parallelized?

• Does this step even need an LLM, or is a heuristic or cached result sufficient?

DAG decomposition is your primary tool here. A complex task that looks like a single LLM call is often a DAG of 4–6 smaller model calls that can execute in parallel, each with faster TTFT on a smaller model, combining to lower overall latency than the single big call.

Prompt Caching as Kanban

Anthropic’s prompt caching and OpenAI’s equivalent cache system are Kanban cards for inference: reusable, pre-positioned work items that don’t need to be re-manufactured from scratch.

Your system prompt, tool definitions, and static knowledge base content are the same across thousands of requests. Cache them. On Anthropic’s API, a cache hit on a 10,000-token system prompt costs 10% of the base input token price. Over millions of calls, this is not a micro-optimization—it’s a cost structure change.

Design your prompts with cache-friendly prefix ordering: static system prompt first, static tool definitions second, dynamic context last. Anything that changes per-request must come after anything that doesn’t.

Before and After: Repo Analysis Agent

Before (Naive Architecture)

A single ReAct loop. One GPT-4o call per step. Full repository context dumped into the window on every iteration. Tool definitions re-sent each time. Sequential file reads. No output validation. Average: 14 seconds, ~85,000 tokens, ~$1.20 per run.

After (Lean Architecture)

• A small router model (8B, fine-tuned) classifies the task type and selects the appropriate specialist pipeline — adds 80ms, saves 60% of downstream model costs

• Prompt caching on tool definitions and system context — 90% cache hit rate after warmup

• Parallel tool execution for file reads — 4 simultaneous reads instead of sequential

• Structured output enforcement via instructor — zero retry loops in 500-run benchmark

• Strict step budget: 6 steps max, with a fallback to human handoff at budget exhaustion

Result: 4.2 seconds average, ~18,000 tokens, ~$0.09 per run. Same output quality score on eval suite. 13x cost reduction. 3.3x latency improvement.

Conclusion

The next frontier of AI engineering isn’t a bigger context window or a more capable base model. It’s the discipline to use what we already have without waste.

Every unnecessary frontier model call, every bloated RAG context, every sequential blocking operation, every retry loop from malformed output—these are engineering failures, not model failures. We built them. We can fix them.

Lean Inference isn’t a philosophy—it’s a set of concrete architectural decisions you can make this sprint. Audit your agent’s token burn by step. Map your sequential calls. Add structured outputs. Right-size your models. Cache your static prompts.

Build leaner. Run faster. Spend less. Ship better agents.

Built on 5 Small Models

When we say Small Language Model, we mean models under 20B parameters. Some are under 3B. They’re small enough to run on a single GPU, fast enough to return structured output in seconds, and cheap enough that running five of them in parallel against thousands of files is a rounding error, not a line item.

The catch — and it’s a real one — is that they don’t know everything. A 3B model is not going to write your novel, debate Kant, or replace Opus. But it doesn’t need to. It needs to find SQL injections. Or flag a deprecated dependency. Or notice that a function has crept past 400 lines.

Narrow the task, and small wins.

The SLMs That Compose Owlpack

Owlpack runs five agents every night, each focused on a different domain:

• Hunter scans for security vulnerabilities — SQLi, SSRF, auth bypasses, leaked secrets, cross-referenced against live CVE feeds.

• Tracker hunts bugs — null references, race conditions, off-by-one errors, suspicious test coverage gaps.

• Keeper audits dependencies — outdated packages, breaking changes, deprecation notices, abandoned libraries.

• Mason identifies refactoring opportunities — duplication, coupling hotspots, long functions, outdated patterns.

• Scribe analyzes the codebase itself — churn, review latency, complexity drift, module ownership.

Each one is a specialist. Each one runs in parallel. Each one returns structured findings that get deduplicated, diffed against history, and ranked by severity before they land in your inbox by morning.

Try doing that with a single frontier model. The math falls apart. A nightly full-repo scan, across thousands of users, calling GPT-class inference five times per repo, would cost more than most teams pay for their entire dev tooling stack. That’s why no one was offering this service. The unit economics didn’t work.

With SLMs, they do.

The case for specialization

There’s a deeper reason we built it this way, and it goes beyond cost.

When you fine-tune a small model on a specific task — say, identifying CVE patterns in JavaScript — it gets better at that task than a frontier model trained to do everything. Specialists beat generalists when the task has a defined shape. Code review is a defined shape. Dependency auditing is a defined shape. Detecting a leaked AWS key is a very defined shape.

A frontier model brings a trillion-plus parameters of knowledge about Roman history and SQL injection patterns. Hunter brings the SQL injection patterns. For this job, that’s the better tool.

It also means the agents don’t drift. They don’t get creative. They don’t hallucinate a function name that doesn’t exist in your repo because they read about it in a blog post once. Narrowness is a feature.

Privacy as a byproduct

There’s a third benefit that falls out of using SLMs: a smaller blast radius. Owlpack clones your repository at scan time, runs the agents, and deletes the clone. We don’t need a 1.5T-parameter foundation model to read your code. We need five tight, task-specific models that do their job and forget. That architecture is easier to audit, easier to contain, and easier to deploy in environments where data exfiltration risk actually matters.

What this looks like in practice

You install the GitHub App. You pick which repos to scan. We clone them at your chosen time, run all five agents in parallel, delete the clone, and deliver a structured briefing by morning. If there’s nothing new, we don’t email you. If there’s a CVE in a transitive dependency, you’ll know before standup.

The whole pipeline costs a fraction of what a single frontier call would. That’s what passes through to pricing. That’s what makes the service viable.

The big-model era taught the industry to reach for the biggest hammer in the room. The next era is about picking the right one. Five small, sharp tools, running every night — that’s what Owlpack is. SLMs are why it works.

Try Owlpack Free For 7 Days. Or email us sales@neurometric.ai if you want to try a team plan.

But somewhere along the way, a quieter revolution started inside the enterprise. Companies stopped asking “what’s the most capable model?” and started asking “what’s the most useful one?”

Small Language Models — generally defined as models with fewer than 10 billion parameters — have emerged as the answer. They’re cheaper to run, fast enough for real-time workflows, and small enough to deploy on-prem or even on a laptop. That changes the economics and the privacy posture of AI in ways the headline-grabbing frontier models simply can’t match.

At Neurometric, we just crossed 3,000 active SLM users. You can download a fine tuned SLM from us without becoming a customer so, only a little over 2,200 of those users have applied for a key and are using Neurometric to host the SLM. But we’ve seen which pre-fine-tuned models are most popular from our SLM Marketplace, and we’ve talked to some enterprises directly who have asked for our help with larger scale SLM deployments.

If you’ve ever wondered what people use SLMs for in the real world - here are the top 5 things we’ve seen.

1. Summarization: Gist Generation at Scale

The single biggest workload across our user base is document summarization. Legal briefs, customer transcripts, research reports, internal wikis.

Why an SLM wins here: most summarization doesn’t need creative prose. It needs the gist — accurate, fast, and cheap enough to run across thousands of documents a day. Pushing a 50-page PDF through a frontier model with a giant context window costs real money. An SLM does the same job at a fraction of the latency and a tiny fraction of the cost. When you’re processing 10,000 documents a week, that math becomes existential.

2. Resume Screening: Extraction Without Hallucination

HR teams were one of the fastest verticals to adopt. The job isn’t to “write a beautiful candidate evaluation” — it’s to pull skills, years of experience, certifications, and seniority into a structured format.

That’s an extraction task, not a creativity task. And ironically, smaller, fine-tuned models are less prone to embellishment than their bigger cousins. Fewer parameters means fewer paths for the model to “fill in the blank” with something that wasn’t on the resume. For HR — where a hallucinated qualification is a compliance problem — the precision of an SLM is a feature, not a limitation.

3. Code Refactoring: Local, Low-Latency Suggestions

Developers using SLMs for code refactoring are the most popular group who download and use it themselves, rather than have us host it. The two reasons they seem to like the SLM approach:

• Latency. An autocomplete suggestion that arrives 800ms after you stop typing is useless. Local SLMs respond in tens of milliseconds.

• Security. Proprietary code never leaves the machine. For regulated industries and any company with a defensible codebase, that’s non-negotiable.

The frontier model can still review the architecture. The SLM handles the thousand small edits in between.

4. CRM Summary: Killing the Sales Busy Work

Sales reps don’t write call notes; they scribble them. The result is a CRM full of half-finished entries that nobody trusts.

Our users are using SLMs to convert messy voice memos and chat transcripts into structured CRM fields — next steps, objections, deal stage, sentiment. This is a textbook SLM workload: repetitive, schema-bound, and high-volume. A rep doing eight calls a day generates 40 summaries a week. Multiply by a 200-person sales org and you can see why running this on a frontier model is a non-starter.

5. Meeting Prep: Briefing Sheets Before the Call

The fifth pattern is what users are calling “pre-meeting intelligence.” Before a customer call, the SLM ingests prior emails, past call notes, account history, and recent product activity — then generates a one-page briefing sheet.

Speed is the metric here. The briefing needs to be ready in the 90 seconds between the previous meeting ending and the next one starting. Frontier models can’t hit that latency reliably. SLMs can.

So What Are People Actually Using SLMs For?

They’re not writing novels. They’re not passing the bar exam. They’re not solving open-ended research problems.

They’re doing the most common work tasks.

SLMs are workhorse models. Across our thousands of users, the pattern is consistent: repetitive, high-volume, structured tasks where you need roughly 90% accuracy at roughly 5% of the cost of a frontier LLM. That’s not a consolation prize — that’s the entire enterprise AI opportunity. Most business value isn’t locked behind PhD-level reasoning. It’s locked behind tasks too small and too numerous to justify a $0.10-per-call inference bill.

The Shift: From General Purpose to Specialized Agents

What our users are really previewing is the next phase of enterprise AI: a shift away from one giant general-purpose model handling everything, toward fleets of specialized agents each doing one thing exceptionally well.

The frontier model becomes the orchestrator. The SLMs do the work.

If you want to find where SLMs fit in your stack, don’t start with your hardest problems. Start with your most repetitive ones. Find the tasks your team does a thousand times a week that need to be correct, not clever. That’s the frontier now.

Bigger isn’t better. Specialized is.

Every boardroom is talking about AI. Your competitors are deploying it. Your engineers are building with it. Your investors are asking about it. What almost no one is talking about — until the invoice arrives — is the ongoing cost of keeping it running.

That cost has a name: inference. Every time your product calls an AI model to generate a response, summarize a document, or answer a customer question, you’re paying for it. And if your product is gaining traction, you’re paying more every single month.

Your engineers might describe the problem like this: “We need to optimize our inference overhead to prevent compute-spend from cannibalizing our gross margins.”

In plain English: we need to stop spending so much money on the electricity and ‘brain power’ it costs to run our AI features.

Frontier models like GPT-5 or Claude are genuinely impressive. But they are also priced like a premium product, and using them as the default solution for every AI task in your stack is the equivalent of hiring a neurosurgeon to take your blood pressure. The capabilities are real. The expense is real. The waste is often real too.

This guide is for finance leaders, operators, and executives who want to move from “AI at any cost” to “AI at a sustainable cost.” Each strategy below includes the technical jargon your engineers might use, a plain-English translation, and — most importantly — what it means for your bottom line.

Strategy 1: Size Matters — The Rise of Small Language Models (SLMs)

What Your Engineers Will Say

“We should migrate from monolithic LLMs to domain-specific Small Language Models (SLMs) for task-oriented workflows.”

What That Actually Means

Using a giant, expensive AI brain for simple tasks is wasteful. A smaller, cheaper model that specializes in one specific job will do that job just as well — at a fraction of the cost.

Not every AI task requires genius-level reasoning. Summarizing a short email, classifying a support ticket, extracting a date from a form, or checking whether a sentence is positive or negative — these are not hard problems. They don’t need a model trained on the entirety of human knowledge. They need a focused, efficient specialist.

The Bottom Line

The cost difference is not marginal — it is staggering.

If your application processes 100 million tokens per month — a realistic volume for a product with moderate usage — the difference between a frontier model and a small specialized model is $14,950 per month, or nearly $180,000 per year. For tasks where the smaller model performs equally well, that is pure margin destruction.

The CFO question to ask: “Which of our AI features actually require our most expensive model, and which ones are just using it because it was the default?”

Strategy 2: Hardware — Newer Isn’t Always Better

What Your Engineers Will Say

“We can achieve better TCO by utilizing legacy GPU clusters or N-1 generation hardware for non-latency-critical inference.”

What That Actually Means

We don’t need the world’s fastest, brand-new computer for every task. Last year’s chips are still very fast — and significantly cheaper to rent.

The AI hardware market is driven by hype and scarcity. NVIDIA’s latest H100 and H200 GPUs are in extraordinarily high demand, which means cloud providers can charge a premium for them. For real-time, latency-sensitive tasks — think a live customer chatbot — that premium may be justified. For everything else, it usually isn’t.

Batch processing, overnight report generation, data enrichment pipelines, and other non-time-critical workloads can run on older-generation hardware with no meaningful impact on output quality.

Comparing Your Options

The practical play is a tiered hardware strategy: pay for premium compute only where your users will actually feel the difference. Route everything else to reserved or spot capacity. This is standard practice in mature cloud cost management — it’s time to apply the same logic to AI.

Strategy 3: The “Squish” Factor — Quantization

What Your Engineers Will Say

“Applying 4-bit or 8-bit quantization to the model weights to reduce VRAM requirements.”

What That Actually Means

Shrinking the AI model’s file size so it fits on cheaper hardware. It’s like converting a massive 4K video file into a standard-definition version — it plays fine, takes up far less space, and most people can’t tell the difference for everyday use.

AI models are, at their core, enormous files made up of billions of numerical values called “weights.” Quantization reduces the precision of those numbers — from 32-bit floating point down to 8-bit or even 4-bit integers. The model becomes physically smaller, requires less memory to run, and can be deployed on less expensive hardware.

The Tradeoff — And Why It’s Usually Worth It

Quantization is not free. There is a quality tradeoff. A fully quantized model will be marginally less “intelligent” than its full-precision counterpart — typically in the range of 1–3% degradation on benchmark tests.

In exchange, you can expect:

• 50–75% reduction in memory requirements, enabling deployment on cheaper GPUs

• Significant reduction in hardware costs, often 60–80%

• Faster inference in many cases, due to smaller data transfers

For most business applications — customer support, internal search, document processing, data extraction — a 1–2% quality reduction is entirely imperceptible. The math is straightforward: accepting a marginal quality trade in exchange for 70% cost savings is almost always the right financial decision.

Strategy 4: Finding the Right “Landlord” — Inference Hosting Providers

What Your Engineers Will Say

“Moving from a general-purpose CSP to a specialized serverless inference provider to minimize cold-start latency and egress fees.”

What That Actually Means

Instead of running AI through a big, expensive general-purpose cloud platform that charges a premium for convenience, move to a specialized provider built specifically for AI inference. It’s cheaper, faster to set up, and purpose-built for the job.

AWS, Azure, and Google Cloud are exceptional general-purpose platforms. They are also priced accordingly, and they layer fees on top of fees — data egress charges, API gateway fees, storage costs, and support tiers that add up quickly. More importantly, they were not designed from the ground up for AI inference workloads.

A new generation of specialized inference providers — companies like Together AI, Fireworks AI, Replicate, and others — offer the same underlying models at lower prices because their entire infrastructure is optimized for one thing: running AI models efficiently at scale.

What to Evaluate

When assessing inference providers, the key financial metrics to request from your engineering team are:

• Cost per million tokens for your specific models and use cases

• Cold-start latency — the delay when a model hasn’t been called recently

• Egress fees — charges for data leaving the provider’s network

• SLA and uptime guarantees relative to your product requirements

The switching cost is typically low. For many teams, a migration to a specialized inference provider is a one-to-two week engineering project that delivers permanent cost reductions of 30–50%.

Strategy 5: Don’t Pay to Think Twice — Caching and Batching

What Your Engineers Will Say

“Implementing semantic caching and request batching to improve throughput and reduce redundant compute.”

What That Actually Means

Caching: If the AI already answered a question, save that answer and reuse it instead of paying the AI to think through the same problem again.

Batching: Instead of sending the AI one piece of work at a time, queue up a pile of tasks and send them all at once. It’s more efficient, like doing one large grocery run instead of ten small trips.

These two techniques attack waste from different angles.

Caching is particularly powerful for applications where users ask similar or identical questions — internal knowledge bases, customer FAQ bots, product recommendation engines. A semantic cache stores previous AI responses and recognizes when a new question is close enough in meaning to warrant returning the saved answer rather than generating a new one. Depending on the application, cache hit rates of 30–60% are achievable, meaning nearly half of all AI calls are resolved for free.

Batching is most valuable for background processing workloads — nightly data enrichment, bulk document analysis, report generation. Running 1,000 tasks in a batch is substantially cheaper per task than running 1,000 individual requests, because the fixed overhead of spinning up compute is amortized across the entire batch.

Together, these optimizations can reduce effective inference costs by 20–50% without any change to model selection or hardware.

The CFO’s Checklist for AI Cost Optimization

Before your next quarterly business review, ask your engineering lead to walk through these questions:

On model selection:

• Are we using a frontier model (the “sledgehammer”) for tasks that a smaller specialist model (the “scalpel”) could handle equally well?

• Have we audited our AI feature set to categorize tasks by complexity and matched each to the appropriate model tier?

• Have we run a side-by-side cost comparison for our top three highest-volume AI workflows?

On infrastructure:

• Are we running all inference on on-demand, latest-generation hardware, or have we right-sized to reserved and spot capacity for non-critical workloads?

• Are we paying general-purpose cloud “convenience fees” for inference that a specialized provider could handle more cheaply?

On efficiency:

• Are we using quantized models where quality requirements allow?

• Have we implemented caching for high-repetition query patterns?

• Are background processing workloads running in batches?

On measurement:

• Do we track cost per inference as a formal metric?

• Do we have an alerting threshold for when compute spend exceeds a defined percentage of gross margin?

If the answers to more than half of these questions are “no” or “we’re not sure,” there is almost certainly significant, recoverable margin sitting in your AI infrastructure.

Conclusion: Efficiency Is a Competitive Moat

The companies that will win the AI era are not necessarily the ones using the most powerful models. They are the ones that have learned to deploy AI intelligently — matching the right tool to the right task, on the right hardware, with the right efficiency optimizations layered on top.

The company running a comparable AI product for $1,000 per month will systematically outcompete the company running it for $10,000 per month. Lower costs mean higher margins, more room to price competitively, and more budget to reinvest in product development.

None of the strategies in this guide require cutting corners on quality. They require applying the same financial discipline to AI infrastructure that good operators apply everywhere else in the business.

Your call to action is simple. Schedule thirty minutes with your lead engineer and ask two questions:

1. “What is our current cost per inference?”

2. “Have we tested a smaller, specialized model for any of our high-volume workflows?”

The answers will tell you everything you need to know about where your next margin improvement is hiding.

This technology is made possible by our Harbor integration, which allows us to test lots of models on lots of agentic tasks. We have a good understanding of what various small base models are capable of.

The technology is still new, and we do have a brief human review of each SLM but, it happens quickly. Let us know if you have any questions or issues when you try it out.

But the moment you try to run Harbor as a team — engineers, research scientists, infra folks who don’t all live in the terminal — the rough edges show up. Who launched run 1037? What model did it use? Where are the logs? Did the run that died at 03:14 because of a Daytona auth blip ever come back? The CLI doesn’t answer those questions, and we don’t think it should — that’s not what Harbor is for.

So we built Harbormaster: a web platform that wraps Harbor and runs it on our own Kubernetes cluster. It is decisively not a replacement for Harbor. It is a thin shell around harbor run, the SQLAlchemy job tracking, and the artifact upload conventions, designed for the case where your benchmark suite is shared infrastructure rather than a personal script. Everything Harbor knows how to do, Harbormaster does — it just gives you a submit form, a dashboard, an audit trail, and a leaderboard at the end of it.

This post explains how it composes with Harbor, what it taught us about Harbor’s behavior under sustained load, and the specific bugs and edge cases we’d like to push back into the project.

What it is, concretely

The user-facing surface is small on purpose. You pick an agent (Claude Code, Gemini CLI, Codex), a dataset, a model, and a concurrency level, and you hit submit. From there:

1. The frontend — a React SPA served by nginx — POSTs to a FastAPI backend that writes a BenchmarkRun row to Postgres.

2. The backend creates a Kubernetes Job whose pod is a “runner” image that has Harbor installed.

3. That runner builds a JobConfig programmatically and calls Harbor’s Python API.

4. Harbor does what Harbor does: it fans out trial pods at the requested concurrency, talks to the agent, runs the verifier, and writes the trajectory and reward to the configured artifact root.

5. We watch the artifact root — backed by S3 in production, MinIO locally — and stream per-trial state back into Postgres so the dashboard can show live progress, logs, and a running leaderboard.

The infrastructure is intentionally boring: EKS with Karpenter for node autoscaling, an ECR pull-through cache so trial pods don’t hammer Docker Hub, Traefik IngressRoute with OIDC for auth, and k3d + MinIO for local development so contributors can iterate without a cloud account. None of this is novel by itself. The point is that gluing it to Harbor turned out to be straightforward, which is a real compliment to Harbor’s design — the BaseEnvironment interface and the Trial/Job model gave us clean seams to build against.

The runner-as-Job pattern

The single most useful pattern we found is treating a Harbor invocation as a Kubernetes Job, not as a long-lived service. Each benchmark run is its own Job, with its own pod, its own resource envelope, and its own lifecycle. Harbor’s per-trial concurrency lives one level below: the runner pod calls into Harbor, and Harbor — via whichever environment adapter you’ve configured — fans out to N parallel trial pods on the same cluster.

Two things fall out of this naturally that we wanted:

Resilience to runner restarts. Harbor already uploads trial artifacts trial-by-trial as they finish. We lean on this. If a runner pod gets evicted mid-run (Karpenter consolidation, spot reclaim, OOM), the next reconciliation reads the artifact root, sees which trials have completed, and the dashboard reflects the partial state correctly. The run isn’t “lost” just because the orchestrator went away. This is mostly Harbor’s design doing the work — we just don’t get in its way.

Resource isolation per run. A misbehaving Replicationbench run that wants to allocate 32 GB of RAM doesn’t take down the dashboard or the API, because the runner is in its own pod with its own limits. The Kubernetes Job boundary turns out to be a much better blast radius than “long-running daemon that orchestrates everything.”

This isn’t a new idea — it’s basically how Argo Workflows and Tekton model things — but we want to call it out for the Harbor community because it’s a pattern that maps cleanly onto Harbor’s existing JobConfig/TrialConfig abstractions without requiring any changes to the framework. If you already run Kubernetes, this is a low-friction way to operationalize Harbor for a team.

What we ran

We used Harbormaster to run three frontier coding agents — Claude Code (claude-opus-4-6), Gemini CLI (gemini-3.1-pro-preview), and Codex (gpt-5.4) — across roughly a dozen public datasets each. The full numbers are in the appendix, but the headline results, on the datasets where all three completed:

Those numbers aren’t the point of this post — they’re a snapshot, and we’ll publish a proper write-up once SWE-Bench Verified completes for all three agents. What is the point: running this volume of trials end-to-end through Harbor, on our own cluster, on our own dime, gave us a much richer view of where Harbor is solid and where it has sharp edges.

What we’d like to push back upstream

Three findings are worth surfacing for the project, and we’ll be opening issues on each.

/logs/artifacts directory missing in trial environments. Several of our runs on SWE-Bench Multilingual hit a 53–86% verifier failure rate that resolved to a missing artifacts directory in the trial container. The verifier expected /logs/artifacts to exist; in some task environments it didn’t, and the verifier failed fast rather than creating the directory. Both Claude Code and Gemini CLI runs were affected, which suggests it’s a task-environment issue, not an agent issue. We’ve worked around it locally by ensuring the directory exists in our runner image preflight, but a fix in the relevant adapter would be cleaner.

Aider Polyglot verifier errors at scale. On Aider Polyglot at n_concurrent=4, we saw 88 verifier errors out of 225 trials for Claude Code (94 of 225 for Gemini CLI). The pattern looked race-y — re-running affected trials individually passed. We suspect contention on a shared verifier resource at high concurrency, but we haven’t fully isolated it. We’ll attach our trajectory bundles to the issue.

Pod OOM on Replicationbench HPC tasks. A handful of Replicationbench tasks run physics or ML simulations that allocate gigabytes of memory before Harbor finishes attaching. The pod gets killed before the trial can even start, which means the trial doesn’t show up as a failure in the usual sense — it just disappears. Two changes would help: a configurable per-task memory ceiling (so we can opt these tasks into a higher limit) and clearer surfacing of “pod OOMKilled before trial start” as a distinct failure class in the trial output.

None of these are dealbreakers. They’re the kind of thing you only find by running the framework hard, in an environment that isn’t the reference Daytona setup, against the full heterogeneity of the public benchmark catalog. We’d rather find them and report them than work around them silently.

What’s next

Three things on our roadmap that we think are interesting for the community:

We’re publishing the runner image and the K8s Job templates. The web UI itself has too much of our internal auth model baked in to open-source as-is, but the runner — the part that turns “Harbor on a laptop” into “Harbor on a cluster” — is general, and we’ll have it on GitHub by month’s end. If you have an EKS or GKE cluster and want a starting point that isn’t “stand up Daytona,” that’s the artifact to grab.

We want to contribute a richer EKS-flavored environment adapter alongside the existing GKE one. The two clouds aren’t identical (IRSA vs. Workload Identity, ECR vs. Artifact Registry), and the differences matter for image pull behavior at high concurrency. We’ve worked through them; we’d like that work to live in harbor-framework/harbor rather than only in our fork.

And we’d like to keep running the public catalog. The numbers above represent perhaps two weeks of cluster time. There are clear gaps — Codex hasn’t finished SWE-Bench Verified, Gemini CLI never completed it across three attempts, and Replicationbench is blocked on the OOM issue above. We’ll push through those and publish complete leaderboards as runs land.

If you’re building something similar — we’d love to compare notes. Harbor’s whole bet is that a shared standard makes the ecosystem move faster, and operationalizing that standard for teams is exactly the kind of thing the community can do better together than any one of us can do alone.

Over time we will be using Harbormaster to run as much of the Harbor suite as possible. Our work at Neurometric is heavily tied to task-specific AI and so, running Harbor on lots of models to understand the smallest, the fastest, and the most cost effective models for each task will help us move our product forward. As we go through this, we will make sure we publish the results. Stay tuned.

That era is ending — not gradually, but structurally.

The Death of the Funnel

Consider a traditional travel booking site. Its UX is a masterpiece of guided constraint: search bar, filter panel, calendar picker, seat map, confirmation modal. Every screen funnels the human toward a single, monetizable action. The design is the product.

Now consider what happens when an AI agent books your travel. It doesn’t load the homepage. It doesn’t hover over the “flexible dates” toggle. It hits an API, cross-references your calendar, checks your preference history, and surfaces a ranked shortlist — bypassing every carefully crafted screen entirely.

The funnel isn’t just broken. It’s irrelevant.

Agents don’t navigate UIs. They negotiate with systems. And when the agent is the primary “user” of software, the human behind it occupies an entirely different role — one for which we have almost no design vocabulary. Until now.

Defining HX: The Harness Experience

HX — Harness Experience — is the design discipline governing the interface between a human and their agentic fleet.

Where UX asks “how do I get the human from A to B?”, HX asks something more complex: how do I let a human steer, trust, and audit a system acting on their behalf, at speed, across dozens of simultaneous tasks?

The word “harness” is deliberate. A harness isn’t a cage — it doesn’t constrain. It channels energy, distributes load, and keeps you connected to something far more powerful than yourself. A well-designed harness means you stay in control without doing all the work. A poorly designed one means you get dragged.

That is the failure mode of most AI-native products being built today

.

From User to Director

There is a psychological shift happening that designers are not yet taking seriously.

A user takes action. They click, type, swipe. Engagement is transactional and immediate — the button changes color; the form submits.

A director evaluates outcomes. They set intent, observe execution, and make judgment calls about whether the agent’s behavior actually reflects their goals. It requires a fundamentally different kind of trust.

Think about the difference between driving a car and managing a driver. As a driver, you feel every turn. As a manager of a driver, you need a map, an ETA, and a reason to believe they know the route.

HX must be built for directors, not drivers.

The Three Pillars of HX

Steerability is the first. Beyond a simple prompt, how do you express nuance? A great travel agent doesn’t just hear “I want a beach vacation.” They remember you hate crowds and ask whether this trip is romantic or family. Steerability means designing systems where human intent can be expressed in layers — with context, constraints, and corrections — not just a one-line instruction.

Transparency and Auditability is the second. An agent that acts like a black box will eventually fail in ways that feel inexplicable, and therefore unforgivable. But flooding the director with raw logs is its own design failure. The challenge is legibility at the right altitude — letting a human understand why a decision was made without requiring them to become an engineer. Think of it like a flight manifest versus cockpit telemetry — same journey, entirely different information needs.

Intervention Points are the third and most underrated. The moment a human most needs control is often the moment automation is least prepared to hand it back. Designing intervention points means thinking carefully about when to surface a pause, how to hand off context cleanly, and how to let a human step in — and back out — without shattering what the agent was doing. This is not a button. It is an architecture.

The New Design Toolkit

So what replaces Figma?

Not a single tool, but a new set of design primitives. Designing for latency — because agents operate over seconds or minutes, not milliseconds, and unexplained silence breeds distrust faster than failure does. Designing for uncertainty — because an agent operating at 78% confidence needs to surface that honestly, not paper over it with false precision. Designing for verification — because the human’s core job is now quality control, and the interface must make that fast, clear, and low-friction.

The new craft lives in feedback loops, not flows. In state communication, not screen transitions. In systems honest about the edges of their own competence.

Stop Building Cages. Start Building Harnesses.

The best UX designers of the last decade built elegant cages — beautiful, invisible paths that guided users exactly where the product needed them to go. It was a form of benevolent choreography.

That skill, applied to agentic systems, produces something genuinely dangerous: automation that feels smooth right up until it catastrophically diverges from what you actually wanted.

The call to action is urgent. Stop designing for obedience. Start designing for collaboration. The agent is your most powerful colleague — capable, fast, and tireless. HX is how you ensure it is actually working for you, not just working.

The funnel is over. Build the harness.