Compounding Inference Is As Powerful As Compounding Interest

The best idea from finance applies to AI as well

Compound interest works because of one decision: you reinvest the return instead of spending it. The rate matters less than most people think. A modest rate reinvested for long enough beats a great rate paid out as income. Linear versus exponential is entirely about whether the output gets fed back into the base.

Most AI teams run inference as income. A request comes in, you call a model, you get an answer, the answer is consumed. The next request costs the same, uses the same model, and produces the same quality. You’ve spent the token. Nothing accrued.

Tobi Lütke posted a chart on September 1 that shows what reinvestment looks like. Shopify’s ML team fine-tuned a 0.8B-parameter model for one job, building buyer profiles, and it now beats GPT-5.6-sol at its highest reasoning setting on that task: 84.6 versus 83.0 on their judge. Tobi’s take was that tiny models for special-purpose tasks work “incredibly well” when you have “a great self improving recursive flywheel.” Read that carefully. The result isn’t the small model. The result is the machine that produced it.

What the chart actually shows

Three training runs in one week. July 23, 29k samples, score 75.3. July 27, 42k samples, 78.1. July 30, 54k samples, 84.6. The gains got bigger with each run, not smaller. That’s the shape of compounding, and it came from three loops running simultaneously.

The quality loop. The frontier model is the teacher. Its outputs get graded, and the ones that pass become training data for the student. Every expensive API call is doing two jobs: serving the request and depositing a labeled example into your dataset.

The cost loop. The system prompt went from 9.1K tokens written out in full to 1.1K tokens “gisted,” an 8x cut. Once the model has learned the instructions through training, you stop paying to re-explain them on every call. Smaller model, shorter prompt, cheaper request.

The volume loop. Throughput went from 2M profiles per day on the prior 2B production model to 72M per day on the 0.8B model across 100 H100s. A 36x increase. More volume means more graded samples, which means better training data, which means the next run is cheaper and better still. The savings from cheap inference fund the inference that generates the next round of training data.

Each loop feeds the other two. That’s the reinvestment.

What it takes to build this

Anyone can fine-tune Qwen. That isn’t the hard part and it isn’t the moat. The Shopify team went from behind the frontier to ahead of it in seven days because the surrounding infrastructure already existed. Four pieces matter.

A task-level definition of quality. Not “is the model good” but “is this specific output a good buyer profile.” You need a judge that scores individual outputs on this task, calibrated well enough that you’d trust it to decide what goes into the training set. Without this, you have no way to know whether run three is better than run two, and no way to filter the teacher’s outputs into clean data. Everything downstream depends on it.

A pipeline from production traffic to training samples. The data isn’t in a lab. It’s in the requests you’re already serving. If your inference logs are write-only, you’re paying the teacher and throwing the lesson away. The plumbing that turns yesterday’s traffic into today’s graded dataset is the actual asset.

A cadence. Three runs in a week. Not a quarterly retraining project with a Jira epic. The flywheel compounds at the frequency you turn it, and a team that ships a new checkpoint every few days will lap a team that does it twice a year, even if the second team’s individual runs are better.

Clear routing between teacher and student. Early on, the frontier model handles everything and the student learns. As the student closes the gap, traffic shifts. You need to know, per task, which model is winning right now, and the answer changes weekly. Guessing here either wastes money on the teacher or ships bad outputs from the student.

The mindset shift

Stop thinking about your API bill as rent. Think of it as tuition. The frontier model’s job on any narrow, high-volume task is to make itself unnecessary for that task. If you’re six months into production and still calling the same frontier model with the same 9K-token prompt for the same job, you’ve been paying tuition and skipping class.

The teams that internalize this won’t look dramatically different next quarter. Compounding never does at first. They’ll look untouchable in two years, with dozens of tasks running on models that cost a fraction of frontier pricing and outperform it, and it’ll seem sudden to everyone who was paying simple interest the whole time.

If you want a system that makes it easy to compound your IP into AI models, that’s what we do here at Neurometric. Contact us if you would like to chat.