Michael Porter, the HBS professor who brought deep academic rigor to the concept of business strategy, described corporate strategy as a series of tradeoffs.
In an AI world, what are the tradeoffs you have to make? Earlier this year Satya Nadella from Microsoft said he believes the foundation models are becoming commoditized. Even if they were not, their business model is such that we all have access to them so building a competitive advantage around model capabilities is difficult.
Since the launch of OpenAI’s o1 model late last year, the AI world has been focused on squeezing more out of inference. As this excellent blog post shows:
The challenge here is that “inference compute” is its own messy area full of all kinds of tradeoffs and decisions. The opportunity to differentiate your company by your inference time compute decisions is huge. Here are three things to think about think about as you make your inference decisions.
Open vs Closed Models
The trend we have seen in the marketplace is that enterprises are building on OpenAI and Anthropic for proof of concept but then seek to move to their own infrastructure. Open source models like Llama or Qwen are great for privacy and control but the performance challenge is that they don’t have the same reasoning capabilities as what the frontier model labs can provide. This is where a platform like Neurometric can help. We make it easy to build, manage, and deploy inference time compute workflows, particularly on open source models.
For certain use cases, smaller models plus inference time compute algorithms can outperform larger models, allowing you to get the performance of top models with older, smaller models, or with open source models.
Cost vs Latency vs Accuracy Tradeoffs
How good and how fast does your model need to be? How cheap does it need to be? We have spoken to some enterprises who have built agents and determined they are too expensive to run for the given use case. David Magerman from Differential VC recently posted about this with respect to Cursor’s price increases and the coming wave of inference price inflation as we understand the real costs.
Ben Dickson wrote an excellent post about using smaller models plus some inference time compute algorithms to get the performance of a much larger model while saving money. This is a second key use case for Neurometric’s platform. Perhaps you are happy with your system performance but want to run it more cheaply. We can help you figure out how to run cheaper models by trading inference time latency for model size.
Inference As a Value Chain Differentiator
Value chains are the steps you take when you make things. Two competitors usually have slightly different value chains because they are optimized for different target customers segments. We believe this same thing will happen for AI inference.
The most common AI inference use case will probably be AI agents, and because agents will do knowledge work, their value chain will be differentiated in a few ways:
Speed of execution
Total cost
Accuracy of decision/output
Sources of information that it can access
Integration with other workflows and agents you use
These are all areas where companies can make tradeoffs vs competitors. Trading speed for price, or speed for accuracy, might mean selecting different inference algorithms, like using Beam Search instead of Best of N. If you multiply this across the hundreds or thousands of tasks that an agent will be able to do, you realize that as an enterprise, your inference compute configuration - meaning the specific hardware and algorithms you use to generate your inference outputs, might be a source of differentiation and competitive advantage. If that’s the case, you will want that to be proprietary.
Neurometric can help you build and manage your inference time compute configuration while keeping it all proprietary so that you don’t have to share it with the foundation model labs.
Summary
These are three of the biggest strategic decisions you have to make when designing your inference infrastructure. If you are working on this, here is how Neurometric can help:
We can help you deploy open source models with the same or better performance as closed proprietary models.
We can help you tweak smaller models to perform as well as larger models so you can meet your cost performance targets.
We can help you manage your inference infrastructure to keep it proprietary to your business, and out of the spotlight of the top foundation model companies.
If you are working on your AI inference decisions, please reach out so we can help.