Part 1, which lays out why inference time compute matters, is here. Part 2, which shows our results on CRMArena, is here.
Inference time compute and the impact it has on different tasks and different models is still a vastly underexplored area, and our goal at Neurometric is to explore it and make it widely available to the companies and engineers building AI systems. To that end, our early research shows a lot of areas of promise where we should extend our efforts.
One area that is interesting is consistency of performance. See the chart below.
This chart looks at the consistency of performance by each model. The closer the dots are together, the more consistently it performed on that metric across various runs.
This is important because when we try to optimize a system for performance, that can mean a lot of things. Are we optimizing for cost, accuracy, latency, or even consistency of performance? There are use cases that would require optimization for each of those.
Our next phases of research are going in a few directions.
More ITC algorithms. We’ve released best-of-n and CoT results, but have run some beam search results as well, and are starting to work on lots of variations on the major ITC algorithms.
More compute platforms. We ran all of this on Amazon Bedrock. Some of this work will require us to get closer to the hardware and so, expect to see some tests on how these things run on different platforms.
More benchmark/eval sets. We are working to get more real world data sets to try these algorithms on, but if you have some, or want to partner on making some, please reach out.
More models. We’ve already had several other model providers reach out to be part of our results. Stay tuned.
If you are a company who can benefit from this work and would like to partner on the research in various ways, please reach out.
Stay tuned for some more research releases. For now follow us here to make sure you don’t miss anything.