Ever since OpenAI launched their first reasoning model, -o1, I’ve been fascinated by the idea that the way you probe these models - the “thinking” algorithms you apply, can get you different outcomes. Over the past 9 months we’ve explored this here at Neurometric. We published some research showing that test time scaling algorithm choice matters on a per-task basis, and now we’ve decided to launch a tool to help you explore the difference.
Today we are excited to launch a Leaderboard for test time scaling algorithms. It allows you to see which algorithm plus model combination performs best on various tasks in the CRMArena data set. At the moment, the only benchmark we support is CRMArena because we are focused on more work task related benchmarks, not academic ones, and CRMArena is the best widely available benchmark that meets this criteria.
More benchmarks are coming, and we are in talks with several companies about unique work-related data sets that might make good work task benchmarks as well. In the meantime, please give us feedback on what else you would like to see in this leaderboard. We have our own long roadmap but, community feedback is always useful as well
The blue boxes at the top highlight a few key features:
top row:
Best Ensemble / 72.7% / 5 systems
Cost-Efficient Ensemble / 61.3% ($0.008) / 2 systems
Latency-Efficient Ensemble / 61.7% (24.9s) / 2 systems
bottom row:
Best Single System / 63.8% / gpt.oss.120b + Weighted (n=3) - Haiku 3.5 judge
Cost-Efficient Single / 54.6% ($0.011) / gpt.oss.120b + CoT
Latency-Efficient Single / 56.3% (6.4s) / gpt.oss.120b + Weighted (n=3) - Nova Pro judge
Interestingly, while gpt-oss-120b is kind of the clear choice on average, its consistency score isn’t great compared to some of the other models
The reason this leaderboard is important to us is because we’ve seen a trend in companies as they move along the AI maturity curve. While nearly everyone starts out building a prototype on one single model, usually a frontier lab model, as AI products start to scale it becomes obvious that some workloads are better handled with other models. Multi-model systems become the norm as you become more AI mature. But figuring out which models to choose and why is not intuitive. Our leaderboard is a small step towards a more data driven approach to AI systems design.
We are also working over time to add more functionality and make the leaderboard more useful. In the meantime, if you have a model that you would like to add to our testing suite, or if you would like to run these tests on your own data privately, please reach out.