In this episode of Inference Time Tactics, Rob and Cooper dig into the decisions shaping one of the most important (and least understood) layers of AI infrastructure: inference time compute.
We explore why enterprises often begin with closed models like OpenAI or Anthropic, then migrate to open stacks, and what makes that transition challenging. From cost-performance trade-offs to the impact of pricing shifts and governance, we break down how inference-time configuration is becoming a competitive differentiator.
Drawing on insights from reviewing over 250 research papers and dozens of enterprise conversations, this discussion covers the real-world constraints and opportunities in optimizing inference workflows.
In this episode we cover:
Why many enterprises start with closed models and then move to open source stacks
Challenges of switching, from capability gaps to hardware considerations
Cost-performance trade-offs in inference architectures
How configuration can be a competitive edge in the AI value chain
Pricing dynamics, vendor lock-in, and their impact on adoption
Governance and policy considerations for inference workflows
The growing variety and complexity of reasoning algorithms
Why benchmarking reasoning and multi-step tasks remains a major challenge
Why it matters
Inference time compute is no longer just an engineering choice. It is a strategic one. The architectures, algorithms, and configurations you choose at runtime will increasingly define performance, reliability, and cost structure. For builders, researchers, and decision-makers, understanding these trade-offs is critical.
Listen to Episode 2: Inference Time Tactics - Episode 2