On Tuesday, Microsoft and NVIDIA announced they’re investing $15 billion in Anthropic, while Anthropic committed $30 billion to Azure compute capacity. That’s $45 billion in combined commitments announced in a single day. Add NVIDIA’s $4 trillion market cap, nearly $400B in combined annual capex from hyperscalers, and foundation models raising billions in record rounds. The AI infrastructure buildout is historic. Yet the layer that enables production deployment doesn’t exist as infrastructure.
The compute buildout is necessary but insufficient. Our CRMArena benchmarking reveals something surprising: the performance delta between strategic orchestration and naive prompting can exceed the gap between models themselves. We tested multiple leading models (Llama, Qwen, GPT-OSS, Deepseek, Amazon Nova) on CRMArena tasks using three inference-time compute algorithms. Models plateau at similar accuracy levels on production workflows. But how you orchestrate can outweigh which model you select. Algorithm choice, verification strategy, and context management all can matter as much as raw model capabilities.
This finding expands the infrastructure challenge. The industry is racing to build better models and bigger data centers. Enterprises are evaluating which foundation model to standardize on. But if orchestration matters as much as model selection, the bottleneck isn’t just model capability—it’s the infrastructure layer between models and production applications.
Princeton’s HAL research demonstrates this pattern across multiple domains through 21,730 agent evaluations. Their analysis revealed that higher reasoning effort reduced accuracy in the majority of runs tested. Routing strategies that match task complexity to model capability outperform both always-using-the-best-model and always-using-the-cheapest-model approaches. How a model approaches a problem matters more than raw capability scores.
Physical infrastructure is scaling. But enterprises still can’t deploy effectively. The gap is orchestration infrastructure. Enterprise spending on generative AI grew 6x to $13.8 billion, yet 95% of enterprise AI pilots fail to deliver measurable business impact. These aren’t failures of model capability. MIT’s research found the problem is how enterprises attempt to integrate and orchestrate AI in production workflows. Meanwhile, the model market continues fragmenting. A16z reports that among startups using open-source models, 80% are building on Chinese open-source AI, adding dozens more routing decisions to an already complex landscape. The chip market is diversifying too (TPUs, GPUs, custom AI accelerators) each optimized for different workloads. Edge deployment compounds this: orchestration decisions must account for device constraints, intermittent connectivity, and real-time performance requirements.
What’s Missing
Much of the AI stack is maturing. Physical infrastructure is scaling, foundation models are diversifying, serving layers work reliably, applications are being built rapidly. But there’s a gap between serving and applications—the Context Layer. This is the orchestration infrastructure that makes models work in production. It doesn’t exist as integrated infrastructure, so every company builds it from scratch.
It handles several functions that must work together:
Intelligent routing: Match tasks to models based on latency, cost, and accuracy requirements. A customer support FAQ goes to a fast model. Complex troubleshooting escalates to a reasoning model.
Quality assurance: Verification, error handling, hallucination detection before outputs reach users. Check responses against knowledge bases, flag uncertain outputs, validate structured data against schemas.
Stateful optimization: Learn from production patterns to improve routing decisions over time. Track which model-task pairings produce good outcomes, optimize for domain-specific quality-cost tradeoffs, build institutional knowledge. The system maintains context across time, tracks intent across states, preserves identity across environments, and captures meaning across tasks.
Here’s why these need to be co-designed: routing without verification ships hallucinations. Verification without state management repeats the same mistakes. State management without routing has nothing to optimize. The value comes from integration, not from bolting together three separate systems.
Model routing platforms proved enterprises need orchestration infrastructure and their adoption validates the pain point. But routing alone creates new problems. When you can easily switch between dozens of models, you’ve multiplied your verification surface area. Inference-time compute algorithms need coordination across models. Verification systems need domain-specific configuration. Specialized small models need training and deployment infrastructure. Routing platforms made model selection easy. The hard part is making it work.
Leading ML teams are building verification systems here, context management there, optimization logic elsewhere. All in-house, all from scratch, because integrated platforms don’t exist. Every sophisticated team is rebuilding the same infrastructure independently.
Why This Captures Value
The Context Layer captures value because task-specific performance data compounds in ways that create genuine switching costs.
Consider what a mature Context Layer learns over time in production: routing rules that match task complexity to model capability, quality thresholds that balance accuracy against speed for different use cases, domain patterns that improve both verification and context management. Once a system learns the tasks, quality-cost tradeoffs, and production patterns, switching means rebuilding that institutional knowledge from scratch. The routing rules, verification logic, and optimization strategies encode how AI works in a specific context.
As models continue fragmenting and improving, orchestration becomes more valuable, not less. More model options means more routing complexity. Better model capabilities means verification matters more (higher stakes when things go wrong). Faster model iteration means state management is critical (learn quickly or get left behind). The Context Layer doesn’t compete with models. It makes them more useful in production.
The Infrastructure Opportunity
Physical infrastructure is scaling. Foundation models are diversifying. Applications are being built rapidly. The Context Layer represents the next infrastructure opportunity. Currently, sophisticated teams rebuild it in-house because integrated platforms don’t exist.
Routing platforms proved demand exists and captured early value by solving one function. But enterprises need integrated systems. CTOs are rebuilding infrastructure that should exist as platforms. Developers shipping AI features are either building orchestration themselves or discovering quality problems in production. The patterns exist in fragmented implementations across thousands of companies.
The question is whether these patterns get extracted into general-purpose infrastructure or remain locked in proprietary systems. Foundation models will keep improving. Infrastructure will keep scaling. But the Context Layer will determine which enterprises actually capture value from AI, and which spend their engineering resources rebuilding the same orchestration infrastructure everyone else is building.
The companies that integrate routing, verification, and stateful optimization into production-grade infrastructure will define how AI gets deployed at scale. Foundation models will keep improving. Infrastructure will keep scaling. But the Context Layer will determine which enterprises actually ship AI to production.