Optimizing AI Inference with MoE Routing for Efficiency and Cost Savings

Introduction: The Cost of Using Uniform AI Models

Consider a request that just reached your API:

"Are there any syntax issues here?
prices_usd = {'laptop': 1200, 'mouse': 25, 'monitor': 300}
expensive_items_eur = {k: v * exchange_rate for k, v in prices_usd.items() if v > 50}
print(expensive_items_eur)"

This is a simple syntax check, and the answer is no, there are no issues. Such a task can be efficiently handled by a model priced at $0.10 per million tokens in less than a second.

Now, imagine a more complex request a few seconds later:

"We're migrating our monolith to microservices. The current architecture uses a shared PostgreSQL instance with 47 tables. Identify which tables are safe to split into separate service databases without introducing distributed transaction risk, and propose a phased decomposition strategy."

This query requires deep architectural reasoning and understanding of distributed systems, suitable for a more advanced model.

Most systems today treat both requests the same, routing them to a high-capability model due to the lack of sophisticated infrastructure to differentiate. This approach incurs high costs, as simpler requests are overcharged.

In typical coding sessions, most tasks are simple, such as syntax checks and short explanations, while only a few require advanced reasoning. Using a model priced at $15 per million tokens for all tasks results in unnecessary expenses.

Key Insights

• Costly Single-Model Usage: Routing all requests to a single high-cost model leads to excessive spending, especially when many tasks are simple.
• Smart Model Selection: An automated routing system can match tasks to the most suitable model, optimizing costs.
• High Routing Accuracy: The Inference Router achieves an accuracy of 87.84%, outperforming other models like GPT-5.1.
• Efficient Agentic Loops: Session pinning ensures consistent model usage throughout a session, significantly reducing input token costs.
• Easy Integration: Transitioning to this system involves a simple change in your API call configuration.

Challenges in Current Routing Practices

• Hardcoded Logic: Writing static routing rules in application code is prone to errors and lacks context awareness.
• Classifier Models: Using general-purpose models for routing increases latency and doesn't optimize for the task-specific accuracy.

True Requirements for Effective Routing

To scale routing correctly, consider these essentials:

1. Semantic Intent Resolution: Understand the task's context beyond keyword matching.
2. Live Performance Metrics: Adapt to real-time changes in model cost and latency.
3. Infrastructure-Level Execution: Keep routing logic separate from application code, allowing seamless updates.

Research Background

Research supports using different models for cost and quality balance. Studies like FrugalGPT and RouteLLM demonstrate significant cost savings without compromising on quality by intelligently routing requests.

Implementation of MoE Routing

The Inference Router utilizes a Mixture-of-Experts (MoE) model, specifically designed for routing multi-turn conversations. This model, called Plano-Orchestrator, is available in different sizes to suit various needs.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://inference.do-ai.run/v1",
    api_key=os.environ["MODEL_ACCESS_KEY"]
)

## Before: All requests went to a single model
response = client.chat.completions.create(
    model="openai-gpt-5.2",
    messages=[{"role": "user", "content": prompt}]
)

## After: Requests are routed by complexity
response = client.chat.completions.create(
    model="router:software-engineering",
    messages=[{"role": "user", "content": prompt}]
)

print(response.model)  # e.g., "openai-gpt-oss-120b" or "anthropic-claude-sonnet-4.5"

Advantages of MoE in Routing

The MoE architecture allows only the most relevant sub-networks to activate, optimizing resource use while maintaining high capacity. This makes it ideal for classifying tasks and routing requests efficiently.

Dense vs. Sparse Models

While dense models activate all parameters for each input, MoE models selectively activate only necessary ones, reducing computation costs.

Gating Network Functionality

The gating network in an MoE model determines which experts to activate based on input, ensuring efficient handling of diverse requests.

Bridging Token and Request Routing

In MoE models, tokens are routed to expert sub-networks. Similarly, the Inference Router uses MoE to route entire requests to the appropriate models, enhancing efficiency.

How the Inference Router Operates

Every request undergoes two phases: intent resolution and model ranking, ensuring the most suitable model handles the task.

Intent Resolution

The system matches conversations to configured tasks, outputting a simple JSON routing decision.

Model Ranking

Using live data, the system ranks models based on cost and speed, selecting the most appropriate one for the task.

Deployment and Integration

Setting up involves simple configuration changes, allowing quick integration into existing systems. The session pinning feature helps maintain consistency in agentic loops, optimizing costs.

Performance and Cost Benefits

Routing intelligently across tasks can significantly reduce costs while maintaining high-quality responses. Benchmarks show substantial savings compared to static routing configurations.

Conclusion: The Impact of MoE-Based Routing

Implementing MoE-based routing addresses cost inefficiencies and enhances model selection accuracy. By integrating this approach, systems can achieve substantial cost savings and performance improvements in AI inference tasks.