Creating a Cost-Effective AI Support Triage API with Serverless Inference
Most AI driven applications initially incorporate a single model embedded within the application. This simplistic approach is fine for a prototype but becomes problematic when a...
Most AI-driven applications initially incorporate a single model embedded within the application. This simplistic approach is fine for a prototype but becomes problematic when a single endpoint needs to handle diverse and complex tasks. These tasks—such as classification, urgency scoring, generating customer responses, and summarizing long-form content—require distinct models due to varying cost, latency, and quality demands.
Support triage is a prime example of this complexity. For instance, the cost per token remains the same whether processing a simple query like “how do I reset my password?” or a detailed escalation from an enterprise client. You might branch based on ticket types within your application, selecting different models for each, but this leads to having model selection logic embedded in your handler. This setup results in a rigid try/except fallback strategy and necessitates redeployment with every pricing update. Consequently, you might end up using a 70B model for simple tickets, facing delays when the model is slow, and frequently redeploying.
This tutorial demonstrates how to use serverless inference through an inference router to efficiently build a FastAPI support triage endpoint, resolving these issues. By the end, you'll be able to automatically route tasks to the appropriate model, complete with built-in fallback, and without hardcoding model names in your application. This setup will yield a production-ready API that is up to 71% cheaper than using a singular advanced model for everything.
What You’re Building
You'll create a single endpoint, POST /triage, that processes a ticket payload to return:
- Classification: Categorizes the issue (billing, bug, how-to, account, etc.)
- Urgency + Sentiment: Provides a severity score and assesses customer mood
- Drafted Reply: Crafts a short, customer-focused response
- Escalation Summary: Generates a structured brief for complex tickets
The architecture transitions from:
App → hardcoded model (one model handles every task)
to:
App → Serverless inference via Inference Router → best-fit model per task
The inference router enables this transformation by abstracting model selection from your application.
Serverless Inference and Inference Router
The inference router allows defining tasks and model pools, routing incoming requests to the optimal model based on task definitions and selection policies. A task is a designated job with a description, and a model pool is a set of candidate models that the router can choose from, governed by policies like lowest cost, lowest latency, or a manual hierarchy. Configuration occurs at the router level, meaning your app communicates with the router instead of specific models.
Serverless inference facilitates sending API requests to models without managing infrastructure, allowing quick deployment without handling the components behind an inference endpoint.
Project Setup
To proceed, ensure you have Python 3.10+, an account with serverless inference enabled, and a model access key. A pre-configured project is available, but you can follow along to construct your version and understand the API choices made.
Project structure:
support-triage/
├── main.py
├── sample_tickets.json
├── requirements.txt
└── .env
Clone the repository and install the necessary components:
git clone https://github.com/Jameshskelton/triage_app
cd triage_app
python3 -m venv venv_triage
source venv_triage/bin/activate
pip install -r requirements.txt
Step 1: The Baseline - Direct Model Calls
Initially, developers might hardcode one model to handle all tasks. Below is an example setup using this method:
import os
import re
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(
base_url=os.environ["DO_INFERENCE_BASE_URL"],
api_key=os.environ["DO_MODEL_ACCESS_KEY"],
)
MODEL = "llama3.3-70b-instruct"
app = FastAPI()
class Ticket(BaseModel):
subject: str
body: str
def call_model(system: str, user: str) -> str:
resp = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
)
return resp.choices[0].message.content.strip()
@app.post("/triage")
def triage(ticket: Ticket):
text = f"Subject: {ticket.subject}\n\n{ticket.body}"
category = call_model(
"Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
text,
)
urgency = call_model(
"Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
text,
)
reply = call_model(
"Write a short, professional reply to this customer. Maximum 4 sentences.",
text,
)
summary = call_model(
"Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
text,
)
return {
"category": category,
"urgency": urgency,
"reply": reply,
"escalation_summary": summary,
}
Run it with:
uvicorn main:app --reload
Test it with simple and complex tickets to observe the responses.
Step 2: Configure the Inference Router
In the control panel, navigate to the Inference Router and create a new router with appropriate tasks and model pools. Ensure each task has a clear description and that each pool includes fallback models.
Step 3: Refactor the App to Use the Router
Replace hardcoded model references with the router ID and pass the task name through requests. This setup allows the router to select models dynamically:
ROUTER = "your-router-id"
def parse_urgency(urgency_text: str) -> int:
match = re.search(r"score:\s*(\d)", urgency_text, re.IGNORECASE)
return int(match.group(1)) if match else 3
def call_router(task: str, system: str, user: str) -> dict:
resp = client.chat.completions.create(
model=ROUTER,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
extra_body={"task": task},
)
return {
"content": resp.choices[0].message.content.strip(),
"served_by": resp.model,
}
@app.post("/triage")
def triage(ticket: Ticket):
text = f"Subject: {ticket.subject}\n\n{ticket.body}"
category = call_router(
"classify_ticket",
"Classify this support ticket into one of: billing, bug, how-to, account, other. Reply with one word.",
text,
)
urgency = call_router(
"urgency_detection",
"Score urgency from 1 (low) to 5 (critical) and note sentiment. Reply as 'score: N, sentiment: X'.",
text,
)
reply = call_router(
"draft_customer_reply",
"Write a short, professional reply to this customer. Maximum 4 sentences.",
text,
)
urgency_score = parse_urgency(urgency["content"])
summary = None
if urgency_score >= 4:
summary = call_router(
"escalate_complex_issue",
"Summarize this ticket for a human agent. Include the problem, what's been tried, and recommended next steps.",
text,
)
return {
"category": category["content"],
"urgency": urgency["content"],
"urgency_score": urgency_score,
"reply": reply["content"],
"escalation_summary": summary["content"] if summary else None,
"routing": {
"classify_ticket": category["served_by"],
"urgency_detection": urgency["served_by"],
"draft_customer_reply": reply["served_by"],
"escalate_complex_issue": summary["served_by"] if summary else None,
},
}
Step 4: Run Mixed Tickets Through the Router
Test with a variety of ticket complexities to see the router's efficiency in selecting appropriate models. You can use sample_tickets.json to run tests.
What This Actually Saves You
Evaluate cost savings by comparing the router-based approach with hardcoded models. Using serverless inference rates, the router offers significant savings by efficiently routing tasks to the most suitable models.
Production Checklist
Before deploying, ensure logging, task evaluation, fallback models, and regular updates for routing rules to optimize performance.
Closing Thoughts
The final setup simplifies the application by moving model selection logic to the router. This flexible and efficient approach allows for easy adjustments without code changes. Future enhancements might include streaming replies, integrating summaries into ticketing systems, or deploying additional routers for other workflows.