Skip to main content
Back to Blog
AI/MLCloud ComputingProduct Development
14 August 20264 min readUpdated 14 August 2026

Creating a Customer Support RAG Assistant on a Cloud Platform

When you ask a language model something specific about your product, it responds quickly and fluently, but sometimes the information is incorrect. This issue was the starting po...

Creating a Customer Support RAG Assistant on a Cloud Platform

When you ask a language model something specific about your product, it responds quickly and fluently, but sometimes the information is incorrect. This issue was the starting point for a workshop focused on improving the accuracy of such responses. On a particular day, a live session was held to construct a customer support assistant, named HelpBot, using a cloud-based AI platform. Initially, HelpBot was as uninformed as any standard model, but step-by-step modifications were implemented to enhance its capabilities.

Key Workshop Topics:

  • Identifying the Problem: Demonstrations showed HelpBot providing incorrect answers about a product, highlighting the need for improvement.
  • RAG and Knowledge Bases: By integrating our documentation, HelpBot began to provide answers based on real content rather than guesswork, utilizing specific document sections.
  • Inference Router: A router was used to select the best model for each request, based on criteria like cost and speed. This setup allowed for automatic failover without needing code changes.
  • Guardrails: Safety measures were added to prevent leaks of personal information, block unauthorized access attempts, and manage inappropriate content.
  • Evaluations: An additional AI model evaluated HelpBot's responses for accuracy and safety, enabling automated quality checks.
  • Final Integration: The entire system was tested with the original query to demonstrate improvements.

By the end of the workshop, attendees witnessed the transformation of the assistant from a basic command to a deployable solution. Prior experience with calling a language model API is sufficient to follow along, with no advanced AI background needed. Detailed instructions are available through both video and written guides.

Preparations Before Starting:

  1. Account Setup: Ensure you have access to a cloud account with inference capabilities and a model access key.
  2. Documents for Knowledge Base: Have a small set of documents ready for the assistant to reference.
  3. Optional API Token: For automation, an API token with specific permissions may be needed.

Step-by-Step Guide:

Step 1: Orientation and Initial Inference Call

Objective: Understand the platform and make an authenticated inference call.

  1. Confirm your model access key in the Console.
  2. Execute your first call using a standard payload format.
curl https://inference.do-ai.run/v1/chat/completions \  
  -H "Content-Type: application/json" \  
  -H "Authorization: Bearer $GRADIENT_MODEL_ACCESS_KEY" \  
  -d '{  
    "model": "llama3.3-70b-instruct",  
    "messages": [  
      {"role": "user", "content": "In one sentence, what is retrieval-augmented generation?"}  
    ]  
  }'

Step 2: Establishing RAG and Knowledge Bases

Objective: Create a knowledge base, understand document processing, and validate retrieval methods.

  1. Navigate to the Knowledge Base section to create a new base.
  2. Choose a data source, ensuring the content is relevant.
  3. Configure chunking and embedding options before creating the knowledge base.
  4. Use the RAG Playground to test retrieval and ensure accurate answers.

Step 3: Utilizing the Inference Router

Objective: Implement a router to enhance resilience, cost-effectiveness, and latency management.

  1. Set up a router with tasks, selection policies, and a model pool.
  2. Use the router as a replacement for direct model calls.

Step 4: Implementing Guardrails

Objective: Add security features to the assistant to handle sensitive data and content moderation.

  1. Attach guardrails to the agent for monitoring both prompts and responses.
  2. Customize and test these guardrails in the agent playground.

Step 5: Evaluations for Quality Assurance

Objective: Use an AI model to evaluate the assistant’s responses against predefined metrics.

  1. Prepare an evaluation dataset and upload it.
  2. Run evaluations and analyze results to ensure quality standards are met.

Step 6: Finalizing and Deploying the Assistant

Objective: Integrate all components and deploy the assistant with a robust endpoint.

  1. Set up the agent’s API for external access.
  2. Test the full functionality, including guardrails and evaluations, through the final endpoint.

Building Your Own AI Assistant

The cloud platform provides all necessary tools for creating a robust AI assistant, including serverless inference, model routing, and configurable safety measures, all within an integrated system.