Transitioning from OpenAI API to DigitalOcean's Serverless Inference

DigitalOcean's Serverless Inference offers API endpoints compatible with OpenAI, allowing many current OpenAI SDK workflows to transition with minimal changes. For a basic Chat Completions request, you can retain the OpenAI Python SDK and simply adjust the base URL, API credentials, and model ID. Not all OpenAI APIs are supported, and some features may behave differently depending on the model. This guide first presents working code, then details the supported endpoints, limitations, and unsupported features.

OpenAI to DigitalOcean Serverless Inference: Code Comparison

Original OpenAI Code

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY"),
)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a fun fact about octopuses."},
    ],
)

print(resp.choices[0].message.content)

DigitalOcean Serverless Inference Code

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.do-ai.run/v1",
    api_key=os.getenv("MODEL_ACCESS_KEY"),
)

resp = client.chat.completions.create(
    model="llama3.3-70b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me a fun fact about octopuses."},
    ],
)

print(resp.choices[0].message.content)

Changes Between Examples

| Item | OpenAI | DigitalOcean Serverless Inference | |-----------------------|------------------------------|---------------------------------------| | base_url | Default (not set) | https://inference.do-ai.run/v1 | | Credential used | OpenAI API key | DigitalOcean model access key | | Example env variable | OPENAI_API_KEY | MODEL_ACCESS_KEY | | Model ID | gpt-4o | llama3.3-70b-instruct | | Auth scheme | Bearer token | Bearer token (unchanged) | | SDK method used | client.chat.completions.create() | client.chat.completions.create() (unchanged) | | Message format | Role-based list | Role-based list (unchanged) |

Credential Acquisition: Obtain the model access key via the DigitalOcean Control Panel under Inference > Serverless Inference. This differs from the OpenAI dashboard key.

The authentication process uses a Bearer token, requiring you to replace the OpenAI API key with a DigitalOcean model access key or a supported personal access token. Model IDs are unique to DigitalOcean. The catalog ID llama3.3-70b-instruct is not interchangeable with OpenAI names. Check the Model Catalog or use GET /v1/models for model details.

Model behavior and request parameters may differ between models and endpoints. Verify current documentation before deploying advanced parameters.

Streaming Example

stream = client.chat.completions.create(
    model="llama3.3-70b-instruct",
    messages=[{"role": "user", "content": "Write a haiku about Kubernetes."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Raw HTTP Call

curl -X POST https://inference.do-ai.run/v1/chat/completions \
  -H "Authorization: Bearer $MODEL_ACCESS_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.3-70b-instruct",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 0.7,
    "max_completion_tokens": 256
  }'

Supported OpenAI-Compatible Endpoints

The following endpoints are pertinent for those transitioning from OpenAI:

• Chat Completions: /v1/chat/completions - Supported. Tool support varies by model and API surface.
• Responses API: /v1/responses - Supported. Not all OpenAI features have equivalents.
• Embeddings: /v1/embeddings - Supported. Suitable for semantic search.
• Image Generation: /v1/images/generations - Supported. Images returned as base64.
• Batch Inference: /v1/batches - Supported separately. Asynchronous workflow, not real-time.

Endpoint Details

• Chat Completions: Tool support depends on the model. OpenAI and Anthropic models are accessible through this API, but compatibility varies. Verify each model's supported API surface.
• Responses API: Supports text, multimodal responses, and prompt caching. Check for feature compatibility.
• Batch Inference: An asynchronous workflow that differs from real-time endpoints. It accepts inputs formatted for OpenAI or Anthropic APIs and uses separate rate limits. Check documentation before migrating jobs.

Unsupported OpenAI APIs

The following OpenAI endpoints lack equivalents in the DigitalOcean Serverless Inference API:

• Assistants API: Not supported.
• Threads API: Not supported.
• Fine-tuning: Not supported.
• Moderation: Not supported.

DigitalOcean offers separate agent-building tools, which are not direct substitutes for OpenAI Assistants code.

DigitalOcean Serverless Inference as a Replacement

Summary: DigitalOcean's solution can act as a near drop-in replacement for basic Chat Completions calls but not for the entire OpenAI API suite.

Key Differences

| Aspect | OpenAI | DigitalOcean Serverless Inference | |-------------------------|---------------------------------|----------------------------------------| | Model IDs | OpenAI model names | DigitalOcean catalog IDs | | Credential | OpenAI sk- key | DigitalOcean model access key | | Rate and usage limits | Based on OpenAI account tier | Varies by DigitalOcean account tier | | Access to commercial models | Based on OpenAI plan | Varies by account tier and model | | "OpenAI-compatible" | N/A | Refers to request/response format |

For comprehensive endpoint details, refer to the DigitalOcean API documentation.

Conclusion

Switching to DigitalOcean Serverless Inference for basic Chat Completions involves changing the base URL, credentials, and model ID. Beyond these adjustments, consider it OpenAI-compatible in format, not feature parity. Verify model IDs, supported parameters, and tools before deploying in production environments.