Understanding Token Economics for GPUs in AI Inference

Introduction

The cost of running large language models (LLM) on dedicated GPUs is influenced by two main factors: the hourly rate of the GPU and the number of tokens it can process per hour, considering its actual usage. While benchmarks often show maximum performance under full load, operators still pay the hourly rate regardless of how much the GPU is being utilized. For instance, a GPU on a cloud service might cost $3.44 per hour, and this charge applies whether the GPU is fully utilized or idle. The effective cost per million tokens is influenced more by traffic patterns than by the hardware specifications alone.

This article outlines a cost framework for using the llama3.3-70b-instruct FP8 model on a specific GPU setup. It explains how to calculate costs based on different traffic profiles and examines factors like batch size and quantization. All calculations are detailed step-by-step for reproducibility.

Note on Environment: The performance data presented here was obtained using a specific configuration. Your results may vary depending on different models, hardware, or system loads.

Key Takeaways

• The cost of LLM inference on dedicated GPUs is more about GPU utilization than hardware specs. At 73% utilization, a specific GPU setup costs $0.643 per million tokens, comparable to serverless options costing $0.65 per million. At lower utilization, costs escalate.
• There is a utilization threshold where dedicated GPUs become more cost-effective than serverless solutions. For example, below 72.2% utilization, serverless is cheaper, while above it, dedicated setups save costs.
• The shape of the traffic significantly impacts costs. Sustained workloads at 73% utilization can be cost-effective, while bursty patterns with lower average utilization can increase costs substantially.
• Batch size greatly affects token cost per million, independent of hardware choice. Small batch sizes lead to underutilization and higher costs, while larger batches can make dedicated setups more competitive.
• FP8 quantization reduces the hardware footprint needed for llama3.3-70b-instruct, allowing for more efficient GPU use.

The Core Cost Formula for Dedicated GPU Inference

The following formula calculates the cost of using dedicated GPUs:

effective_cpm = (hourly_rate ÷ (peak_tps × 3600 × utilization)) × 1,000,000

Here, effective_cpm is the cost per million tokens, hourly_rate is the GPU's price per hour, peak_tps is the maximum token throughput, and utilization is the percentage of the GPU's capacity that is used for work.

Define Your Inputs: GPU Rate, Throughput, and Utilization

To apply the formula, you'll need:

• GPU rate: The hourly cost for a GPU in a specific setup.
• Peak throughput: The number of tokens the GPU can process per second at full capacity.
• Utilization: The average fraction of the GPU's capacity used over time.

Measure Token Throughput at Target Utilization

The output tokens per second vary based on utilization:

effective_tps = peak_tps × utilization

Derive Cost per Token and Cost per Million Tokens

Using the inputs, calculate the cost per million tokens.

Example Calculation:

• Inputs:
  • GPU rate: $3.44/hr
  • Throughput: 2,036 tokens per second
  • Utilization: 73%

Steps:

1. Calculate effective tokens per second at target utilization.
2. Determine tokens processed per hour.
3. Compute cost per token.
4. Calculate cost per million tokens.

GPU Utilization as a Key Factor

Utilization is a primary variable in determining costs. It reflects how well the GPU's capacity is leveraged over time.

Cost per Token Below the Crossover Threshold

When utilization is below the threshold, costs exceed serverless alternatives.

Utilization-Latency Tradeoff

Higher utilization often requires larger batch sizes, which can increase latency, affecting real-time applications.

Reference Table: Cost per Million Tokens by Utilization Rate

The table below shows how costs vary with utilization:

| Utilization | Effective cost per 1M tokens | Comparison to Serverless | |-------------|------------------------------|--------------------------| | 100% | $0.469 | 28% cheaper | | 80% | $0.587 | 10% cheaper | | 40% | $1.173 | 80% more expensive |

Traffic Profile Changes Everything

Different traffic patterns lead to varying costs, even with the same hourly GPU rate.

Modeling a Sustained-Traffic Workload

A consistent workload with predictable demand can achieve cost parity with serverless solutions.

Modeling a Bursty-Traffic Workload

Variable workloads with low off-peak utilization can significantly increase costs.

Conclusion

This framework helps evaluate the cost-effectiveness of dedicated versus serverless GPU setups. By analyzing traffic patterns and utilization, operators can make informed decisions about infrastructure choices.