Analyzing the Cost and Efficiency of Multi-Model Synthesis in Serverless Inference

DigitalOcean has developed a tool for model synthesis that has been tested for cost, latency, and consistency. This evaluation focuses on mechanical aspects such as costs, time efficiency, model disagreement frequency, and answer consistency with the same configurations.

Key Findings

• Cost: Depending on the configuration, costs range from 13× to 93×, averaging 35× for the launched configuration. Specifically, a call costs $0.1928 compared to $0.0055 for GLM-5.2, and about 26× for a single frontier model.
• Latency: The median latency is 216 seconds, with a maximum observed latency of 19 minutes. This is approximately 7.6 times longer than a single model call, with some calls exceeding the typical timeout limits.
• Task-Dependent Disagreement: Model disagreement varied, being nonexistent in tasks with clear evidence, but reaching 64% in tasks with ambiguous evidence.
• Answer Consistency: The synthesized answer matched the judge model's answer 86% of the time, differing from a single panel member only 5% of the time.
• Reproducibility: The launch configuration showed lower reproducibility (82%) compared to a single model (89%), with worse performance in underdetermined tasks (72%).
• Panel Configuration: All three presets run two-model panels, contrary to documentation hints of larger panels.
• Quiet Failures: In 23% of four-model calls and 32% of quality preset calls, a panel member dropped without affecting the API response.

Methodology

The study involved 25 tasks across four categories: infrastructure decisions, policy analysis, incident diagnosis, and open-ended research. Tasks were pre-labeled as determinate or indeterminate. Configurations included various model combinations and repeated tests to assess stability.

Model Disagreement Frequency

Disagreement rates varied significantly with task type. For determinate tasks, models always agreed, while for indeterminate tasks, disagreement occurred 64% of the time. These findings underscore the importance of task specificity when evaluating model agreement.

Impact of Additional Models

The addition of models rarely changed final answers. In cases of genuine disagreement, the final answer often aligned with the judge model's solo response, indicating limited influence of added models on outcome.

Reproducibility Comparison

Multi-model synthesis was less reproducible than single models, with synthesis adding variability through additional stages like sampling and synthesis rewrites. This suggests that for applications needing consistent outputs, single models or presets are preferable.

Cost Analysis

Comparisons reveal that multi-model synthesis incurs significantly higher costs than using a single model, primarily due to additional processing stages.

Latency Considerations

Synthesis calls are slower than single model calls, with the slowest panel model determining the overall call duration. This latency makes synthesis unsuitable for interactive applications.

Preset Composition and Behavior

Preset configurations consist of two-model panels, with cost differences driven more by model caliber than panel size. The judge role is consistently filled by the top-level model.

Silent Failures in Synthesis Calls

Panel failures often occur without signaling, affecting output format degradation and masking the true panel composition from the user.

Deviations from Initial Benchmarks

Initial benchmarks showed better performance due to task-specific conditions. This highlights the importance of matching workload characteristics when evaluating synthesis tools.

Recommendations for Synthesis Use

Multi-model synthesis is most valuable for tasks with significant ambiguity where incorrect answers incur high costs. For routine tasks, single models are more cost-effective.

Limitations and Suggestions for Improvement

The study did not assess answer quality, and the synthesis tool could benefit from exposing panel outputs and documenting preset configurations to improve transparency and reliability.

Future Considerations

Enhancements could include making disagreement signals visible and improving documentation on preset model composition.

---