Evaluating AI Platform Security: Insights from Developing a Private-Document AI App

Why AI Workloads Demand Unique Platform Security Considerations

Security claims are common among AI platforms, with many asserting a high level of commitment. However, the real challenge arises when these platforms handle sensitive data such as internal documents, source code, or customer information. Prior to transmitting sensitive information through these platforms, it is crucial to determine what security measures can be personally verified. To address this, a private-document chatbot was developed, tested across various platforms, performing tasks such as document upload, local indexing, querying, and deletion. This hands-on approach was combined with a review of security documentation, emphasizing access controls, data retention, isolation, logging, disclosure, and shared responsibility.

Methodology: Combining Research with Practical Verification

Six platforms were scrutinized: DigitalOcean, Baseten, Nebius, Fireworks AI, Modal, and Together AI. Each was evaluated through documentation review and practical application of a private-document workflow. The workflow involved uploading a document, indexing it locally, querying it, and subsequently deleting it. A canary phrase was used to verify if responses were genuinely based on the uploaded content, and a deletion marker checked if the document still influenced responses post-deletion. Only selected document chunks, rather than the entire document, were sent to inference providers to maintain focus on platform inference rather than retrieval capabilities.

Vital Security Aspects

Six key security questions were identified:

• What credentials are provided, and how narrowly can they be scoped?
• What data is stored post-request: prompts, outputs, logs, or none?
• Is workload isolation possible without sales interaction?
• What logs are accessible to self-service users?
• Does the platform allow for external scrutiny via bug bounty or disclosure programs?
• Is the shared responsibility model clear and actionable?

These considerations are more practical than compliance checklists, as they directly relate to verifiable security measures before data transmission.

Overview of AI Inference Platform Security

Six platforms were assessed: DigitalOcean, Baseten, Nebius, Fireworks AI, Modal, and Together AI. Here are the findings from documentation and initial testing:

DigitalOcean

DigitalOcean combines a cloud platform with AI inference capabilities, offering features like self-service VPCs and a mature bug bounty program. Its shared responsibility documentation is detailed, with specific guidance for each major service, making deployment decisions clearer. However, DigitalOcean lacks a company-level ISO 27001 certification, and its data retention policies vary depending on the model used.

Baseten

Baseten stands out by not storing inference data by default, unlike other platforms where retention settings must be adjusted. Its API keys support a robust least-privilege model, allowing for fine-grained access control. While it lacks a public bug bounty program and ISO 27001 certification, Baseten offers a clear escalation path from shared to dedicated infrastructure.

Nebius

Nebius excels in compliance, boasting a broad certification portfolio. Its governance infrastructure is well-documented, providing a clear shared responsibility model. However, by default, Nebius's Token Factory retains data unless zero data retention is explicitly enabled, which is contrary to default privacy expectations.

Fireworks AI

Fireworks AI defaults to zero data retention for standard inference, although its Response API retains data for 30 days unless specified otherwise. While it offers strong isolation options, audit logs are only available to enterprise users, creating a gap for smaller teams needing visibility.

Modal

Modal uses gVisor for enhanced workload isolation but does not support BYOC or customer VPC options. Its impressive sandboxing capabilities are offset by a limited compliance portfolio and architectural constraints.

Together AI

Together AI provides explicit encryption specifications and a public VPC deployment model. However, its default data retention settings require users to opt-out, and while it documents a data loss prevention solution, practical verification is limited.

Security Touchpoints

Secrets and Identity

Platforms were evaluated on credential accessibility, scope, validation capabilities, and identity control maturity, including SSO, RBAC, and MFA.

Data Flow, Retention, and Deletion

The focus was on whether zero data retention was the default or an option, influencing how providers stored data post-inference.

Network Isolation and Deployment Control

Consideration was given to the accessibility of network controls and the ability to deploy workloads without requiring sales negotiations.

Logging, Auditability, and Evidence

The depth and accessibility of logging varied, with some platforms offering audit capabilities only at the enterprise level.

Vulnerability Disclosure and Security Transparency

Platforms with public bug bounty programs, like DigitalOcean and Modal, demonstrated greater transparency and willingness to invite external scrutiny.

Conclusion: Practical Security Verification

In practical tests, the priority shifted from documentation to real-world usability, focusing on key validation, request-level evidence, and ease of setup. DigitalOcean emerged as operationally trustworthy, offering accessible security features and clear shared responsibility documentation. In contrast, some providers with strong documentation faced challenges in practical verification.