AI Safety Evaluations Must Test Deployed Systems, Not Just Base Models

AI practitioners understand that a deployed system can behave differently from its base model. Behavior may change when teams modify the system prompt, add retrieval layers, apply fine-tuning, or introduce other components. Yet safety evaluations often assume that results transfer unchanged, with teams citing a base model’s safety score from its system card when assessing a system built on top of it.

This creates a contradiction: teams expect behavior to change during development, but may still make purchasing and deployment decisions as though the base model’s safety score remains valid after modification.

What system composability means

System composability describes whether a configuration or evaluation result remains valid in a different context. A home baker, for example, understands that “30 minutes at 350°F,” whether provided in a recipe or printed on a cake-mix box, is guidance rather than a guarantee. A different oven, pan, or level of humidity can change the result, so the baker checks the cake with a toothpick instead of relying only on the stated time.

AI researchers use composability to describe how reliably evaluation results for an upstream system carry over to systems built on it. The assumption generally held in classical software. In AI safety, however, results often do not transfer reliably without testing the modified system.

Safety scores can shift in either direction

Recent research supports the practical observation that AI safety is not necessarily composable. CDT’s AI Governance Lab and MIT’s Algorithmic Alignment Group compared publicly available medical and legal fine-tunes with their base models. Some fine-tuned models scored as safer than their base models, while others scored as less safe. No single fine-tuning choice predicted the direction of the change, and the amount of modification did not predict how much the safety score would change.

Some modifications can reduce safety relative to the base model. This is the subject of the paper “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!”. The reverse can also occur. Adding runtime guardrails to an open-weight model generally improves safety scores, for example, but the size and nature of that improvement are difficult to determine without testing.

Measurement must account for uncertainty

The evaluation process itself can introduce noise. As Solomon Messing showed in a recent paper, many widely used benchmarks are too noisy to support reliable comparisons between pre- and post-deployment systems, changes over time, or different systems.

A comparison is useful only when it can distinguish real differences between systems from variation introduced by the evaluation. Sources of that variation can include the selected test prompts, the wording of those prompts, and the evaluator that grades the responses. Without a normalized test, a changed score may reflect actual movement in system safety or only measurement noise.

MLCommons’ safety benchmarks are designed to support comparability. Its changes to the evaluator include using classifiers instead of LLMs as judges and applying Item Response Theory (IRT) to benchmark content development. These methods support defensible margins of error.

The November releases of AILuminate Safety and Jailbreak will include margins of error that account for variance across the evaluation pipeline. MLCommons also plans to identify which sources of variance those margins cover. Its AIRR working groups include participants from more than 80 organizations, including AI developers, consultancies, universities, and civil-society groups. Many of these participants develop and operate their own benchmarks.

With comparable evaluations and margins of error, a deployer can more reliably determine whether a system change caused a safety score to change.

Making composability testable

The defensible position today is based on observation: evaluation results do not necessarily transfer from a base model to a deployed system after certain changes. Once measurement uncertainty is understood, researchers can test which modifications preserve an evaluation result, which ones invalidate it, and whether those outcomes can be predicted in advance.

These findings would benefit teams that already test their own systems. If certain types of changes are known to preserve a particular safety score, teams could make more informed decisions about which changes require no further testing, which deserve a low-cost smoke test, and which require a full evaluation with an annotated report.

If a base model’s score reliably predicted the scores of systems built on it, teams could use upstream results when selecting a model and determine how much modification could be introduced before those results stopped applying. Reliable and comparable scoring could also help regulators distinguish developers’ obligations from deployers’ obligations, while reducing the measurement and compliance burden for organizations that deploy AI systems and for the groups that assure them.

Testing the complete deployment

Because organizations deploy complete systems rather than isolated base models, safety testing needs to cover the complete deployment. That includes the base model, system prompt, safeguards, retrieval components, tools, and filters.

MLCommons’ Candela benchmarking-as-a-service product applies AILuminate measurement to this complete assembly in a private tenant. Testing the deployed configuration helps organizations evaluate the system they actually operate instead of assuming that the base model’s score applies unchanged.