Roulette Mode: Switching Language Models in mini-SWE-agent

What happens when an agent uses a different language model on every turn? In an experiment with mini-SWE-agent, randomly switching between GPT-5 and Sonnet 4 produced a higher SWE-bench score than either model achieved separately.

The experiment used the same setup described in an earlier evaluation: mini-SWE-agent, a minimal software engineering agent with no tools other than Bash. This configuration also forms the basis of the SWE-bench bash-only leaderboard.

What is the mini agent?

The mini agent was designed to be as minimal as possible. Its main agent class contains fewer than 100 lines of code, and the same implementation is used for the SWE-bench leaderboard.

Compared with swe-agent, mini has several notable properties:

• Bash is its only tool: The agent does not use the language models' tool-calling interfaces, so it can run with virtually any model. In sandboxed environments, it also requires no package installation beyond Bash.
• Its history is completely linear: Each agent step appends to the message history. The resulting trajectory is therefore identical to the messages sent to the language model, which simplifies debugging and fine-tuning.
• Actions run through subprocess.run: Each action is independent rather than relying on a persistent shell session. This makes sandbox execution straightforward, since subprocess.run can be replaced with an equivalent command such as docker exec, and supports scaling the agent across many runs.

The model-selection change is equally small. Instead of calling:

model.query(history)

the agent calls:

random.choice([model1, model2]).query(history)

The prompts remain unchanged apart from configuring the models.

This approach produced a higher SWE-bench score than either GPT-5 or Sonnet 4 achieved independently.

Cost analysis

The total cost fell between the costs of the two models, reaching approximately 30 cents per instance at maximum performance.

Cost comparisons for agents can be difficult because much of the spending comes from instances the agent cannot solve. As a result, the average cost depends heavily on runtime limits, including the maximum number of steps.

Performance gains became marginal at a limit of approximately 50 steps, even though Sonnet 4 alone continued to improve slowly toward its maximum. The resulting curve more closely resembled the GPT-5 curve. One possible explanation is that either model can end a run by submitting a solution, making the combined behavior closer to the model that submits earlier.

Average cost followed a sigmoid-like curve as the step limit increased, reaching roughly 30 cents per instance at maximum performance. This placed the cost near the midpoint of the two individual models.

More models and experiments

Additional experiments were run on a smaller sample of 50 randomly selected instances from SWE-bench Verified. The GPT-5 and Sonnet 4 combination produced the most notable result and was the only combination to score higher than both of its component models. These were also the two models involved in the closest head-to-head comparison.

When models with larger performance differences were combined, both the score and cost generally fell between the individual results. Examples included GPT-5 with Gemini 2.5 Pro, and GPT-5 mini with GPT-5 nano. None of these combinations appeared especially relevant in practice.

Combined models

Individual-model baselines

These results should be interpreted cautiously because the sample contained only 50 instances. It was not fully representative of the complete 500-instance evaluation, and the randomly selected instances appeared to be somewhat easier. Consequently, the scores were generally higher than the results on all 500 instances.

For GPT-5 and Sonnet 4, a similar experiment alternated between the two models instead of switching randomly. It solved 333 instances, or 66.6%, again exceeding the score of either model alone.

The combinations involving Gemini 2.5 Pro were also tested on slightly larger samples, but they did not show notable improvements.

Running the experiment

A wrapper model class for roulette-style model selection is available in the mini-SWE-agent project. SWE-bench evaluation can use it by selecting the swebench_roulette configuration:

mini-extra swebench \
  --subset verified \
  --split test \
  --shuffle \
  -o roulette-sonnet4-gpt5 \
  --workers 20 \
  -c swebench_roulette