Turning Database Grading into Reinforcement-Learning Rewards with ThinkingBox

Turning Database Grading into Reinforcement-Learning Rewards with ThinkingBox

Published October 22, 2026

ThinkingBox was designed to evaluate agents by comparing the final state of a database with the expected outcome. The same executable signal can also serve as a reward during reinforcement learning on tasks outside ThinkingBox-Bench.

For Qwen3.8-27B, average success increased from 51.70% to 60.78% after 50 GRPO steps. The number of tasks completed correctly on all 20 attempts increased from 38 to 107.

From database grading to training rewards

ThinkingBox-Bench contains 507 tasks. Each task provides an agent with a real backend, a set of tools, and a simulated user with private context. After the agent works through the task, executable checks inspect the resulting database state.

This approach can detect cases where an agent claims success but leaves the wrong record behind. In one example, an agent closed a customer ticket as resolved even though the related carrier exception remained open. The agent made nine plausible tool calls, but one database row was incorrect.

For most tasks, the executable check produces a binary result directly from the final database state: 1 for success or 0 for failure. That means the grader can also act as a reward function.

The main challenge is selecting useful training tasks. Tasks that a model always solves are already learned, while tasks it always fails may be too difficult to provide a useful learning signal. Tasks that the current policy solves only sometimes can produce the contrast required for GRPO training.

Difficulty is policy-relative: a task is hard for the current model only until that model learns it.

This property is called offline policy-relative difficulty. Each candidate task was evaluated against the base policy before training, and the selected training pool remained fixed for the complete run.

Results after training

The training pool was selected using offline policy-relative difficulty. Qwen3.8-27B was then fine-tuned with GRPO for 50 steps.

Each of the 507 benchmark tasks was evaluated 20 times:

• pass@1 measures average success over all 10,140 attempts.
• pass@20 measures the share of tasks solved at least once.
• Observed 20/20 measures the share of tasks that passed every recorded attempt.

Training substantially increased repeatable success, with a small decline in at-least-once coverage.

One hotel-booking task illustrates the change. A traveler requested a room downgrade, but company policy required manager approval. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After fine-tuning, it preserved the booking and payment state in all 20 attempts. All expected records matched in 19 of those attempts.

What ThinkingBox-Bench measures

ThinkingBox-Bench evaluates the database records an agent leaves behind. Each task starts a backend, provides MCP tools, and uses a simulated LLM user. At the end, executable checks inspect the terminal database state rather than requiring the agent to follow a particular sequence of tool calls.

This allows agents to discover alternative solution paths, provided that the final state is correct.

Earlier benchmark results showed that headline accuracy does not necessarily indicate dependable behavior. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 tasks, or 13.41%, on all 20 attempts. Claude Opus 5 showed narrower coverage but greater consistency.

Model size or general capability did not automatically eliminate this difference. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, but both models passed exactly 241 tasks on all 20 attempts.

For agents that modify production database records, pass@1 should therefore be considered alongside repeatability. Observed 20/20 identifies tasks that succeed on every attempt, which more closely reflects consistent production behavior.

Out-of-distribution benchmark results

The selected step-50 checkpoint was evaluated on all 507 benchmark tasks, none of which were included in RL training. The benchmark domains, databases, and most MCP tools were not encountered during training.

The fine-tuned 27B model increased its observed 20/20 result from 38 tasks, or 7.5%, to 107 tasks, or 21.1%.

Average success and repeatability produced different rankings. GPT-5.6 Sol achieved a slightly higher pass@1 score than the RL fine-tuned Qwen3.8-27B, 61.91% versus 60.78%, but passed fewer tasks on all 20 attempts, 82 versus 107. Claude Sonnet 4.6 had a lower pass@1 score of 59.19% but passed 102 tasks on every attempt.

The results indicate that training sharpened the model's behavior: repeatable success increased while at-least-once coverage declined slightly. One possible explanation is that probability shifted toward trajectories that worked consistently while some lower-probability paths disappeared.

Domain-level changes

Pass@1 and observed 20/20 increased in all five domains. Pass@20 was mixed, increasing in neobank and consulting while declining in retail, travel, and auto insurance.

Auto insurance experienced the largest breadth loss, at -7.0 points. Retail had the largest observed-20/20 gain, at +24.5 points.

Cost and dependability

The evaluation cost analysis divides the estimated cost of the complete 507-task, 20-run campaign by the number of tasks passed on every attempt:

Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20

At $8.00 per dependable task, the fine-tuned model was close to the companion comparison values for GPT-5.4 at $6.80, GPT-6 Astra at $7.45, and Claude Opus 5.5 at $7.80.

These dollar values are a comparative serving index, not invoices or cloud bills. Serving the fine-tuned checkpoint would cost the same as serving the base model. The calculation includes model inference cost but excludes the training run and the policy-probe cost used to curate the training dataset.

The fine-tuned checkpoint's pass@1 increased from 51.70% to 60.78%, while estimated cost per successful attempt decreased from $0.177 to $0.139.

The training loop

The training process has six stages. Candidate tasks are collected, screened, probed against the base policy, classified by estimated difficulty, used to generate rollout groups, and finally scored through terminal-state checks before the GRPO update.

Task selection occurs once, offline, against the base policy, which was Qwen3.8-27B in this run. Training then uses the fixed curated pool rather than re-probing difficulty during the run.

Each rollout is a complete episode rather than a text completion. It receives a fresh backend, an isolated MCP session, and GPT-5 Chat as the simulated user. Most rewards come from deterministic terminal-state checks. A minority of narrow rubric requirements can invoke the configured GPT-5 Chat judge.

Training and evaluation must interpret Qwen output identically. This includes reasoning extraction, tool calls, argument typing, rendering, and isolated task state. Otherwise, the reward may describe a different agent from the one evaluated. The shared interpretation code is in trainer/core/tb_interpret.py.

The reference run used full-parameter RL fine-tuning on 187 tasks with token-mean GRPO. Each update generated 18 groups of 8 rollouts, or 144 episodes, across 24 H100 GPUs. Training ran synchronously with parameter and optimizer offload. AdamW used a 1e-6 learning rate and zero weight decay, and training lasted 50 steps.

Zero-variance groups were replaced using a DAPO-style procedure. System errors were handled separately: a backend timeout was not treated as an incorrect agent outcome, so affected groups were refilled rather than scored as failures.

Offline policy-relative difficulty

No ThinkingBox-Bench task was included in the RL training pool. The benchmark remained reserved for evaluation.

The benchmark and training pool share the ThinkingBox framework, simulated-user machinery, executable terminal-state grading, some common tools, and a similar task style. However, their domains, databases, policies, and most tools are distinct. These shared components limit the scope of the transfer claim.

GRPO benefits from groups containing both successful and unsuccessful rollouts. If all eight rollouts fail, the reward vector is:

[0, 0, 0, 0, 0, 0, 0, 0]

If all eight succeed, it is:

[1, 1, 1, 1, 1, 1, 1, 1]

In both cases, centering within the group produces no task-reward advantage because no trajectory performed better than another. A mixed group such as the following provides a contrast that GRPO can learn from:

[0, 0, 0, 1, 1, 1, 1, 1]

This distinction is costly for a stateful agent because every rollout provisions an isolated backend, runs a real tool session, and calls a separate model to act as the user.

Estimating difficulty from limited probes

Each candidate task is run n times, producing k successes. Its latent success probability is modeled with a Jeffreys posterior:

success probability | k successes in n probes ~ Beta(k + ½, n - k + ½)

The unit interval is divided into five buckets with boundaries at 0.08, 0.40, 0.60, and 0.90. These boundaries were selected as hyperparameters for the runs. Each task is assigned to the bucket containing the greatest posterior mass:

• Rarely solved [0, 0.08): drop
• Difficult [0.08, 0.40): keep
• Intermediate [0.40, 0.60): keep
• Frequently solved [0.60, 0.90): keep for full-parameter RLFT
• Nearly always solved [0.90, 1]: drop

Tasks the base policy always fails are unlikely to teach it immediately. Tasks it always passes are already learned. Tasks it solves intermittently are more likely to produce the contrasting rewards needed for training.

One task from selection to training

For the Qwen3.8-27B full-parameter pool, GPT-5.4 first screened a candidate task for solvability. The base policy then succeeded on 3 of 8 probes, producing a Beta(3.5, 5.5) posterior. The largest posterior mass, 54.0%, fell into the Difficult band.

When sampled during training, the task produced this reward group:

[0, 0, 0, 0, 1, 1, 1, 1]

Successful trajectories received above-mean task advantages, while failed trajectories received below-mean advantages. This gave GRPO a contrast from which to learn. A different policy could place the same task in a different difficulty band.

The final curated Qwen3.8-27B training pool contained 187 tasks.

What earns a reward

The reward is based on the task's executable verdict, not on the appearance of the trajectory.

When an episode ends, the sandbox extracts the changes made to the backend and runs deterministic checks against the terminal state. A trajectory passes only when it produces the correct final state. Wrong, missing, or extra side effects result in failure. The reward is binary: 1 or 0.

There are two reward paths:

1. Executable check: The large majority of tasks use deterministic tests that accept or reject outcomes represented in the terminal state or side effects. No LLM judge call is required.
2. Judge-assisted check: A minority of narrow requirements cannot be determined cleanly from database state. In these cases, the configured LLM judge answers a binary rubric question used by the executable test.

Running the training stack

The data-selection, rollout, reward, and Verl integration used for Qwen3.8-27B are available in the microsoft/thinkingbox-training repository. Its documentation covers data preparation, the reference Qwen3.8 configuration, and checkpoint evaluation.

Requirements

The stack was tested on Linux and WSL with Python 3.12 and uv. It requires NVIDIA GPUs with a compatible CUDA/NCCL stack, a local Qwen3.8-27B checkpoint, and OpenAI-compatible endpoints for the simulated user and judge.

The validated reference configuration uses 3 nodes with 8 H100 GPUs each. Smaller configurations depend on model size, context length, and rollout concurrency.

The public sample task list exercises the interface but is not the private 187-task training pool used for the reported run. An authorized training source that remains separate from the public benchmark is required.

Install the repositories

mkdir -p ~/thinkingbox-workspace
cd ~/thinkingbox-workspace

git clone https://github.com/microsoft/thinkingbox.git
git clone https://github.com/microsoft/thinkingbox-data.git
git clone --branch main https://github.com/microsoft/thinkingbox-training

git -C thinkingbox checkout 892964e7226044e5463ad188da119df838f4bec1
git -C thinkingbox-data checkout thinkingbox-bench-v1.0

Create the validated environment

The released stack targets Linux, Python 3.12, Verl 0.9.0, and the pinned package lock.

cd ~/thinkingbox-workspace/thinkingbox-training

uv venv --python 3.12
source .venv/bin/activate

uv sync --frozen --extra fast --extra tracking --extra eval-proxy
uv pip install -e ../thinkingbox
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/thinkingbox_tools
uv pip install --config-settings editable-mode=compat \
  -e ../thinkingbox-data/servers/tb_business_ops_servers_202606

python scripts/prepare_verl.py --dest .deps/verl --install
python scripts/verify_verl_install.py
uv pip check

The final two commands should report a verified patched Verl checkout and no broken requirements. Another automatic package sync should not be run afterward because it can replace the patched source installation with the standard Verl wheel.

Prepare the training data

Start with the public samples or add an authorized dataset in ThinkingBox format. Then run:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
python data/prepare_data.py --help

The preparation script aggregates policy probes, applies offline policy-relative difficulty filters and benchmark exclusions, and writes the selected train.yaml and test.yaml files.

Inspect the launch configuration

The reference run guide configures Typesense and the MCP servers, the simulated user and judge, the model, task list, checkpoint, and Ray topology. Credentials should be provided through runtime secret injection rather than committed files or shell history.

Before allocating a training job, run a dry run:

DRY_RUN=true ./scripts/run_train.sh \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

The command should exit with status zero without starting Ray or allocating GPUs. The generated python -m verl.trainer.main_ppo command and all resolved overrides should be reviewed before launching training.

Start training

The launcher configures training but does not provision hosts. On the reference three-node topology, the same environment must be available on every node, and CHECKPOINT_DIR must be mounted at the same absolute path.

## head node
ROLE=head NODE_IP="$HEAD_IP" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh

## each worker node
ROLE=worker NODE_IP="$WORKER_IP" \
  HEAD_ADDRESS="$HEAD_IP:6379" GPUS_PER_NODE=8 \
  ./scripts/start_ray.sh

After ray status reports all 24 GPUs, launch the training run from the head node:

export RAY_ADDRESS=auto
export GPUS=0,1,2,3,4,5,6,7

./scripts/run_train.sh \
  trainer.project_name=thinkingbox-training \
  trainer.experiment_name=q38-full-rlft \
  trainer.total_training_steps=50 \
  trainer.save_freq=5

Evaluate the checkpoint through OpenEnv

The checkpoint must be merged according to the evaluation guide and then served with vLLM. The historical checkpoint evaluation used eight A100 GPUs:

vllm serve "$MERGED_MODEL" \
  --served-model-name q38-rlft \
  --host 127.0.0.1 \
  --port 8001 \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --language-model-only \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

A compatibility proxy for the Qwen chat template listens on port 8014 and forwards requests to vLLM on port 8001:

cd ~/thinkingbox-workspace/thinkingbox-training
source .venv/bin/activate
bash scripts/launch_sysmerge.sh

Check readiness with:

curl --fail http://127.0.0.1:8014/readyz

The agent entry in thinkingbox.yaml should point to the proxy while retaining the MCP, simulated-user, and judge configuration:

orchestrator:
  agent_model:
    type: aoai
    endpoint_url: http://127.0.0.1:8014/v1/chat/completions
    deployment: q38-rlft
    credential:
      type: api-key
      api_key: <your_api_key>
    is_reasoning: true
    temperature: 1.0
    top_k: 20
    top_p: 0.95
    max_completion_tokens: 8192
    timeout: 1800

Once OpenEnv is ready, run the complete evaluation protocol of 507 tasks across 20 attempts each:

uv run --project envs/thinkingbox_env --frozen \
  thinkingbox-eval \
  --config "$PWD/thinkingbox.yaml" \
  --base-url http://127.0.0.1:8000 \
  --dataset "$THINKINGBOX_DATA" \
  --output q38-rlft-full.jsonl \
  --errors-output q38-rlft-full-errors.jsonl \
  --repeat 20 \
  --coverage-profile full \
  --aggregate-output q38-rlft-full-aggregate.json \
  --message-timeout 1800

The full profile validates all 10,140 expected task-and-repetition keys before aggregation.

Implications for agent training

The main result is not a single Qwen3.8-27B checkpoint. It is the training loop and task-curation process that turns executable database outcomes into reinforcement-learning rewards.

For agents that modify database records:

1. Probe before training. Run the base policy against the task set n times and inspect the distribution of success probabilities. This identifies whether the pool is dominated by tasks that are already solved or currently unreachable.
2. Verify parser parity. Replay the same recorded model output through the training and evaluation parsers and confirm that both extract identical tool names and arguments.
3. Report coverage and repeatability. State how many tasks pass at least once in k attempts and how many pass all k attempts. Occasional success is not the same as consistent success.

ThinkingBox code is MIT-licensed, and the benchmark data is licensed under CDLA-Permissive-2.0. Every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on enterprise agent patterns, and the customers are not real.

Paper citation

@article{li2026thinkingbox,
  title   = {One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows},
  author  = {Li, Zhuochun and Ko, Youngmin and Keramati, Ali and Kundu, Tuhin and
             Tsai, Liang-Chun and Ferri, Nicola and Milletari, Mirco and Liu, Jiaxiang and
             Lopez Pelaez, Susana Palmaz and Wang, Yuepeng and Smolyakov, Vadim and
             Jiang, Xiang and Olafsson, Kjartan and Guy, Tommy},
  journal = {arXiv preprint arXiv:2608.19741},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.19741}
}