The follow-up post is by Tuhin Kundu and Mirco Milletari. It covers three things: a wider model comparison, a cost model built around dependable runs, and a reinforcement learning experiment. All figures below come from Microsoft's post unless I attribute them elsewhere.
What ThinkingBox actually checks
ThinkingBox runs an agent in an isolated, stateful tool environment. A simulated user answers the agent's questions. Then the framework inspects the records the agent left behind instead of trusting its words. The accompanying arXiv paper, titled "One Success Isn't Reliability," describes the setup as multi-turn interactions among a simulated user, an LLM agent, and isolated MCP-compatible tools, evaluated on terminal state, side effects, and dialogue with executable judges.
The benchmark covers 507 workflows across retail, auto insurance, travel, neobank IT and consulting support. Each task runs 20 times per model, which gives 10,140 attempts per model. When the framework first appeared in early October, coverage described it as released through the OpenEnv interface on Hugging Face, with the dataset and evaluation harness under permissive licenses.
Microsoft opens with two failure stories that run in opposite directions:
- The overreach. A traveler asks to change a $1,450 hotel booking from a suite to a deluxe room. Policy says a manager must approve first. In 15 of 20 attempts, Qwen3.8-27B made the change anyway. It issued a refund, marked the ticket solved and told the traveler it was done.
- The phantom change. In the first post, an agent said a quiet-room request had been added, but the special-requests field stayed empty.
Failed runs that looked finished
Across 12 models and 121,680 valid trials, 79,853 failed the executable check. In 67.24% of those failures, the agent ended with a completion-style reply, made at least one state-changing tool call, and saw no explicit error in its last tool response. About two in three failures would therefore pass a transcript-level review.
The database told a different story. These categories overlap and don't sum to 100%:
- 98.95% of failures left a final database that differed from the expected one.
- 77.61% had a wrong field value.
- 43.30% included a change nobody requested.
- 25.36% omitted a required change.
The paper's authors say the same thing in the abstract. Many failed trials terminate cleanly after valid state-changing actions, so response- or tool-call-level signals poorly proxy end-to-end completion.
Each failed run also gets one dominant failure label. Averaged across models, 79.9% were tool-usage failures. In those cases the agent hit a tool error, a failed precondition or an empty lookup and never recovered, sometimes carrying on as if the action had succeeded.
Microsoft cautions that these labels describe what the trace shows, not a proven root cause. It suggests three mitigations, and says it has not measured how much each one helps:
- Classify tool and system errors so retries target the recoverable ones.
- Keep the tool surface limited to what the workflow needs.
- Require human approval for changes that are expensive to reverse.
Average success versus repeatable success
The post reports three views of each model:
- Pass@1: the share of all attempts that passed.
- At least once: tasks passed on one or more of 20 attempts.
- Pass^20: tasks passed on all 20 attempts.
Selected results (overall pass@1, then tasks passed on all 20 out of 507):
| Model | Pass@1 | Pass^20 tasks |
|---|---|---|
| Claude Opus 5.5 | 67.16% | 241 |
| Claude Opus 5 | 66.50% | 241 |
| GPT-5.4 | 65.36% | 128 |
| GPT-5.6 Sol | 61.91% | 82 |
| Qwen3.8-27B (after RL) | 60.78% | 107 |
| GPT-6 Astra | 58.31% | 231 |
| Kimi-K3 | 57.37% | 68 |
| Qwen3.8-27B (base) | 51.70% | 38 |
Claude Opus 5.5 was evaluated for the blog post but does not appear in the paper.
The rankings shift depending on which column you read:
- Kimi-K3 passed 476 tasks at least once, the most of any model shown, but only 68 on every attempt.
- Claude Opus 5 passed 401 at least once and 241 on every attempt.
- GPT-6 Astra trails GPT-5.4 by about seven points on pass@1. It still passed 231 tasks every time, against 128 for GPT-5.4.
The paper makes the same point. It reports that Claude Opus 5 falls from 66.50% pass@1 to 47.53% pass^20, and Kimi-K3 from 57.37% to 17.60%. That Kimi-K3 figure is a percentage on its own measure. The blog's 68-of-507 count is 13.41%, so the two may not use identical denominators. I couldn't reconcile the difference from the material available.
Microsoft's practical framing is sensible. If a person reviews every result, a model that eventually finds an answer may be fine. If the agent edits production records unattended, pass^20 is closer to what you need. It is still only an observation over 20 attempts, not a guarantee about attempt 21.
What a dependable task costs
Microsoft estimates the agent inference cost of the whole 507 x 20 campaign, failed attempts included. It then divides by either successful attempts or all-20 tasks. Prices use the cheapest eligible OpenRouter rates as of 2026-09-20. The estimates exclude the simulated user, the judge, infrastructure, negotiated rates and downstream failure costs. They are not cloud bills.
| Model | $ per success | $ per all-20 task |
|---|---|---|
| GPT-5.6 Sol | $0.127 | $9.76 |
| GPT-5.4 | $0.131 | $6.80 |
| Claude Opus 5.5 | $0.276 | $7.80 |
| GPT-6 Astra | $0.291 | $7.45 |
| Kimi-K3 | $0.242 | $20.68 |
| Claude Opus 5 | $0.475 | $13.30 |
| Qwen3.8-27B (base) | $0.177 | $24.36 |
Two readings stand out:
- The frontier moves. On cost per success, the Pareto frontier is GPT-5.6 Sol, GPT-5.4 and Claude Opus 5.5. Price per all-20 task instead, and GPT-5.6 Sol drops off while GPT-6 Astra joins.
- Cheap attempts can still add up to expensive reliability. Base Qwen3.8-27B is inexpensive per success but costs $24.36 for each task it passes every time.
Training on the same check
The check returns a 1 or a 0. That is already the shape of a reinforcement learning reward. For most tasks the verdict comes from deterministic final-state checks. A minority add a narrow yes/no judge question for requirements a database field can't capture, such as telling a customer a preference isn't guaranteed.
The team used GRPO, which scores each attempt relative to its group. With eight attempts and a binary reward, an all-fail or all-pass group gives no learning signal. Only mixed groups do. Each attempt is a full episode with its own backend, tool session and simulated user, so wasted groups are expensive.
To avoid that waste, the team selected tasks once, offline, against the base model:
- GPT-5.4 attempted each candidate as a solvability check.
- Base Qwen3.8-27B ran each candidate several times.
- A Jeffreys posterior, Beta(k + 1/2, n - k + 1/2), placed each task in one of five difficulty bands.
They kept tasks in the three middle bands, between 0.08 and 0.90. That left 187 training tasks. Microsoft's example is a task solved in 3 of 8 probes. It gets a Beta(3.5, 5.5) posterior with 54.0% of its mass in the Difficult band, so it stays in the pool.
Other engineering details:
- Zero-variance groups were replaced at batch time, following the DAPO approach.
- Groups lost to system errors were refilled, so a backend timeout was never scored as a wrong answer.
- Training and evaluation use the same code to extract reasoning, parse tool calls, type arguments and render conversations. Otherwise the reward would measure a different agent from the one being evaluated.
- Training was full-parameter, token-mean GRPO on 24 H100 GPUs. Each step ran 18 groups of eight attempts.
- Hyperparameters: learning rate 1e-6, no KL penalty, zero weight decay.
- GPT-5 Chat served as the simulated user and judge in training. The benchmark evaluation used GPT-5.4-mini.
- Results are for checkpoint 50.
The results
| Measure | Base | After RL | Change |
|---|---|---|---|
| Pass@1 | 51.70% | 60.78% | +9.08 pt |
| Pass^20 tasks | 38 | 107 | +69 |
| At least once | 453 | 446 | -7 |
| $ per success | $0.177 | $0.139 | -21.4% |
| $ per all-20 task | $24.36 | $8.00 | -67.2% |
No benchmark task was used in training. Training tasks used different MCP servers, data structures, domains and policies. Microsoft calls the benchmark out-of-distribution, but concedes the shared sandbox, tool setup and task style could allow some overlap.
The model got more repeatable, not broader. Microsoft says this is consistent with "sharpening": probability shifts toward approaches that work repeatedly, and some rarely used paths vanish. The post reports the trained model as the highest-scoring evaluated model under 100B parameters.
On the hotel task, the trained model left the booking and payments untouched in all 20 attempts. All expected records matched in 19 of 20, and one attempt missed an expected tag. As Microsoft notes, leaving the booking alone is necessary for that task but not the same as passing it.
Smaller LoRA runs with the same reward also improved. Qwen3.6-27B rose from 32.94% to 53.17% pass@1, and Qwen3.5-9B from 5.65% to 13.33%.
Caveats before you act on this
My read is that the findings are useful but come with real limits:
- It's self-reported. This is Microsoft's own benchmark, with no independent replication in the material I reviewed.
- The tasks are synthetic. They are reconstructions of enterprise workflows with simulated users, not production traffic.
- The costs leave things out. Post-RL costs apply the base model's list rates to measured tokens, because no provider serves the checkpoint. They exclude the 24-GPU training run, the selection probes, the simulator, the judge and infrastructure. The 67.2% drop is a drop in inference cost only, not total economics.
- A check only rewards what it specifies. The paper notes a task can pass on state while misreporting the outcome to the user. A reward built from executable checks encourages exactly what those checks require and nothing else.
- Transfer is untested. Microsoft hasn't measured how well this carries over to unrelated agent settings. It also hasn't measured how much its suggested operational fixes help.
What admins and developers can take from it
Microsoft distills four practices for teams building agents that change real records:
- Check state, not transcripts. Write assertions against the persisted records.
- Report repeated success. Run each task more than once. Report pass@1 alongside the count of tasks passed on every attempt, and say which estimator you use.
- Probe before you train. Run the base model several times on candidate tasks. Tasks it always passes or always fails burn rollouts and teach little.
- Keep training and evaluation in sync. Replay the same recorded model output through both parsers and confirm they extract identical tool names and arguments.
The framework and a training repository are public. The framework includes a tb CLI, a simulated-user loop, an MCP Session Proxy and an evaluation harness. The training repo covers data selection, rollouts, rewards and a validated reference configuration.
If you're piloting agents against line-of-business data, the main lesson is to measure the database and not the chat log. Also count how often a run succeeds every time, not just how often it succeeds once.