Composio ran DeepSeek V4 Flash through Claude Code, Codex, OpenCode, and Oh My Pi on 30 tasks involving connected services including Gmail, GitHub, Slack, and Notion. Claude Code averaged 122 seconds per task, while Oh My Pi took 272 seconds. OpenCode had the lowest cost per successful task; Oh My Pi posted the highest number of successful runs, at 17 of 30, while OpenCode finished 14.
The headline needs a narrower reading than “Claude Code is the fastest agent framework” suggests. It was fastest in this one 30-task test with one model and one tool environment. The more consequential finding is not that any one agent won overall, but that similar success results came with sharply different operating costs.
The wrapper changed the economics more than the pass rate
The four frameworks were all driving DeepSeek V4 Flash rather than their usual flagship models. That design choice is important: it limits the test’s ability to say which vendor has the strongest model, but makes it better at showing how much the surrounding software changes the same model’s behavior.
According to The Decoder, overall scores were close enough that there was no clean winner on reliability. Oh My Pi solved 17 tasks, OpenCode solved 14, and the remaining two harnesses fell somewhere in that narrow band. Seven individual tasks reportedly passed or failed depending solely on which framework ran them, suggesting the agent loop—not a different foundation model—was decisive in almost a quarter of the cases.
The difference in cost was much larger. At $0.195 for each successful task, Claude Code cost about 2.67 times OpenCode’s $0.073. That is not quite a threefold increase, but it is close enough to become material at any sustained volume. A team that automates hundreds of repository, ticketing, messaging, or documentation actions every day would see a small per-task gap turn into a meaningful monthly variance.
Speed showed a similar spread. Claude Code’s 122-second average was about 2.23 times faster than Oh My Pi’s 272 seconds. For an engineer waiting at a terminal, two minutes versus more than four and a half minutes is noticeable. For a background workflow handling a backlog of routine requests, paying more to save those minutes may be harder to justify.
The benchmark therefore gives IT teams two separate procurement questions: how much time does an agent save for the person supervising it, and how much does it cost when it succeeds? Those are related measurements, but they are not interchangeable.
Claude Code’s token result is the part that deserves scrutiny
Composio’s reported data contains a result that runs against the usual assumption that fewer tool calls and fewer generated tokens should mean a lower bill. The Decoder says Claude Code used the fewest tool calls and generated the least output, yet was the most expensive option per successful task.
That does not prove Claude Code is intrinsically wasteful. It proves the published metrics are insufficient to explain its cost.
Output tokens are only one line item in an agent workload. The expensive part can be input context repeatedly supplied to the model, long tool results returned into the conversation, reasoning tokens, cache misses, retry loops, context compaction, or a framework’s handling of failed runs. Cost per successful task also includes failures indirectly: an agent that spends money on unsuccessful attempts can look expensive even if its successful executions are lean.
The mismatch is especially relevant with tool-using agents. A framework can make fewer calls but inject larger system prompts, reload more conversation history, preserve verbose tool output, or choose a reasoning configuration that drives API usage in ways a simple tool-call count will miss. Conversely, a framework that makes more calls can still be cheaper if it keeps each interaction small, preserves cache-friendly prompt prefixes, or terminates bad paths sooner.
Composio did not publish, at least in the material available through The Decoder, the raw token accounting needed to isolate that cause. There is no disclosed breakdown of input, output, cached, and reasoning tokens; no per-task cost distribution; no retry count; and no complete run logs. There is also no indication of whether the quoted dollar figures include every request made during failed attempts, which matters because the benchmark reports cost per successful task rather than cost per attempted task.
That omission prevents a confident claim that Claude Code’s wrapper itself caused the higher cost. It does support a narrower, practical conclusion: token output and tool-call counts are not reliable proxies for an agent’s real API cost.
A small benchmark can expose a real operational problem
Thirty tasks is enough to uncover failure modes, but it is not enough to settle a platform choice for a production engineering organization. A single task changes the success rate by 3.3 percentage points. The difference between Oh My Pi’s 17 successful tasks and OpenCode’s 14 is three tasks—useful signal, but too small to establish a stable reliability hierarchy without repeated runs.
The unreported test details are at least as important as the ranking:
- The published summary does not identify the exact DeepSeek V4 Flash release, API price schedule, reasoning setting, or cache treatment used in each harness.
- It does not specify framework versions, installed extensions, system prompts, permission modes, or concurrency settings.
- It does not disclose task prompts, acceptance criteria, authentication setup, retry policy, or whether a human intervened after a tool error.
- It does not say whether each harness received identical tool schemas and identical preloaded context.
Those details can dramatically change results. Coding agents are not passive API clients. Claude Code, Codex, OpenCode, and Oh My Pi each bring different tools, prompts, context-management rules, editing strategies, and assumptions about permissioning. A benchmark that allows each harness to operate as designed measures the real user experience, but it does not identify a single technical reason for the outcome.
The DeepSeek documentation for Oh My Pi illustrates how configuration alone can alter observed results. DeepSeek says recent Oh My Pi builds list DeepSeek V4 models but require a custom
models.ymlconfiguration for reliable tool use in thinking mode. The vendor specifically warns that missing compatibility fields can generate HTTP 400 errors during long, tool-using conversations. If a test used incomplete defaults rather than the documented configuration, its reliability and speed results could be distorted before the agent had a fair chance to operate.
There is no evidence in The Decoder’s report that Composio made that mistake. The point is more basic: agent benchmarks need configuration records, not just rankings. Without them, another administrator cannot reproduce the result or tell whether it applies to their own environment.
Windows developers should measure the workflow, not the brand
For Windows users, the immediate decision is likely to involve a mix of native terminals, PowerShell, Windows Terminal, WSL, remote Linux hosts, Git repositories, browser automation, and Microsoft 365 or SaaS integrations. The cheapest agent for a GitHub-and-Slack workflow may not remain cheapest when it must analyze a large monorepo through WSL, work with Azure DevOps, execute PowerShell scripts, or operate behind an enterprise proxy.
The right comparison is a controlled local pilot. Use the same repository snapshot, the same issue set, the same approved credentials, and the same machine or VM. Record wall-clock time, complete API cost, successful completion without human repair, retry count, and the amount of review required before merge. Include failures in the cost ledger rather than dividing only by successful results.
That last point is where Composio’s result is most valuable. A tool that appears cheap on a successful-task basis may still consume plenty of budget chasing failures; a tool that looks expensive may save staff time if it succeeds more consistently on high-value work. Cost per success is a useful business metric, but it should sit alongside cost per attempted task and engineer review time.
Independent evaluation data from Vercel’s Next.js agent tests also shows that agent, model, time, and cost do not move in lockstep. Those tests use a different workload and different models, so they cannot validate Composio’s figures. They do reinforce the broader observation: selecting an AI coding assistant by model name alone misses the software layer that governs how the model actually works.
The practical outcome: instrument before committing
Composio’s reported benchmark does not establish a permanent winner between Claude Code, Codex, OpenCode, and Oh My Pi. It does establish that the agent harness can produce a 2.67-fold cost difference and a 2.23-fold latency difference even when the underlying model is held constant.
For teams paying usage-based API bills, the implication is concrete. Do not approve an agent rollout based on a demo, a model leaderboard, or output-token totals. Run the candidate harnesses against the actual tools and repositories your staff use, retain the raw usage data, and calculate the spend associated with both completed and failed work.
Claude Code’s fast result may justify its higher cost where an engineer is blocked and every minute matters. OpenCode’s reported $0.073 per successful task is the more attractive starting point for high-volume, asynchronous work. But until Composio releases task-level methodology and billing detail, the benchmark should be treated as evidence that the wrapper matters—not as a final purchasing verdict.
References
- Primary source: the-decoder.com
Published: August 6, 2026 at 4:33 PM UTC
Claude Code is the fastest agent framework but costs nearly three times more than the cheapest rival
Composio tested Deepseek V4 Flash across four agent frameworks on 30 real-world tasks. Success rates were mostly similar, but costs varied by nearly 3x: OpenCode came in cheapest at $0.073 per task, while Claude Code cost $0.195 despite using the fewest tool calls and output tokens. The choice...the-decoder.com - Related coverage: api-docs.deepseek.com
Using DeepSeek with Oh My Pi | DeepSeek API Docs
Oh My Pi is a terminal AI coding agent. As of v14.5 it ships DeepSeek V4 model entries, but the built-in compat is incomplete — a custom models.yml is still required for reliable use.api-docs.deepseek.com - Related coverage: anthropic.com
How Claude Code is used in practice \ Anthropic
New Anthropic research looking at interactive agentic coding. We evaluate the composition of tasks, human-AI collaboration, and success rates.www.anthropic.com
