Anthropic’s Claude Code lead Boris Cherny is urging enterprises to stop treating AI token consumption as a proxy for success. The better metric, he argues, is whether an AI-assisted task replaced work the company would otherwise have assigned to engineers—and how many manual engineering hours that work would have cost.
In a series of posts reported by Business Insider on July 17, Cherny said usage dashboards still have a role, but primarily show activity. High token counts can reflect experimentation, inefficient prompts, long-running agent loops, or simply a hard problem. None of those outcomes proves that an AI deployment saved money or improved delivery.
For IT leaders, the proposed calculation is straightforward: identify work completed with AI assistance, decide whether it would have been done without the tool, estimate the avoided engineering effort, then compare that value with the model, platform, review, and operational costs. That is a more defensible ROI number than a graph showing tokens consumed per developer.
Token and request data are still useful for capacity planning, budget controls, and detecting unexpected usage spikes. Anthropic itself has recently expanded Claude Enterprise analytics, including usage, cost, and value views for administrators. But cost telemetry needs an outcome beside it.
A Windows engineering organization, for example, might use an AI coding agent to update deployment scripts, investigate a PowerShell failure, modernize an internal .NET application, or draft test coverage for a Win32 integration. Counting the prompts or tokens involved says little. The relevant questions are whether the change shipped, how much review it required, whether it avoided an incident, and whether the same task would have entered a backlog without the assistant.
That distinction matters because AI-assisted code is not free once it leaves the model. Teams still need code review, testing, security checks, source-control discipline, and accountability for production changes. A task that generates thousands of lines but creates cleanup work is not necessarily a win, even if the token bill is modest.
That calls for measurement at the workflow level rather than at the model gateway. Organizations should track completed work against a baseline: lead time for a class of ticket, time from bug report to verified fix, release frequency, backlog age, or the volume of maintenance work closed without delaying planned development. Those measures will not isolate AI perfectly, but they are closer to business and engineering outcomes than token totals.
Admins should keep spend controls and audit logs, especially where coding agents can access repositories, terminals, or internal tools. But dashboards should pair model costs with delivery data and review burden, not become a leaderboard for consumption.
The practical next step is to pilot AI on a defined category of engineering work and compare completed, reviewed outcomes against the team’s pre-AI baseline.
In a series of posts reported by Business Insider on July 17, Cherny said usage dashboards still have a role, but primarily show activity. High token counts can reflect experimentation, inefficient prompts, long-running agent loops, or simply a hard problem. None of those outcomes proves that an AI deployment saved money or improved delivery.
For IT leaders, the proposed calculation is straightforward: identify work completed with AI assistance, decide whether it would have been done without the tool, estimate the avoided engineering effort, then compare that value with the model, platform, review, and operational costs. That is a more defensible ROI number than a graph showing tokens consumed per developer.
Why token dashboards fall short
Token and request data are still useful for capacity planning, budget controls, and detecting unexpected usage spikes. Anthropic itself has recently expanded Claude Enterprise analytics, including usage, cost, and value views for administrators. But cost telemetry needs an outcome beside it.A Windows engineering organization, for example, might use an AI coding agent to update deployment scripts, investigate a PowerShell failure, modernize an internal .NET application, or draft test coverage for a Win32 integration. Counting the prompts or tokens involved says little. The relevant questions are whether the change shipped, how much review it required, whether it avoided an incident, and whether the same task would have entered a backlog without the assistant.
That distinction matters because AI-assisted code is not free once it leaves the model. Teams still need code review, testing, security checks, source-control discipline, and accountability for production changes. A task that generates thousands of lines but creates cleanup work is not necessarily a win, even if the token bill is modest.
Measure work that would otherwise not happen
Cherny’s more interesting point is that the biggest return may not be a direct substitution of AI for a developer hour. It can come from maintenance, fixes, and routine engineering work happening with less hands-on effort, allowing teams to take on improvements that previously were too small, too slow, or too expensive to justify.That calls for measurement at the workflow level rather than at the model gateway. Organizations should track completed work against a baseline: lead time for a class of ticket, time from bug report to verified fix, release frequency, backlog age, or the volume of maintenance work closed without delaying planned development. Those measures will not isolate AI perfectly, but they are closer to business and engineering outcomes than token totals.
Admins should keep spend controls and audit logs, especially where coding agents can access repositories, terminals, or internal tools. But dashboards should pair model costs with delivery data and review burden, not become a leaderboard for consumption.
The practical next step is to pilot AI on a defined category of engineering work and compare completed, reviewed outcomes against the team’s pre-AI baseline.
References
- Primary source: Business Insider
Published: 2026-07-17T06:09:45.346000+00:00
Claude Code's Creator Has Better Way to Measure AI Success Than Tokens - Business Insider
Boris Cherny says companies should track human hours saved, not just tokens spent.www.businessinsider.com - Official source: anthropic.com
How Claude Code is used in practice \ Anthropic
New Anthropic research looking at interactive agentic coding. We evaluate the composition of tasks, human-AI collaboration, and success rates.www.anthropic.com - Related coverage: techcrunch.com
Clawdmeter turns your Claude Code usage stats into a tiny desktop dashboard | TechCrunch
A new open source gadget called Clawdmeter turns Claude Code usage stats into a tiny desktop dashboard for AI coding power users.techcrunch.com - Official source: claude.com
Improving skill-creator: Test, measure, and refine Agent Skills | Claude by Anthropic
Skill authors now have tools to verify their skills work, catch regressions, and improve descriptions—no coding required.
claude.com
- Related coverage: fortune.com
The head of Claude Code hasn't 'written a line of code by hand' in 8 months | Fortune
Boris Cherny was asked at Brainstorm Tech if he was concerned about the rapid progress of AI: "Yes."fortune.com
- Related coverage: claudeainews.com
Claude Code Creator: ROI Focus Is Fine, But Experiment Too | ClaudeAINews
Boris Cherny, creator of Claude Code, argues companies pursuing AI ROI are on the right track but must carve out space for open-ended experimentation to stay competitive.www.claudeainews.com