The core argument: fuel gauges aren't finish lines
Finlayson opens with a NASCAR analogy. Imagine a team that declares the car burning the most fuel the winner. Telemetry looks spectacular and the car finishes last. He applies this to "tokenmaxxing," which he describes as internal leaderboards that rank staff by AI tokens consumed. He also describes a swing in the other direction, "tokenminning," where the goal becomes using as few tokens as possible. Both fail for the same reason. Burning tokens or saving them is an input measure, not an outcome.
The column's own anecdotes about a Meta leaderboard and Uber spending its annual AI budget are asserted without independent corroboration. Treat them as the author's framing. Likewise, "tokenminning" is the author's label, not an established industry term.
A related TechRadar Pro piece by Greg Holmes, EMEA Field CTO at Apptio, an IBM Company, was published a month earlier. It argues that companies without a reliable way to measure AI ROI end up cutting the only metric they can see, which is consumption. Holmes cites a Gartner estimate that 84% of finance leaders haven't been able to measure the ROI of AI initiatives. That is corroborating commentary from a vendor, not independent validation. It does show the argument is a recurring theme.
What counts as a vanity metric
Finlayson defines a vanity metric as a number that looks great on a slide but doesn't change how you run the business. His examples:
- GPU utilization: it shows how much compute you hold, not whether it solves problems.
- AI features shipped per quarter: it rewards bolting a summarizer onto every legacy product.
- "95% adoption" of GitHub Copilot: it often just means licenses were assigned, not that the tool is used well.
These are plausible examples from an experienced practitioner. They aren't industry findings. Usage data isn't useless. It is a diagnostic signal that only makes sense next to an outcome and a quality check.
The engineering trap: faster code, same bottleneck
The column's best point is about software teams. If developers generate more code faster, pull requests can grow and the bottleneck moves to human review. Cycle time can stay flat. Defects can rise if reviewers rubber-stamp output. That matches DORA's own explanation. It says AI makes code generation faster, which often leads to larger batch sizes that are slower to review and more prone to instability.
A correction on the DORA figures
Finlayson writes that the 2025 DORA report found a 25% rise in AI adoption correlated with lower delivery stability. He implies this was the typical outcome for teams deploying AI into underprepared systems. The details are more nuanced, and the specific numbers are often misattributed.
- The 1.5% and 7.2% figures come from the earlier model. DORA's "Impact of Generative AI in Software Development" page says a 25% increase in AI adoption is associated with a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. Secondary analyses attribute these numbers to DORA's 2024 research. Those analyses include Unblocked's DORA-in-the-AI-era write-up.
- The 2025 report changed the throughput picture. Google Cloud's announcement says that, unlike last year, DORA observed a positive relationship between AI adoption and both software delivery throughput and product performance. The same announcement says AI adoption does continue to have a negative relationship with software delivery stability. The 2025 report did not simply repeat the earlier finding.
- The explanation holds up. DORA says without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability. This supports Finlayson's broader "amplifier" point even though the citation was imprecise. Google Cloud's summary also says AI accelerates software development, but that acceleration can expose weaknesses downstream.
Two caveats apply. These are survey-based associations, not proof of cause. DORA is also a Google Cloud program, and Google sells AI tools, so readers should weigh the findings with that in mind. One independent summary makes the same point about the DORA data's limits: it is a modeled correlation from survey data and does not measure cause.
Perception versus performance
The 2025 DORA announcement also reports that 90% of survey respondents use AI at work and more than 80% believe it has increased their productivity. It adds that 30% report little or no trust in AI-generated code. Those are self-reported views. A team can feel faster while its delivery system is not. That gap is why Finlayson's call for outcome measurement matters.
Frameworks: DORA and SPACE as instruments
Finlayson recommends starting with the outcome you want to move. Then use DORA or SPACE to check whether you got there. The framework is the instrument, not the goal.
- DORA looks at delivery pipeline health: how often you deploy, how long changes take, how quickly you recover, and how often changes cause incidents. RedMonk notes that DORA's metric grouping has shifted over time. It added rework rate as a stability metric in 2024. The 2025 report also moved away from low, medium, high and elite performance clusters toward team archetypes. Check which definitions your own tooling uses before comparing numbers across years.
- SPACE comes from a Microsoft Research and GitHub-affiliated author group. Microsoft Research lists the paper by Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Tom Zimmermann, Brian Houck and Jenna Butler in ACM Queue, February 2021, volume 19, issue 1. The abstract says developer productivity cannot be measured by a single metric or dimension. Its five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Activity is one input among several, not the verdict.
The column says neither framework was designed for AI. That is a reasonable point: they measure whether the work got better, not whether AI got used.
A practical measurement loop
Combine the column's advice with the DORA findings, and the process looks like this:
- Name the outcome. Pick one result you already want to move, such as cycle time, rework rate, cost per change, response time, pipeline conversion or forecast accuracy.
- Record a baseline before AI enters the workflow. Holmes makes the same point. Without a baseline, any gain is guesswork.
- Measure the same outcome afterward. Add guardrails such as defect rate, delivery stability, reopen rate and developer experience.
- Test attribution. Other process changes can explain a shift. A before-and-after comparison alone doesn't prove AI caused it.
- Decide whether it is repeatable. If it moved, find out why. If it didn't, that is still useful.
Finlayson's illustration is a support team that baselines first response at nine hours. After two months of AI-drafted replies it is at three hours, with reopen rates unchanged. It is a hypothetical, not a verified case study. Its structure is the useful part: one outcome, a baseline, and a quality guardrail.
His threshold numbers are examples too. The "30% cycle-time drop" and "25% adoption" figures are not universal benchmarks. Set your own targets from your own baseline.
What Windows and Microsoft shops should take from this
For teams rolling out Copilot, GitHub Copilot or other AI assistants, the lesson is to resist reporting license counts as success. Seat activation and prompt volume are easy to pull from admin dashboards. They are weak proxies for value. Pair them with delivery metrics from your pipeline and with business measures that predate the rollout.
The DORA findings also point to prerequisites. Automated testing, version control discipline and fast feedback loops determine whether faster code generation becomes better delivery. If those are weak, expect AI to magnify the weakness.
The bottom line
Finlayson's central question is worth taping to a monitor: what outcome would move if this number did? If you can't name one, you're watching the fuel gauge. His DORA citation is imprecise, but the argument survives the correction. Measure outcomes first, treat usage data as diagnostics, and be skeptical of any single number, including the ones in a vendor's dashboard.