Your AI Pilot Isn't Failing — Your Scoreboard Is

Last month a VP told me their AI coding rollout was "a wash." I asked how they knew. The answer: they'd divided the annual seat cost by the number of developers, eyeballed a few sprint velocities, and surveyed the team. Most people said they felt faster. The CFO didn't see it in the burndown. Verdict: inconclusive, probably not worth it.

That verdict is almost certainly wrong — but not because the tools are magic. It's wrong because the scoreboard measures the wrong things on both sides of the ledger. If you're going to decide whether AI is paying off, you have to fix the scoreboard first, or you'll keep making expensive decisions on bad data.

The cost side: seats lie now

The seat-license mental model is a holdover from the SaaS era, and in 2026 it actively misleads you. The platforms have moved to usage-based pricing — Anthropic, for example, shifted to charging enterprise customers based on how much AI they actually consume rather than a flat per-seat fee. That means your real cost isn't seats × price. It's tokens, and tokens scale with how aggressively your best engineers lean on the tools.

Here's the trap: a healthy ROI on AI coding tools lands around 2.5–3.5x on average, and 4–6x in the top quartile — but only when the cost denominator includes actual token and usage-based spend, not just license fees (Exceeds.ai). Teams that budget by seats get a nasty surprise when the bill arrives, conclude the tool is "expensive," and pull back — right as their power users were starting to compound returns. You can't manage a usage-based cost with a seat-based spreadsheet. Instrument token spend per team, per repo, ideally per workflow, and put it next to the value you're capturing.

The value side: "feels faster" is a trap in both directions

The survey data is where it gets genuinely interesting, because perception and reality diverge — hard. In a randomized controlled trial, METR found experienced open-source developers were actually 19% slower with AI tools while reporting they felt 20% faster (CallSphere). That's not a rounding error; that's a 39-point swing between feeling and fact, in the wrong direction, among exactly the senior people whose opinions carry weight in a go/no-go meeting.

So if your evidence base is a team survey, you have no idea what you measured. The fix isn't to distrust your developers — it's to stop asking them to be the instrument. Pull the signal from the system:

• Cycle time from first commit to merged PR, segmented by daily-AI-users vs. non-users.
• PR throughput and size — empirical analytics show material gains for daily users, on the order of a few hours saved per week and higher merge rates (getPanto).
• Rework rate — the percentage of merged code that gets reverted or substantially rewritten within 30 days. This is the one most teams skip, and it's the one that catches the failure mode.

That last metric matters because the productivity number has a dark twin: AI-assisted code can increase issue counts roughly 1.7x and raise security findings if it ships without governance (Uvik). A team can look 20% faster on throughput and be quietly accumulating defects. If you only watch velocity, you'll celebrate right up until the incident review.

What a real scoreboard looks like

I tell teams to track four things, together, and never one in isolation:

• Token spend per team — your true cost, refreshed weekly.
• Cycle time, AI-cohort vs. control — your true speed, measured not felt.
• Rework and defect rate — your quality tax, so speed isn't hiding rot.
• Adoption depth, not breadth — daily active use, not "we bought 200 seats." The value lives in the ~50% of pros who use these tools daily, not the badge count.

Put those four on one dashboard and the conversation changes. Instead of "is AI worth it" — an unanswerable question at that altitude — you get "Team A is at 4x with flat rework, Team B is burning tokens with no cycle-time gain, let's find out why." That's a manageable problem. "AI is a wash" is not.

The takeaway

Before you judge an AI rollout, audit the instrument you're judging it with. Nine times out of ten the pilot isn't ambiguous — the measurement is. Count real token spend, not seats. Measure cycle time, not vibes. Watch rework, not just velocity. Do that, and you'll usually find the value was there the whole time, unevenly distributed and waiting to be managed. The teams that win with AI aren't the ones with the best tools. They're the ones who can actually see what the tools are doing.

Sources: Exceeds.ai enterprise ROI studies · CallSphere on the METR RCT · getPanto AI coding statistics · Uvik AI coding assistant statistics