A follow-up to The Golden Hammer Syndrome — six months later, AI is delivering real ROI. The problem is we’re still asking the wrong questions.
In January I wrote about using LLMs to check if a number is even. People laughed — and then sent me screenshots of their own codebases doing the same thing. The Golden Hammer Syndrome post was about over-engineering through AI. This one is about a different dysfunction that I’ve been watching develop since: the obsession with measuring AI adoption rather than what adoption actually produces.
I initially thought this was a tooling problem — that companies just hadn’t found the right dashboard to track AI usage. After several conversations with engineering managers and a few uncomfortable retrospectives on projects I was involved in, I no longer think so. The problem is the question itself.
The Numbers That Actually Matter
Customer service was the first domain to produce consistent data, partly because the economics are simple enough to measure cleanly.
Benchmark data from cross-industry analyses in 2025 and 2026 places the cost of a human-handled support interaction anywhere from $2.70 for simple retail tickets to $60 or more for complex B2B enterprise cases, with a global baseline across all industries somewhere around $6–$7 per contact. That range matters. The ROI calculation for AI deflection looks completely different at $3 per ticket than at $30, which is why blanket claims about “savings” deserve scrutiny. What is consistent across sectors is the direction — AI-resolved interactions cost materially less, and the delta compounds at volume. According to Fin.ai’s 2026 benchmark analysis, organizations with mature AI customer service deployments reported average first-year ROI above 40%, increasing substantially in subsequent years. Gartner projects that by 2026, conversational AI in contact centers will reduce agent labor costs by $80 billion globally.
Software engineering is harder to measure cleanly because the variables are messier. The most rigorous production study I’ve found is a longitudinal analysis of 300 engineers at 1mg.com (arXiv:2509.19708), which tracked the deployment of an in-house AI-assisted development platform over one year — real production environment, not a controlled benchmark. Their results: a statistically significant 31.8% reduction in PR review cycle time (p = 0.0018), 60.1% overall increase in code shipped to production, with no corresponding increase in defect rates reported by the authors and junior engineers (SDE1) showing the highest productivity gains at 77%. Critically, the top 30 adopters saw a 61% increase in shipped code — while the bottom 30, who used the tools sporadically, showed an 11% decline. That divergence is worth sitting with.
The 2025 DORA Report — which renamed itself “State of AI-assisted Software Development” — confirmed the same pattern: AI amplifies existing team strengths rather than compensating for weaknesses.
The Subscription Window Is Closing
There’s a financial reality sitting underneath most AI strategy discussions that doesn’t get named directly. The current pricing of AI subscriptions is not sustainable. It is a large, deliberate subsidy — and it is operating in an environment where sustainable margins matter far more than growth-at-any-cost pricing.
SemiAnalysis published research in June 2026 that put numbers to something many people suspected but couldn’t quantify. Their method was direct: they bought one plan each from Anthropic and OpenAI, then ran long coding tasks to the weekly usage limit simultaneously via subscription and via API, measuring token consumption at both ends. The gap is significant. A $200/month ChatGPT Pro plan, pushed to its usage limits, can represent API-equivalent compute SemiAnalysis estimates at up to $14,000 per month under their testing conditions. A Claude Max 20x plan at the same price reached approximately $8,000.
To be clear: these are maximum utilization figures, not typical usage. Most subscribers don’t come close to the limits. But the break-even analysis is what’s operationally relevant — the point at which these plans stop being profitable for the vendor, assuming a standard 75% API margin:
| Plan | Break-even utilization |
|---|---|
| ChatGPT Pro 20x | ~5.7% of limit |
| Claude Max 20x | ~10% of limit |
| ChatGPT Plus / Pro 5x | ~11.4% of limit |
| Claude Pro / Max 5x | ~20% of limit |
If you’re a heavy user, the vendor is almost certainly losing money on your account. The API business operates at healthy margins. These subsidies exist to build market position — but pressure to demonstrate sustainable unit economics is increasing, and subscription pricing will adjust accordingly.
The implication for your organization: teams that have built workflows around current subscription economics should be thinking about what those workflows cost at API rates. The engineering that makes sense to do now — before repricing — is building automations efficient enough to justify those costs.
Why “Use AI More” Fails as a Strategy
Last month I reviewed an internal dashboard where AI tool adoption across an engineering team had reached 94%. Someone had put real effort into tracking it: IDE plugin installs, prompt counts, weekly active users. Nobody in the room could explain whether throughput had improved. Change failure rate was, if anything, slightly up.
This is not unusual. Multiple independent research threads from 2025 converge on the same uncomfortable finding. S&P Global reported that 42% of companies abandoned most of their AI initiatives by mid-2025 — up sharply from 17% the year before. MIT’s NANDA research, analyzing generative AI deployments across enterprise companies, found that 95% of GenAI pilots showed no measurable P&L impact — a finding Fortune reported in August 2025, citing the report’s own language: “The 95% failure rate for enterprise AI solutions represents the clearest manifestation of the GenAI Divide.” RAND Corporation’s analysis found over 80% of AI projects fail to reach meaningful production — roughly twice the failure rate of comparable non-AI technology projects.
The companies getting real results shared one characteristic: they defined what success looked like before they started, not after.
Mandating “AI adoption” without defining outcomes is the organizational equivalent of telling office workers in 1985 to use the computers. Some people learned spreadsheets and rewrote their workflows. Many played Minesweeper. Both groups had 100% computer adoption.
A Framework That Actually Works
None of the following metrics ask whether AI was used. All of them ask whether something changed.
a) Automation Coverage: What Changed, Not What Tool Was Used
The question worth asking in every sprint review is not “did anyone use AI this week?” It is: what got automated that wasn’t automated before, and how fast did we build it?
Automation leaves artifacts — a script running in CI/CD, a webhook that replaced a manual step, a test suite that didn’t exist last sprint. These are auditable. Reported AI usage figures are not. I’ve seen teams where AI adoption was high by any tool-tracking metric, but sprint after sprint produced no new automation, no reduction in manual steps, no tooling that other team members could actually use.
The 1mg.com study makes the compounding dynamic visible: the top adopters didn’t just use AI more — they built with it consistently, and the gap between them and minimal users widened to a 72-percentage-point difference in code output over just a few months. After 12 to 18 months of working seriously with AI tools, an engineer with genuine aptitude has built capabilities that would have taken three or four years otherwise. They understand what they generate, they catch errors before shipping, they build things for their teammates.
An engineer who learned only to prompt — without building a real model of how the underlying systems work — produces output whose quality tracks the model’s quality on any given day. When the model is wrong, they can’t tell.
The market is separating into two groups faster than most teams realize.
The uncomfortable implication is that AI does not reduce the importance of engineering fundamentals. It increases it. When model quality becomes available to everyone, the differentiator becomes the ability to evaluate, debug, and operationalize the output. Hiring someone because they can generate code quickly is the wrong frame. The better question is: can they tell when the generated code is wrong? That skill is not taught by prompting. It is built by years of understanding what happens underneath.
The metric I use: number of new automations shipped per sprint, regardless of how they were built.
b) Velocity with Accountability
If AI is genuinely making a developer faster, it should show up in how they scope work. The signal: do they declare smaller story point estimates for familiar problem categories, and do they actually deliver on them?
This takes time to show. Tracking it week-to-week produces noise. Over rolling quarters, the trend becomes readable. An engineer integrating AI effectively should progressively size known categories of work smaller — not because they’re being optimistic, but because the work genuinely takes less time. If estimates haven’t moved after six months of AI adoption, either the tooling isn’t helping or it isn’t being used deeply enough to change the developer’s actual working model of effort.
c) TQA as a Force Multiplier
QA engineers building AI-assisted validation tooling are, in my view, the most underrated application of AI in an engineering team right now.
The relevant metric is not “does QA use AI tools.” It’s change failure rate and PR revert rate. If AI in QA is working, CFR drops — coverage expands without headcount scaling. If the team is accepting AI-generated test cases without understanding what they actually validate, CFR stays flat or climbs. More test files doesn’t mean more coverage. The output of good QA AI integration is fewer production incidents.
d) DevOps, Observability, and the Datadog Problem
AI is embedded in observability tooling now. Datadog’s Watchdog, anomaly detection, and log pattern analysis surface root-cause hypotheses faster than any engineer scanning dashboards manually, and are genuinely useful in time-sensitive incidents.
But there’s a failure mode worth naming, and it closely mirrors the cognitive atrophy problem from the original Golden Hammer post.
Watchdog flags a latency anomaly post-deployment. It generates a root cause analysis: “Probable cause: database connection pool exhaustion.” The on-call engineer pastes it into Slack and escalates to the DBA team. The DBA investigates for an hour. The actual cause was a misconfigured Kubernetes health check restarting pods at the wrong interval. Watchdog diagnosed the symptom correctly, the cause incorrectly. An hour of senior engineering time, gone.
Watchdog pattern-matches against historical telemetry. It doesn’t know your specific deployment, your infrastructure topology, or that you pushed a change 40 minutes before the alert fired. It is fast and often right. It is not a substitute for someone who understands the system.
Before forwarding an AI-generated incident report, three questions:
- Does the root cause match the timeline of the deployment?
- Have you looked at the raw logs, not just the AI summary?
- Can you explain the failure in one sentence without using the AI’s phrasing?
The third question is the real one. If you can’t answer it, you haven’t understood the incident yet. The operational metric is MTTR — worth augmenting with one additional dimension: what percentage of AI-generated root cause analyses required correction before the incident was resolved? That number tells you whether your team is using AI observability or forwarding it.
The New Litmus Test
The original Golden Hammer post ended with a simple test: before importing @google/genai, check if new Date() does the job.
The team-level version for 2026:
Before reporting on AI adoption, answer: what changed in production because of it?
Three questions for every sprint:
- What did we automate this week that we couldn’t before?
- Did our change failure rate move?
- Are we on subscription or API pricing — and does our team understand what happens to that number when the subsidy ends?
The subscription window is time-limited. The gap between AI-amplified engineers and AI-dependent prompters is widening faster than most teams realize. The organizations extracting real value stopped tracking adoption metrics and started tracking what shipped.
Boring metrics. Stable systems. Real outcomes. That still works.
← Related: The Golden Hammer Syndrome: Why We Need Less AI and More Engineering