How to Measure Returns on AI (Economist)

Summary

A practical measurement framework for enterprise AI: track inputs, outcomes and organisational learning together. Token use can reveal adoption but not value; outcome measures can reward low-quality volume; narrow near-term ROI misses the productivity J-curve and the capability an organisation is building. The article's useful contribution is not a single ROI formula but a three-layer scorecard with explicit quality and change-management checks.

Key claims

1. Inputs: usage and token spend remain useful during adoption, but only as diagnostics. A finance function's AI use should not be zero, yet usage cannot establish value.
2. Outcomes: measure team productivity and customer satisfaction, with speed, ease and quality. Google engineers are evaluated on all three to prevent faster production of worse software.
3. Organisational capability: track employee satisfaction with implementation and progress building expertise that AI will not automate.
4. J-curve: established manufacturers often suffer an initial productivity dip while workers learn and management practices change; projected returns must include that disruption.
5. Financial attribution: avoided future hiring is often cleaner than claiming realised layoffs as savings. Revenue attribution needs baseline data and A/B testing.

The paper example is the warning against volume-only metrics: one health-and-nutrition dataset generated about four papers a year in 2014–21, then 190 in the first nine months of 2024, without assurance that the extra work was useful.

Connects to

• Companies Are Scrambling to Curtail Soaring AI Costs (Economist) — predecessor on the shift from tokenmaxxing to budget discipline.
• AI Productivity Disconnect — scattered individual time savings do not automatically become firm-level output.
• Verification Tax — quality must be measured alongside speed and ease.
• Agentic Pods — an example of pairing compressed cycle times with workflow-level outcomes.
• Token Scarcity · Model Routing — input-cost control is necessary but insufficient.