When a measure becomes a target it ceases to be a good measure. Gaming is usually not malice; it is a rational response to a one-sided metric. So we do not ship one-sided metrics.
The four pathways
Numerator gaming
Splitting PRs, trivial commits, micro-deploys to inflate frequency. Easy, invisible, and it feels like compliance.
Denominator gaming
Deploying less to suppress change failure rate; not declaring incidents. Protects a stability number.
Input gaming
Farming AI usage — tokens, prompts, acceptance clicks — wherever adoption is a KPI.
Classification gaming
Mislabelling incidents, reclassifying unplanned work, under-declaring AI provenance. The cheapest of all; requires no behaviour change.
Sincere self-report error. In METR's trial developers believed they were 20% faster while measurably 19% slower. A measurement system built on perception is wrong even when everyone is honest.
The eleven controls
- No individual-level reporting, everTeam is the minimum unit; survey cells of five or more; author identifiers hashed at ingest; appraisal use prohibited in writing.
- Counter-metric adjacencyThe scorecard physically cannot render a speed metric without its pair.
- No AI-usage targetsWe will not design, report against or endorse a target for AI-assisted share, acceptance rate or tokens.
- Definitions frozen by contractAny change needs an amendment, both signatures and a restatement of history.
- Query provenanceEvery number resolves to a version-controlled query ID in a client-owned repository.
- Provenance hand-audit30–50 PRs classified by hand monthly; the measured error rate is printed beside every attributed figure.
- Distribution reportingMedian and p85 with n; a p85 worsening while the median improves is the signature of batching around the measurement.
- Within-team comparison onlyCross-team league tables create pressure to game without creating information.
- TriangulationTelemetry, perception and artefact inspection; a finding is strong only when two agree.
- Anomaly review before publicationAny movement over ~25% in a period is investigated against the intervention register first.
- Rubber-stamp detectionAlert when review latency falls while review depth degrades and zero-comment approvals rise.
The forty-five-second sales answer
Three things. We never report individuals and we never set AI usage targets — the moment adoption becomes a KPI you get token farming instead of value. Every speed metric on our scorecard has a counter-metric locked beside it, so if lead time improves while churn and review depth degrade, the scorecard shows you bought the first with the second. And every number resolves to a version-controlled query in your repository, with the definitions frozen in a signed contract. Gaming is usually a rational response to a one-sided metric; we design so there is no one-sided number to respond to.
Red flags in a baseline
- Deployment frequency high but concentrated in a release window — batching disguised as flow.
- Change failure rate implausibly low (under 2%) with no incident-to-deploy linkage.
- A large share of PRs approved in under five minutes with zero comments.
- Acceptance-rate dashboards displayed in a leadership forum.
- An existing "AI adoption %" OKR with a numeric target attached to teams.
- Incident severities clustered just below the reporting threshold.