This is the book. Everything else in the library is an instrument that this guide tells you when to pick up. Read it once end to end before your first engagement; re-read the discovery section before every first call and the measurement section before every readout.
Why this work exists
Three independent bodies of evidence converge on the same shape. Adoption is effectively universal: around 90% of technology professionals report using AI at work, at a median of about two hours a day. Value capture is rare: 39% of organisations report any enterprise-level EBIT impact and most put it below 5%, while roughly 6% qualify as high performers. A separate study of enterprise generative-AI deployments found about 95% of pilots producing no measurable effect on the P&L, attributing the failure to a learning and integration gap rather than model quality.
In engineering specifically the gains are real and they leak. DORA's 2025 research found AI adoption associated with higher delivery throughput — a reversal of its 2024 finding — and simultaneously associated with higher delivery instability. The qualitative follow-up explains why: AI has overwhelmingly targeted inner-loop activities, while the outer loop absorbs the extra volume without having been redesigned.
Self-reported productivity is not evidence
The most important study for this practice is the one that embarrasses our industry. METR ran a randomised controlled trial with 16 experienced open-source developers across 246 real tasks on repositories they knew well. When AI use was allowed, tasks took 19% longer. The developers had predicted a 24% speed-up and, after finishing, still believed they had been about 20% faster.
AI's effect on delivery speed is large, real, highly context-dependent in sign and size, and systematically mis-estimated by the people doing the work. Every element of the method follows from that sentence. If the effect were uniformly positive nobody would need us; if practitioners could feel it accurately, a survey would do.
The five rules of the practice
- Baseline before intervention, alwaysNo baseline, no engagement claim. Minimum eight weeks or three complete delivery cycles per team, whichever is longer. Where AI is already rolled out — the usual case — use a retrospective baseline from before first licence activation and state the design weakness explicitly.
- Each team is compared with itselfCross-team comparison is confounded by domain, codebase age, test coverage, regulatory burden and skill mix. A Google trial's AI effect vanished once proficiency and task familiarity were controlled; league tables reproduce exactly that error at scale.
- Every speed metric carries a counter-metricThroughput beside stability, lead time beside review load, AI-assisted output beside churn and duplication. This is the only defence against optimising a number into a worse system.
- Perception is measured, but never aloneRun the developer-experience survey alongside system telemetry and expect them to disagree. The disagreement is a finding, not an error.
- We hand over the instrument, not just the insightTelemetry lives in the client's systems, defined by a written contract, reproducible from query IDs they own. If our departure breaks the measurement, we failed.
What we refuse to do
- Rank named individuals on any metric we produce.
- Report AI value using vendor usage dashboards as the primary evidence.
- Promise a percentage improvement before baselining.
- Quote a published illustrative ROI example as a client expectation.
- Let a client set an AI-usage target — the mechanism by which measurement destroys itself.
State them in the first call. They cost some deals, and they are the reason the remaining deals believe the findings.
The engagement arc
Baseline
Capture what the delivery system did before we touched it, with enough history that normal variation cannot be mistaken for an effect. Four delivery metrics, flow metrics, maintainability counter-metrics, a perceptual survey and the AI estate — especially licence activation dates, which define the retrospective boundary.
Instrument
Make value measurable continuously inside the client's systems, with AI provenance captured from the start. The telemetry contract is signed here: definitions, sources, owners, and — critically — who may act on each metric and who may not.
Change
Intervene where the measurement says the constraint is, not where the tooling is most exciting. Rank candidate interventions by measured time lost. Changes land as pull requests in the client's repositories, pair-programmed with their engineers.
Measure
Report what changed against the team's own baseline, with uncertainty stated and counter-metrics in view. Then hand over: queries, runbook, named owner, and a quarterly cadence in calendars.
Attribution, handled honestly
The question the board will ask is "how much of this is AI?" The truthful answer has three parts and we deliver all three: what moved (measured deltas against baseline), what we changed (the intervention register with dates), and what share is reasonably attributable to AI-assisted work (the comparative analysis, with the provenance error rate stated).
Assignment to AI-assistance is not random — engineers choose which work to use AI on, and that selection biases the comparison. Saying so is the difference between a consultant and an analytics vendor.
The economics
Two operational consequences. First, set the expectation in week one: a client who was not told about the dip reads the first quarter as failure and cancels in the trough. Second, measure the dip's depth and duration, because they are the levers — independent stress-testing of DORA's own ROI calculator shows that stretching the dip from three months to twelve, with nothing else changed, turns an illustrative positive case strongly negative.
Failure modes
| Failure mode | Early signal | Prevention |
|---|---|---|
| Baseline theatre | Analysts cannot name the query behind a number | Gate 2 sign-off; query IDs mandatory |
| Usage substitution | Client asks for an adoption dashboard first | Reframe in discovery; adoption is an input, not value |
| Metric gaming | Sudden improvement with no intervention behind it | Counter-metrics; anomaly review; no individual reporting |
| The trough cancellation | Quarter-one metrics worse; sponsor goes quiet | Pre-commit the J-curve; weekly leading indicators |
| Review-queue blindness | Growing gap between PR creation and first review | Review metrics in the core set from day one |
| Pilot island | Playbook exists only as the pilot team's habits | Role-based playbooks; platform fixes; wave rollout |
| Skill atrophy | Review comments shift from design to syntax | Explain-before-accept; deliberate unassisted practice |
| Consultant dependency | Client cannot rerun the pipeline | Handover gate with a payment milestone attached |
What good looks like at exit
- Every metric has a written definition and a named owner inside the client.
- The telemetry pipeline runs on client infrastructure and has survived one refresh without us.
- Baseline, intervention register and query IDs sit in a client-owned repository.
- At least two teams can read their own scorecard and name their current constraint.
- The executive scorecard has been presented once by a client leader, not by us.
- Guardrails are enforced by the system, not by our presence.
- Next quarter's review is in calendars with an owner.