AI in development: what the data actually shows

Nearly every development team has adopted some AI tooling by now. The question almost nobody answers with data is the uncomfortable one: is it actually shipping software faster? Controlled studies on this exist, and their conclusions disagree with each other. Understanding why they diverge is far more useful than picking the number you like.

Two studies, two opposite results

In 2023, a controlled experiment asked 95 freelance developers to implement an HTTP server in JavaScript. The group with GitHub Copilot access finished 55.8% faster than the control group. It is one of the most quoted numbers in the industry, and it is real.

In 2025, METR ran another controlled experiment: 16 experienced open-source developers, 246 real tasks, inside the mature repositories they maintain themselves. The result: with access to early-2025 AI tooling, they were roughly 19% slower.

Both studies are methodologically serious. They do not contradict each other. They measured different things.

What explains the gap

The decisive variable is not the tool. It is the task context.

The 2023 study used a greenfield, self-contained, well-specified task, solved by people with no prior familiarity with that code. That is the scenario where a model shines: the pattern is common, the context it needs fits in the window, and there is no history to respect.

The METR study was the opposite extreme: a large, mature codebase with implicit conventions, worked on by developers who already knew every corner of it. Here the model lacks context it cannot get, and the human spends time reviewing, correcting and discarding suggestions. The cost of verifying starts competing with the gain from generating.

The practical read: AI gains are high in new and peripheral code, and drop (possibly turning negative) in core, older code that its maintainers know deeply.

Perception lies, including yours

The most uncomfortable detail in the METR study is not the 19%. It is that participants believed they had been around 20% faster, while actually being slower. Before starting, they forecast a 24% speedup.

In other words, the feeling of speed survived the evidence against it fully intact. That matters because most enterprise AI adoption decisions rest on exactly that feeling, collected through an internal satisfaction survey. Perception is legitimate data, but it is not a throughput measurement.

DORA's framing: AI is an amplifier, not a shortcut

DORA's 2025 report on AI-assisted software development lands on a framing that reconciles both experiments well: AI amplifies the characteristics your delivery system already has. Where there is a solid foundation, small batches, reliable tests and fast feedback, AI accelerates. Where there are bottlenecks, it increases the pressure on them and surfaces the problem faster.

That is why the same tool produces opposite results in two teams at the same company. The variable is not the model, it is the platform around it.

How to measure without fooling yourself

The most common mistake is measuring activity: accepted lines, suggestions used, commits per day. Those are metrics AI inflates by construction, and they say nothing about delivered value.

The SPACE framework exists precisely for this. It proposes five dimensions (satisfaction and well-being, performance, activity, communication and collaboration, efficiency and flow) and argues productivity cannot be reduced to a single metric. Combined with DORA's delivery metrics (lead time, deployment frequency, change failure rate, time to restore), you can see whether AI is shortening the path to production or just generating more code to review.

An honest, cheap test: take two comparable teams, change AI adoption in one of them, and watch lead time and change failure rate for a quarter. If neither moved, the gain you are celebrating is perception.

What this means for your team

Three conclusions you can act on:

1. Do not expect uniform gains. Point AI first at new code, scaffolding, tests, repetitive migrations and exploring unfamiliar code. Keep expectations low in core, mature code.
2. Do not measure by feel. Instrument lead time and stability before scaling adoption, or you will not be able to tell real gains from enthusiasm.
3. Treat it as a platform investment. If tests are slow and flaky and review is the bottleneck, AI makes the queue worse. Fixing that comes first, and it decides whether AI pays off at all.

The useful question is not "does AI work?". It is "where in our flow does it pay, and how will we know?".