08/06/2026
How should organizations measure AI productivity?
METR’s latest productivity update is a good example. Their original 2025 study suggested experienced developers completed tasks more slowly with AI than they expected. A follow-up study produced different estimates, but the researchers were careful to treat it as a final conclusion.
Instead, they highlighted a more important insight: as AI becomes part of everyday development, measuring its impact becomes much harder. Developers choose different tasks, rely on AI in different ways, work with multiple agents simultaneously, and increasingly don’t want to work without AI at all.
This raises a practical question for potential AI adopters: if the work itself changes, should we still apply the same productivity metrics?
Time saved writing code only tells us part of the story. AI also changes how teams review, plan, test, document, and take on entirely new types of work.
That’s why it makes sense to start measuring AI adoption before deployment: establish a baseline, define the outcomes you want to improve, and decide which metrics actually reflect them. At EffectiveSoft, we have these conversations before any new tool or workflow goes live. Because success is difficult to assess without first agreeing on what it looks like.
If you were evaluating an AI initiative after six months, which metrics would convince you it had actually improved software delivery?
Still figuring out what "success" looks like for your AI initiative? Explore our AI expertise and let's define it together: https://hubs.la/Q04s5zny0