The recent randomised trial of AI usage on developer productivity published by METR caused a lot of discussion last week. The study looked at 16 experienced open source developers, working in repositories they were very familiar with, and found that on the whole the AI slowed them down.
The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
These studies are important as they generate valuable discussions about how to measure the impact of AI tools, what kinds of tasks they can help with and what kinds of people can benefit - there's too little usable evidence at the moment. The thread linked above is the most interesting deeper dive I've seen, with subsequent personal views from one of the 16 developers who participated in the study. The main conclusion is that it is too early to judge. It will take time for new cultures and habits to emerge; for instance, knowing when to fix an issue yourself and when to see how the LLM does, when the latter will give a rush of satisfaction:
LLMs are a big dopamine shortcut button that may one-shot your problem. Do you keep pressing the button that has a 1% chance of fixing everything? It's a lot more enjoyable than the grueling alternative, at least to me.
Andrew Ng: Building Faster with AI
The Y Combinator AI Startup School keeps producing big hitters - the recent talk by Andrew Ng is great. You can also access it as a podcast (via Spotify or Apple). Andrew brings a unique perspective as one of the early deep learning pioneers, founder of successful companies like Coursera, leader of AI groups at Google and Baidu, and via the AI Fund helping build a huge portfolio of AI startups. To pick just one particularly thought provoking moment, he discusses the previous rule of thumb that you need one product manager to 4-7 engineers. Now one of his teams are suggesting 2 product managers to 1 engineer as a ratio, given the speed of AI-assisted development. Quite a counterpoint to the METR study.
The Political Economy of AI: A Syllabus