Dwarkesh Patel is an interviewer who prepares intensely, understands the subject, and has attracted a who's who of AI luminaries (among others) to his podcast. Over the last week this 2.5 hour conversation with Andrej Karpathy has garnered a lot of attention (I'd also recommend last month's interview with Richard Sutton). Karpathy has been so immersed in the creation of LLMs for so long that his views on the evolution of the technology are well worth listening to (his opinions on the social or economic impacts I found less compelling).
An example to give you the flavour of the intellectual curiosity and openness:
Dwarkesh Patel 01:40:05
Can you give me some sense of what LLM culture might look like?
Andrej Karpathy 01:40:09
In the simplest case it would be a giant scratchpad that the LLM can edit and as it’s reading stuff or as it’s helping out with work, it’s editing the scratchpad for itself. Why can’t an LLM write a book for the other LLMs? That would be cool. Why can’t other LLMs read this LLM’s book and be inspired by it or shocked by it or something like that? There’s no equivalence for any of this stuff.
There's an interesting explanation of his new work on education, towards the end. He talks about the joy and reward of learning "depth-wise" (following a specific learning path deeper and deeper, on-demand), as opposed to the more traditional "breadth-wise", where a student is taught a broad 101 course motivated by “Oh, trust me, you’ll need this later." A great tutor (that in future could be an AI tutor) enables the depth-wise model.
Introducing nanochat: The best ChatGPT that $100 can buy
It's a double Karpathy week! I think this is going to end up as part of the LLM course they'll be doing at his company Eureka Labs. Nanochat is a full open source (MIT license) implementation of a from-scratch system to train an LLM chatbot using less than $100 of compute. Obviously at that price it'll be quite a small model, but it can be scaled by increasing the number of layers. The real value is democractising what's been seen as the exclusive domain of silicon valley machine learning engineers on insane salaries. Linked to by Simon Willison among others.
It’s trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone.
Claude Skills are awesome, maybe a bigger deal than MCP
Figuring out how to personalise and expand the capabilities of chat models has kept the big AI companies busy for a few years, with a confusing array of options being offered: custom "GPTs", GPT actions, ChatGPT plugins (deprecated), connections via Model Context Protocol (MCP). Simon Willison has explored the new skills framework from Anthropic in detail and has a great explanation. It seems really nice as it takes advantage of existing local file systems and resources in an easy to understand way. This means it won't work for lots of use cases that really need things more like the online app / app store model, but it will likely drive a surge of creativity and new functionality.
Via Barry Zhang (@barry_zyj) on X (he's a research engineer at Anthropic):
Skills actually came out of a prototype I built demonstrating that Claude Code is a general-purpose agent :-) It was a natural conclusion once we realized that bash + filesystem were all we needed
It is a good sign that Anthropic used this framework interally to provide functionality like being able to read and generate Excel, Powerpoint, PDF, before explaining and releasing it. I also like that we're harking back to the early days of Unix and the philosophies laid out in the late 1970s by people like Doug McIlroy (one of the original team at Bell Labs who developed Unix, and inventor of the pipe operator). This is the oft-quoted version from A Quarter Century of Unix by Peter Salus:
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
And another aspect, from the Bell Systems Technical Journal 1978 foreword:
Expect the output of every program to become the input to another
Skills seem exactly that: small pieces of functionality that can come together in unexpected ways, but coordinated via an LLM rather than directly by people.
AI Economics Are Brutal. Demand Is the Variable to Watch
This May Google said: "This time last year, we were processing 9.7 trillion tokens a month across our products and APIs. Now, we’re processing over 480 trillion—50 times more.". The figure is now 1.3 quadrillion (there's 1000 trillions in a quadrillion). That's an annualised growth rate of around 2500%. Will greater efficiency leading to lower costs outpace the growth in demand? There's lots of debate at the moment about AI bubbles, the massive infrastructure investments, the circular funding arrangements that spook the markets (that's the Bloomberg article with the diagram below that's been shared a lot).
Azeem Azhar of Exponential View looks at this from several angles: Is AI a bubble? A practical framework to answer the biggest question in tech, and sees more boom than bubble currently.
Making TV advertising more accessible with ITV
A direct pipeline to create and inject AI-generated high quality video advertising into TV streams. streamr use AI tools to generate video, in the correct formats and dealing with the compliance checks, to the ITV streaming platforms for distribution. We expect this kind of thing with social media and internet video platforms, it is now reaching mainstream video streaming (what we used to call "TV"). It means very small business can push high quality TV advertising. A long way still to go, as these will inevitably become more personalised