A few links this week on the theme of "open-ended" AI systems that continuously learn and improve, rather than having a single period of training followed by repeated use of the same model. These aren't entirely new ideas: reinforcement learning and genetic algorithms / genetic programming systems have often been deployed in an open-ended fashion. But this work does bring it all together into a bigger learning loop with LLMs, and the direction feels like the next big step forward.
In a 2003 paper Jürgen Schmidhuber proposed a Gödel Machine that could self-improve, with a problem solver that tries to solve problems set for the machine and a searcher that can rewrite the machine's code to improve it. It's a kind of meta learning (learning to learn). The article above describes work this year on a Darwin-Gödel Machine. This is a coding agent that improves its own code. Why "Darwin"? Because it also has an element of genetic programming. It starts with an agent, attempts to improve it, evaluates its performance with a software engineering benchmark, and adds it to its archive of agents. Next time around, it can select a "parent" agent to modify to create children. The array of possible agents means it can search over a big space of solutions. In this case the agents' LLMs are fixed (it isn't trying to train new foundation models each time, which would be pretty expensive); it is optimising the tool use and workflows to create better coding agents. The result is a significant improvement (from 20% to 50% on the SWE-bench software engineering benchmark, compared to human-designed agents at around 70%).
As the article above notes, the 2017 Asilomar AI Principles include:
22. Recursive Self-Improvement: AI systems designed to recursively self-improve or self-replicate in a manner that could lead to rapidly increasing quality or quantity must be subject to strict safety and control measures.
Open-Endedness is Essential for Artificial Superhuman Intelligence
This is a paper from the 2024 ICML conference; work by Edward Hughes and colleagues from Google Deep Mind (Edward gave an excellent talk at RAAIS 2025 in London). They're thinking about ways to create "ever self-improving" AI systems. They define an open-ended system as one that produces a series of novel and learnable artifacts, from the point of view of an observer. Novelty means artifacts becoming less predictable, whereas learnability means that you're more likely to predict the next artifact if you've seen a longer history of previous ones. The role of the observer is to determine novelty and learnability (different observers may remember more or less history for instance). An example helps. A research student will find a series of publications from a research lab novel if each new paper has something surprising, but also learnable when reading the previous papers will help them predict the next one. An AI example is AlphaGo, that can continually discover new policies to improve its performance at Go. This is a position paper, quite theoretical so it merits several reads to get, but it lights the path towards foundation models that can continually improve themselves, generating new hyoptheses or problems to solve.
SEAL is a framework that enables language models to generate their own finetuning data and optimization instructions—called self-edits—in response to new tasks or information. SEAL learns to generate these self-edits via reinforcement learning (RL), using downstream task performance after a model update as the reward.
Early work from some MIT students; an example of successfully putting similar ideas into practice albeit in some quite specific domains.
Frontier AI systems have surpassed the self-replicating red line
Cute experiment: can an LLM reproduce itself (get another copy of itself running on a virtual machine) given access to a command line. Usually yes. Not convinced this is particularly surprising, and self-replicating computer viruses have existed since the 1970s, but it is another important ingredient for open-endedness.
New jargon watch:
As context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly
I really like the term “context engineering” over prompt engineering. It describes the core skill better: the art of providing all the context for the task to be plausibly solvable by the LLM.
Thanks to Simon Willison