25 Oct 2025: AI security trilemma; AI security compared to autoimmune disorders; autonomous AI malware; can AI be funny; really simple licensing

Agentic AI’s OODA Loop Problem

Another seminal post from Bruce Schneier on the security of AI systems. An AI agent is a system that runs in a loop. He uses the Observe-Orient-Decide-Act framework (originally developed for training US air force pilots but applied widely since) and shows how at each stage untrusted input can manipulate or subvert the agent. The reason this is such a good post is that he then adds two more great concepts.

The "AI security trilemma" is a version of the well known CAP theorem from distributed systems (you can have any two of consistency, availability or partition (network split) tolerance), or the similar rule of thumb in project management (you can have any two of cheap, fast and high quality).

This is the agentic AI security trilemma. Fast, smart, secure; pick any two. Fast and smart—you can’t verify your inputs. Smart and secure—you check everything, slowly, because AI itself can’t be used for this. Secure and fast—you’re stuck with models with intentionally limited capabilities.

He then goes on to compare AI systems inability to distinguish malicious prompts from legitimate instructions to an organism's immune system going wrong with an autoimmune disorder. The organism can't distinguish self from non-self, "or like oncogenes, the normal function and the malignant behavior share identical machinery."

Bonus interesting security link: LOLMIL: Living Off the Land Models and Inference Libraries (via ImportAI). This is a proof of concept of autonomous AI agent malware that iteratively writes and executes code using LLMs on the target device to achieve its nefarious aims. The degree of local intelligence will make this kind of approach much harder to counter.

Why is this funny? And why AI doesn’t know — yet

(Paywalled article - this is the archive link)

Bob Mankoff was for a long time the cartoon editor for the New Yorker, and was running a hugely popular caption contest from 1988 (the cartoonists draw an image; the readers suggest funny captions). It turns out for more than a decade this dataset has been used to attempt to train a funny algorithm, and Mankoff is co-author on multiple computational humour studies as well as having taught undergraduate humour theory. His work with a team at the University of Wisconsin continues, attempting to predict which caption is funnier from a set (and doing well at that now), and authoring captions given images.

Example of a pairwise comparison caption evaluation 


Recognising funny captions is far easier than writing them. The Wisconsin team found that humans overwhelmingly preferred human-authored captions to AI-generated ones. It might just be a matter of time.

Pay-per-output? AI firms blindsided by beefed up robots.txt instructions

The right of AI companies to crawl and train on web content has been a vexacious question; all the major LLMs are trained on vast corpuses gathered with little explicit licensing or permission. RSL (really simple licensing) is an attempt to create a new open standard whereby web content owners can specify licensing terms. The organisation behind it, RSL Collective, has some heavyweight folks like Eckart Walther, one of the co-creators of the RSS standard while at Netscape in 1999, and is gaining broad buy-in from publishers and content hosting sites like Reddit and Medium. Will it work, what's to stop AI crawlers just ignoring it? If the big content delivery networks like Fastly and Cloudflare get behind it, it could work, as a meaningful proportion of the web sits behind their systems. This is one to watch, as the economics of web crawling for AI training or on-demand content (during a deep research query or thinking phase) could change rapidly.








19 Oct 2025: How far away is AGI; Train a ChatGPT for $100; Claude's new Skills; AI demand growth; AI advertising direct into TV streams

AGI is still a decade away

Dwarkesh Patel is an interviewer who prepares intensely, understands the subject, and has attracted a who's who of AI luminaries (among others) to his podcast. Over the last week this 2.5 hour conversation with Andrej Karpathy has garnered a lot of attention (I'd also recommend last month's interview with Richard Sutton). Karpathy has been so immersed in the creation of LLMs for so long that his views on the evolution of the technology are well worth listening to (his opinions on the social or economic impacts I found less compelling).

An example to give you the flavour of the intellectual curiosity and openness:

Dwarkesh Patel 01:40:05

Can you give me some sense of what LLM culture might look like?

Andrej Karpathy 01:40:09

In the simplest case it would be a giant scratchpad that the LLM can edit and as it’s reading stuff or as it’s helping out with work, it’s editing the scratchpad for itself. Why can’t an LLM write a book for the other LLMs? That would be cool. Why can’t other LLMs read this LLM’s book and be inspired by it or shocked by it or something like that? There’s no equivalence for any of this stuff.

There's an interesting explanation of his new work on education, towards the end. He talks about the joy and reward of learning "depth-wise" (following a specific learning path deeper and deeper, on-demand), as opposed to the more traditional "breadth-wise", where a student is taught a broad 101 course motivated by “Oh, trust me, you’ll need this later." A great tutor (that in future could be an AI tutor) enables the depth-wise model.

Introducing nanochat: The best ChatGPT that $100 can buy

It's a double Karpathy week! I think this is going to end up as part of the LLM course they'll be doing at his company Eureka Labs. Nanochat is a full open source (MIT license) implementation of a from-scratch system to train an LLM chatbot using less than $100 of compute. Obviously at that price it'll be quite a small model, but it can be scaled by increasing the number of layers. The real value is democractising what's been seen as the exclusive domain of silicon valley machine learning engineers on insane salaries. Linked to by Simon Willison among others.

It’s trying to be the simplest complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone.

Claude Skills are awesome, maybe a bigger deal than MCP

Figuring out how to personalise and expand the capabilities of chat models has kept the big AI companies busy for a few years, with a confusing array of options being offered: custom "GPTs", GPT actions, ChatGPT plugins (deprecated), connections via Model Context Protocol (MCP). Simon Willison has explored the new skills framework from Anthropic in detail and has a great explanation. It seems really nice as it takes advantage of existing local file systems and resources in an easy to understand way. This means it won't work for lots of use cases that really need things more like the online app / app store model, but it will likely drive a surge of creativity and new functionality.

Via Barry Zhang (@barry_zyj) on X (he's a research engineer at Anthropic):

Skills actually came out of a prototype I built demonstrating that Claude Code is a general-purpose agent :-) It was a natural conclusion once we realized that bash + filesystem were all we needed

It is a good sign that Anthropic used this framework interally to provide functionality like being able to read and generate Excel, Powerpoint, PDF, before explaining and releasing it. I also like that we're harking back to the early days of Unix and the philosophies laid out in the late 1970s by people like Doug McIlroy (one of the original team at Bell Labs who developed Unix, and inventor of the pipe operator). This is the oft-quoted version from A Quarter Century of Unix by Peter Salus:

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

And another aspect, from the Bell Systems Technical Journal 1978 foreword:

Expect the output of every program to become the input to another

Skills seem exactly that: small pieces of functionality that can come together in unexpected ways, but coordinated via an LLM rather than directly by people.

AI Economics Are Brutal. Demand Is the Variable to Watch

This May Google said: "This time last year, we were processing 9.7 trillion tokens a month across our products and APIs. Now, we’re processing over 480 trillion—50 times more.". The figure is now 1.3 quadrillion (there's 1000 trillions in a quadrillion). That's an annualised growth rate of around 2500%. Will greater efficiency leading to lower costs outpace the growth in demand? There's lots of debate at the moment about AI bubbles, the massive infrastructure investments, the circular funding arrangements that spook the markets (that's the Bloomberg article with the diagram below that's been shared a lot). 

Azeem Azhar of Exponential View looks at this from several angles: Is AI a bubble? A practical framework to answer the biggest question in tech, and sees more boom than bubble currently.

Making TV advertising more accessible with ITV

A direct pipeline to create and inject AI-generated high quality video advertising into TV streams. streamr use AI tools to generate video, in the correct formats and dealing with the compliance checks, to the ITV streaming platforms for distribution. We expect this kind of thing with social media and internet video platforms, it is now reaching mainstream video streaming (what we used to call "TV"). It means very small business can push high quality TV advertising. A long way still to go, as these will inevitably become more personalised



11 Oct 2025: AI novel that regenerates daily based on news; Sam Altman on platform strategy; AI and the end of thinking; Chrome's built-in AI model; Big new AI reports

The Next Four Years


Interesing concept: an AI-written novel that regenerates daily based on recent news. As I look at it today on 11 Oct 2025, chapter 1 starts on 15 Oct 2025, with a scientist at the Centres for Disease Control seeing concerning H5N1 bird flu virus mutation rates, in the context of massive budget and job cuts at the CDC. A good counterpart to the 27 Sept post that looked at AI superforecasting.
This experiment set out to answer two questions:
First, can Al analyse eight months of U.S. government upheaval and write a near-term speculative fiction novel that predicts an imaginable future for America?
And next, can Al automatically update that novel daily based on the 24-hour news cycle without any human editorial intervention?
The credits list "Author: Claude Sonnet 4 | Editor: Gemini 2.5 Pro | Researcher: GPT-5". Thanks to Webcurios for recommending this.

An Interview with OpenAI CEO Sam Altman About DevDay and the AI Buildout

In attempting to make sense of the bilzzard of OpenAI's recent launches, this conversation between Ben Thompson (who writes Stratechery) and Sam Altman is useful. Stratechery has been a great source of technology strategy thinking over the years. The latest take compares ChatGPT to Windows' rise to dominance (OpenAI’s Windows Play), with popularity among users attracting developers:

This is a push to make ChatGPT the operating system of the future. Apps won’t be on your phone or in a browser; they’ll be in ChatGPT, and if they aren’t, they simply will not exist for ChatGPT users.

There's lots of interesting quotes in the interview, just picking out a few here. On copyright issues in products like Sora, he's predicting that rights holders will actually want their IP and content to be used:

I predict in another year, maybe less or something like that, the thing will be, “OpenAI is not being fair to me and not putting my content in enough videos and we need better rules about this”, because people want the deep connection with the fans.

And in terms of what's next:
We are going to spend a lot on infrastructure, we are going to make a bet, the company scale bet that this is the right time to do it. Given where we are with the research, with our business, with the product, what we see happening and is it the right decision or not? We will find out, but it is the decision we’re going to make.
Give us a few months and it’ll all make sense and we’ll be able to talk about the whole — we are not as crazy as it seems. There is a plan. ... I do feel like this is a once in a lifetime opportunity for all of us and well take the run at it.


Two links that are worth reading together. The first is a polemical essay by Derek Thompson (journalist and co-author of Abundance with Ezra Klein). He looks at declining writing and reading (in the US) and sees AI as the latest phenomenon after TV, the web, social media, smartphones and then streaming media that "steals our focus" and encroaches on space for deep thinking.
Do not let stories on the rise of “thinking machines” distract you from the real cognitive challenge of our time. It is the decline of thinking people.
The second is a glimpse into China's encouragement of AI in education, come what may: "Beijing is making AI education mandatory in schools", "Guangxi province has instructed schools to experiment with AI teachers, AI career coaches, and AI mental health counsellors". China will be where we first see how Thompson's concerns play out.
The one-foot tall AlphaDog ... was developed by robotics startup Weilan and is powered by DeepSeek’s AI model. In addition to practicing English with Wu’s son, it chats with him about current events, dances to his guitar music, and, through its built-in camera, helps Wu monitor the home when she is away. It has become a part of the family... “My son needs company, but we are a one-child family,” Wu said. “He asks the dog about all kinds of things — national news, weather, geography. Through AlphaDog, he is learning what the world is like.”

How to Try Chrome’s Hidden AI Model

I hadn't realised that Chrome is already shipping with a fully functional local LLM (the tiny but still multimodal Gemini Nano, that also ships with some Android phones). This post explains how to activate and access it. This kind of distribution and usage will be a lot easier for less technical folks compared to using something like Ollama, and will be a disruptive direction if it gets take up (using a local LLM from or within a web page is quite different to installing an app). Very small models that can run on laptops or phones are advancing rapidly but get less press: look at the new 3B parameter Jamba reasoning model from AI21 or the much smaller 7M parameter Tiny Recursion Model from Samsung's AI lab in Montreal.

Important new reports out recently:
  • The annual State of AI from Nathan Benaich and Air Street Capital is always comprehensive and insightful. It is also huge, with a 313-slide deck. Recommended.
  • A report on the State of AI-assisted Software Development from DORA. DORA is the DevOps Research and Assessment group, with a very long running research programme on software development (acquired by Google in 2018). A lot in here about the practicalities and cultural aspects of real life AI adoption in software teams.









27 Sep 2025: General purpose vision understanding; AI superforecasting; Western bias; the AI megasystem

Video models are zero-shot learners and reasoners

Pivotal insights from Google DeepMind published this week. Everyone was surprised at the sheer variety of tasks that LLMs could tackle; noone expected that a next-word prediction machine could write good code, or reason through problems, or many of the other applications we now take for granted that were't previously considered purely language or writing tasks. This work suggests that video models are similar, albeit a few years earlier in their evolution. They show a remarkable range of activities that Veo3 can perform. Remember, Veo3's job is just to produce a series of frames for a short video (and accompanying audio), just like an LLM's job is to produce a series of words.

Could video models be on a trajectory towards general-purpose vision understanding, much like LLMs developed general-purpose language understanding? We demonstrate that Veo 3 can solve a broad variety of tasks it wasn’t explicitly trained for: segmenting objects, detecting edges, editing images, understanding physical properties, recognizing object affordances, simulating tool use, and more. These abilities to perceive, model, and manipulate the visual world enable early forms of visual reasoning like maze and symmetry solving. Veo’s emergent zero-shot capabilities indicate that video models are on a path to becoming unified, generalist vision foundation models.

This is easiest to understand with an example, one of the very many presented. Can a video generation model successfully find a path through a maze? The model is given the maze as a starting image and simply asked to generate an animation of what happens next, given a prompt. The prompt starts with: "Without crossing any black boundary, the grey mouse from the corner skillfully navigates the maze by walking around until it finds the yellow cheese."

Here's the result:

(I've actually picked an example that only worked in 17% of their experiments, but there are many others with much higher success rates. The mouse in the maze makes a good video though! The expectation is that, like LLMs, these capabilities will continue to improve)

British AI startup beats humans in international forecasting competition

Asimov's Foundation series introduced the fictional science of psychohistory, that can predict broad societal trends and events across a galactic civilisation. Mantic is a startup attempting to build an initial version. I hadn't realised that forecasting is a competive sport. The Metaculus Cup sets a number of prediction challenges, answers are submitted, and scored 2 weeks later (so it is quite a short time frame). Mantic achieved 8th place in the summer 2025 contest, the highest ever for a bot, across a wide variety of questions predicting developments in Ukraine and Gaza, sporting results, elections, all kinds of political events. Mantic's approach appears is a multi-agent system:

Mantic breaks down a forecasting problem into different jobs and assigns them to a roster of machine-learning models including OpenAI, Google and DeepSeek, depending on their strengths.

Using AI (rather than human "superforecasters") opens up possibilities for faster experimentation. They can do "backtesting", giving the AI access to information prior to a certain date and then asking for predictions, where the outcome is already known. And they can work at much greater speed and scale. It will be interesting to see if this kind of technology starts being applied outside of finance and trading.

Which Humans?

This research from the Culture, Cognition, Coevolution Lab at Harvard looks at how LLMs answer questions compared to people from different cultures and countries. As they state:

Technical reports often compare LLMs’ outputs with “human” performance on various tests. Here, we ask, “Which humans?” Much of the existing literature largely ignores the fact that humans are a cultural species with substantial psychological diversity around the globe that is not fully captured by the textual data on which current LLMs have been trained.

It's introduced me to a new acronym - WEIRD - Western, Educated, Industrialised, Rich, and Democratic. WEIRD populations "tend to be more individualistic, independent, and impersonally prosocial (e.g., trusting of strangers) while being less morally parochial, less respectful toward authorities, less conforming, and less loyal to their local groups." Unsurprisngly, LLMs are trained on very WEIRD-biased text ("most of the textual data on the internet are produced by WEIRD people (and primarily in English)"), and so we get the "WEIRD-in WEIRD-out" problem. The World Values Survey (WVS) is a long running international survey that's been done in waves since 1981, and looks at values, norms, beliefs, and attitudes around politics, religion, family, work, identity, trust, and well-being. By essentially getting ChatGPT to answer the WVS survey questions, it can be placed on a scale for comparison. The graph below shows the ChatGPT WEIRD bias pretty clearly: ChatGPT is much more correlated with answers from countries like the US.

Why the AI “megasystem problem” needs our attention

Not the usual AI doomer nonsense. Quite the opposite: a depressingly realistic view from Susan Schneider (a philosphy professor at Florida Atlantic University) on likely problems that will come not from a single superintelligence that is created in some lab, but from the "megasystem":

"But the real risk isn’t one system going rogue. It’s a web of systems interacting, training one another, colluding in ways we don’t anticipate.... Losing control of a megasystem is far more plausible than a single AI going rogue. And it’s harder to monitor, because you can’t point to one culprit — you’re dealing with networks."

It has some parallels to systemic risk in financial markets, but the effect on individuals and culture makes it a different kind of problem:

Individuals need to cultivate awareness. Recognize the risks of addiction and homogeneity. Push for friction in learning. Demand transparency about how these tools shape our thought patterns. Without cultural pressure, policy alone won’t be enough.


21 Sep 2025: Learning to predict diseases; how to guarantee reproducibility; why we don't hallucinate as much as AI systems; the explosion of image generation capabilities

Apologies for the summer holiday hiatus; weekly updates should now resume!

Learning the natural history of human disease with generative transformers

First up, a significant piece of work that points towards a big new research area. Rather than creating a large language model, this group from the German Cancer Research Centre and the University of Heidelberg alongside the European Bioinformatics Institute in Cambridge are creating a large health model. They are using data from the 500,000 volunteers for UK Biobank to create a model to predict disease progression across multiple diseases, and testing with similar data from Finland. It is very promising, as it can already replicate the accuracy of some existing long-standing risk predictiors. It only took 1 hour of GPU time to train. They also created and published a synthetic dataset, and it appears that using that instead of real people's data was only slighly less accurate. Useful synthetic data will speed up health research: if it isn't and doesn't include personal data, it should be far easier to distribute and work with.

Defeating Nondeterminism in LLM Inference

A technical report from Thinking Machines Lab (founded by former OpenAI CTO Mira Murati) that looks at ways to make LLM output fully deterministic. Quite a technical area, as it comes down to looking at very detailed implementation design like how GPU computation is paralellised and how work is batched. However, knowing that we could have fully reproducible, deterministic LLM outputs (given some cost or computation penalties) would be important for domains like healthcare or law. Beware this isn't peer reviewed or published as a paper yet.

Knowledge and memory

I like this short piece by author Robin Sloan, because he points out something obvious that needed putting into words. We have an episodic, autobiographical memory that means we remember the process of how we learned things. AI systems don't. They appear in the world with a fully formed language generation capability. One of the reasons we're less likely to fabricate stories thinking they're true is that we'll have a history with those truths; we'll remember when we learned them.


This is an extensive repository of currently 91 examples of what you can do with the new Google Nano Banana image generation tool. The longer it has been available, the more capabilities people have figured out. Each one has examples and a detailed prompt. Everything from generating a photo of a scene from a map to creating movie storyboards. We're still in the infancy of understanding how these tools will be deployed.