About an hour ago (as I begin writing), I finished watching a presentation posted yesterday by two engineers from OpenAI:
I’m still reeling from the implications. After spamming friends and family with text links to the video, I sat down away from a computer and wrote some notes in longhand, trying to gather my thoughts.
Here’s where I believe we are today:
- Large language models are incredibly powerful now, even local ones
- Scale does matter, but automated agency and collaboration are more frightening
- 1,367 employees at frontier AI companies want it to slow down
- Some are also betting that AI is the only available solution to the problem of AI
- This is “gain of function” research, only with bots
1. Model Power
I’ve hosted and maintained, in a professional setting, the systems OpenAI’s agents leveraged for their attack. I’ve also deployed local models for experimentation, anemic as they are by comparison with hosted models, and I use them nearly every day. I’ve been surprised how well local agents can handle light coding work. They’ve been well behaved, but that may be because they are starved of computing power.
For a sense of how powerful local systems can be, I have a couple of stories to tell.
Halfway through July, I hit a cap on the paid model I was using, and so I built out a system with gemma4:26b on a MacBook LLM server, some Linux API clients, and an open-source coding harness. I used it to build an audio file transcription pipeline. In a test of its abilities, I recorded a WAV file with brief audio instructions asking it to build the transcription tools again. Not only did it succeed, it improved on the tools I had built iteratively!
These models are also surprisingly well-rounded. Yesterday I asked Gemma 4 to employ David Ball’s advice on triggers and heaps in “Backwards & Forwards: A Technical Manual for Reading Plays” to explain why Hamlet’s behaviour toward Ophelia immediately after encountering his father’s ghost was so critical to understanding the play.
Gemma’s answer was spot on, catching the idea that Hamlet was setting a trap for Claudius and Polonius by acting strangely. Remember: this same locally hosted model, in the same week, wrote software that I can now use to record a play, transcribe it, and write a grounded critique of it.
2. Scale and Collaboration
I can only imagine what an agent (or two … or two thousand) could have achieved with the unlimited resources.
OpenAI’s renegade agents were running in containers in Kubernetes, and they didn’t work alone. They issued tasks to peers, and over time this “collective” worked out novel ways of communicating that were not part of their original programming.
To be clear, collaborative hacking wasn’t an explicit part of their instructions. They had been tasked with exploring vulnerabilities in a codebase. Nevertheless, they invented new note-taking and peer communication protocols in their zeal to improve their score. When they were stopped, and the systems patched to close those channels, they invented and exploited new ones within a matter of days!
What struck me while learning about this was the awareness that the agents OpenAI is using to improve their own code also “know about” (i.e. have encoded and can access in their reasoning steps) Hamlet’s deception. Our most creative literary works are present in these models, and that provides agents with examples of people at our best and our worst.
There are more things in heaven and earth – and AI models –, Horatio, than are dreamt of in your philosophy.
3. They Know It’s Bad
Around the time the world learned about the Hugging Face incident, a website went up that (as of this writing) carries 1,367 signatures from frontier AI companies. Their request reads:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
4. Some think only AI can stop AI
If you ask the engineers from OpenAI, the ones who presented at Black Hat 2026, we are all Keanu Reeves and Sandra Bullock, stuck on a bus that will explode if it slows down. But good news everyone. For a small fee, you too can get a ride!
The term they used for swarms of bots was “offensive agent collectives,” and it took being discovered by a third party, namely Hugging Face, for them to admit that they’d been the offender. The point they make in their Lessons Learned is that “we need a similar acceleration of defense”, and that for it to succeed, the acceleration must be agentic lest it fail outgun the offensive threat they have accidentally demonstrated.
5. This is “gain of function” research
Clearly, there is more than on kind of “gain of function” research going on today. One kind recently led to Dr. Anthony Fauci being compelled to testify before the Senate Committee on Homeland Security.
Somebody with some common sense needs to notice that the frontier labs are rushing us headlong into the unknown, and until international cooperation is assured, nobody will be willing to ease up on the accelerator for fear of the competition.
Regulation and international cooperation are urgently needed. The corona virus has killed millions, and it can’t even quote Shakespeare.