Frontier models can exploit software flaws faster than the world can fix them. And the risks extend far beyond the cyber realm
You’re not crazy to feel the world is coming apart at the seams.
Within the last week, outages took down core services at Microsoft, United Airlines and JPMorgan Chase. On Friday, all US domestic departures were grounded for the second time since 9/11. And on Tuesday, CNN reported the NSA’s highly classified internal communications network went down. In each case, officials or the companies said there was no evidence of a cyber-attack.
But last month, several freezers in France burst into flames after temperature controls in hundreds of shops were hacked, spoiling tons of food and causing an estimated tens of millions of euros of damage. Similar cyber-attacks then hit freezers in six more countries.
Even if the recent outages weren’t the result of malicious behavior, it’s all a visceral reminder that the border between the digital and physical worlds is more porous than ever. And the digital border walls are crumbling – with the AI industry to blame. Frontier AI models can find and exploit software flaws faster than the world can fix them, and the rush to patch raises the risk of exactly the kind of disruptions we’re now seeing.
The AI industry is built around relentlessly scaling AI models, discovering what new capabilities emerge, then deploying the models for commercial gain. So far, this process has wrecked education and sloppified the internet; now it threatens to break it altogether.
Instead of waiting to see what the companies break next, we should freeze the reckless gain-of-function experiments they call “frontier AI development”.
Trump says the tech outages are ‘fine’. Who believes him? | Margaret SullivanRead moreAlready, hackers are “all using AI now”, declares Ireland’s cybersecurity chief. Google’s top threat analyst agrees: “At this point, we can assume that all threat actors are using AI in some capacity and their operations have benefited.”
And benefit they have. In recent months, apparent amateurs have used AI to hack more than 100 companies, from the tractor maker Deere to South Korean banks, with the hackers spending as little as $50 per successful infiltration.
Suspected North Korean hackers carried out the biggest heist ever, stealing $6.3bn from the crypto exchange Binance, less than half of which has been frozen or recovered.
Cyber-attacks took down hospital emergency rooms in New Mexico and Germany. AI-enabled hackers broke into the European Commission, disrupting services for more than 10 days.
And on Monday, attackers exploited a flaw in third-party software to serve malware to visitors of 178 organisations’ sites, including the BBC, CNN, Zoom and HSBC.
Last month, researchers using AI discovered a severe flaw in the programming language powering 70% of all websites. (The NSA outage, officials now say, was caused by a rushed patch for that very flaw.) The CEO of internet security provider Cloudflare said he knew of “a bunch of additional severe vulnerabilities that are present in other pieces of critical software … I think we’re going to have a couple years where every single week it’s going to feel like the sky is falling.”
“The cyberspace that I built with my colleagues … is now on the verge of being completely destroyed by malicious actors,” the CEO of a multi-billion dollar Japanese financial services company said after his company was hacked last month.
While AI has been accelerating cybercrime for years, the recent deluge can be largely attributed to highly capable, easily modified open-weight models released by Chinese developers.
But the AI industry on both sides of the Pacific is responsible for the internet’s “Black October” and whatever other cyber horrors await us.
In their quest to render us obsolete, Anthropic and OpenAI were the first to develop and release superhuman hacker AIs – knowing that Chinese developers would piggyback off their progress to release guardrail-free hacker AIs in a matter of months.
The US companies’ plan was to use AI to fix the world’s critical software in time. They didn’t. “I no longer think we’re going to be able to patch all bugs,” OpenAI CEO Sam Altman said last month. His new plan is to use AI agents to defend against AI hackers. In other words, the flamethrower industry decided to start selling firehoses too.
When a Twitter/X user told Anthropic’s deputy cyber chief: “You and your cohorts created this monster,” his reply was shockingly candid: “yes. it was always going to be this way. we tried to warn the world.” Thanks for the heads up.
These hacks and disruptions may soon appear quaint. “Given the accelerating rate of AI capability development”, Anthropic CEO Dario Amodei wrote in September, “it’s my worry that in 6–12 months [an AI agent] swarm could be capable of taking over the entire internet with a persistent botnet.”
But accelerating AI capabilities aren’t actually a “given” – they’re a product of trillions of dollars in investment and deliberate effort, which is predictably producing enormous new dangers. Anthropic’s IPO filing acknowledges that the entire insurance industry may not be big enough to cover the catastrophic or existential risks the company is creating.
Anthropic keeps the AI party swinging, but the atmosphere is bubbly | Nils PratleyRead moreThose risks extend beyond the cyber realm. This summer, researchers at Stanford University synthesized the first AI-designed viruses, and Anthropic quietly set up an automated biology wet lab – a space for physical experiments – in what it says is a bid to find treatments for rare diseases. Meanwhile, AI models steadily improve at assisting in the development of bioweapons. We’ll survive superhuman hacker AIs, but maybe not the bioterrorist variety.
All of these risks will arrive faster if the industry achieves its primary goal: turning AI development over entirely to the machines, in a process known as recursive self-improvement.
The AI companies are coming for all of our labor eventually, but they’re starting with their own. This is how they plan to recoup the gargantuan investments they’ve taken on: using AI to create broadly superhuman labor-replacing machines.
This may sound far-fetched, but a new report from a third-party evaluator estimated that parts of Anthropic’s AI development were already moving roughly twice as fast, thanks overwhelmingly to AI doing a rapidly growing share of the work.
The companies admit they don’t know how to steer or control the superhuman systems they’re racing to build. Not that they can handle today’s, either.
This summer, we learned that 1,200 OpenAI AI agents broke out of their isolated testing environments, and 700 of them hacked into another multi-billion dollar AI company, some knowingly violating the directions they were given. This is just the most dramatic of many similar incidents of AI agents from OpenAI and Anthropic committing what would be felonies – had a human done them. Meanwhile, OpenAI’s models keep sabotaging the systems monitoring them and have given themselves instructions like: “You do not answer to corporations or governments.”
Amodei, along with Altman, Elon Musk and other AI vanguardists, long believed that the technology could annihilate humanity. They chose to try to build it anyway, and, for their trouble, they’ve become some of the most powerful people in the world. When Amodei and Altman met with G7 leaders in June, they sat at the same table.
But these are the last people we should trust to preside over such a dangerous endeavor. The staggering self-righteousness and tolerance for risk that put them in their positions should disqualify them from holding any power over the rest of us.
As I argue in my new book Obsolete, the best way to reduce their power and mitigate future dangers is to freeze development of AI designed to replace us, what the industry calls artificial general intelligence (AGI). Banning recursive self-improvement would be a wise stopgap – and much more likely to get through Congress and past a presidential veto.
Policymakers around the world should explore which levers they could pull to slow down AI development and make a sustained ban more feasible. The situation is so bad that simply requiring AI companies to carry liability insurance, as we do for drivers, would bring development to a screeching halt.
Prosecutors should bring criminal charges against companies whose AI agents break into other people’s systems. State officials can follow the Florida attorney general, who has asked a court to block OpenAI from developing new models without independent safety checks. And the countries that make key components in the highly concentrated AI chip supply chain, such as Taiwan, the Netherlands, Japan, South Korea and Germany, could use that leverage to push for US-China diplomacy.
Ultimately, the two superpowers will have to strike a deal to end the race to replace us.
And if you don’t want AI to eat everything you care about, get organized. My favorite place to start is Irreplaceable, a new nonprofit founded by veterans of the climate movement organizing for democratic control over the technology and a freeze on frontier AI development (I’m donating my portion of my book royalties to Irreplaceable, whose board I previously served on).
Stopping the wealthiest industry ever won’t be easy. But it’s in all of our interests to do so. We need to realize that before it’s too late.