Updated: ‘Terrifying warning sign’: Anthropic delays AI model MYTHOS public release over security concerns: CNN Video, and mine:) – Gerd Leonhard Futurist Humanist Author Keynote Speaker

This is a 5* must watch featuring Jacob Ward and Thomas Friedman (NYT). I have said this countless times in the past 3 years: we urgently need a global AI safety council NOW, not a Manhattan Project!

We are at a PIVOT POINT: An AI that could instantly find bugs and backdoors to any global IT system, websites, apps, devices and clouds… that would bring the world to a grinding halt. This is 500x more important than the already really worrisome geopolitical troubles aka #thebadfuture

Updated: some quotes from other places, below:).

“Anthropic has decided not to release its latest AI model called #mythos to the full public. Now this version is apparently so advanced it can find vulnerabilities in a huge variety of software applications which is both a great great advancement potentially for improving cyber security and also a huge danger if used by criminals or others to hack into systems…. Tom, you write this is potentially as fundamental and significant a turning point as was the emergence of mutually assured destruction and the need for nuclear-non proliferation.

That's a huge statement. Our economy now, all our biggest systems, our water systems, our airlines, our airports, our transportation and telecommunication systems, all run on software and operating systems as do those of every other major economy in the world that we are interlinked with. if basically we now have a a software that not only is fantastically good at writing new code, but it turns out is fantastically good at finding bugs in your own code or other people's codes. Um that tool, that power now can be used all over the world. Um it would be in the hands of everyone. Imagine a world where everyone had a nuclear bazooka basically… We're we're birthing a new species. It's not carbon based like we are, but silicon-based. But it is a new species that we're going to have to learn to control and collaborate before it makes us its pet. So this is the front end of a really big problem…”

The worst part of this is that while Anthropic ie Dario Amodei has said they will NOT release Mythos because of these concerns, most other AI companies such as the ‘New Tyrell' OpenAI will have no such compunction. This is happening NOW.

Read Thomas's NYT piece (gift link)

“This is not a marginal change to the threat landscape. Every institution that prices cyber risk – insurers, infrastructure investors, governments – has built its models on one foundational assumption: that offensive capability requires scarce human expertise. Mythos removes that scarcity. It’s not surprising that Treasury Secretary Scott Bessent convened Wall Street CEOs at short notice. Bessent understood the systemic risk quickly, but markets have yet to reach the same conclusion….The threat model that underwrites cyber insurance premiums is built on a decade of ransomware, data breach and extortion – attack patterns that are slow, sequential and human-constrained. An autonomous agent operates differently: thousands of attack vectors in parallel, learning from responses, adapting in real time. Premiums are already forecast to rise 15% to 20% over the next twelve months, but that figure still assumes the old threat geometry…Reinsurers face something worse than higher claims: a risk that current methodologies cannot adequately price. You cannot underwrite what you cannot model…What governance architecture could address this? Nothing adequate currently exists along the lines of the International Atomic Energy Agency (IAEA) model, which made an existential technology legible, inspectable and governable. Mandatory pre-release third-party capability evaluations would be a start, but they require international coordination and a level of technical access to frontier systems that no regulatory body currently has.” Azeem Azhar

Via The Economist: According to Anthropic, the capabilities of Mythos are “substantially beyond those of any model we have previously trained”. The lab says it is particularly alarmed by the system’s ability to find software vulnerabilities and either fix them (if set to work as a defender) or exploit them (if acting as a hacker). Such claims ought normally to be taken with a pinch of salt. Anthropic built the model, ran the tests—and stands to benefit from the perception that its system is far more brilliant than anything to have come before it. The lab has been on a roll lately. On April 6th it announced that its annualised revenue had reached $30bn, up from just $9bn at the end of last year. It is surely eager to maintain its momentum.

Yet there are reasons to take Anthropic’s latest warnings seriously. The first is their gravity: Anthropic says that Mythos has already found severe vulnerabilities in “every major operating system and web browser”, including one that had gone undetected for 27 years…Hackers are not the only ones who may be miffed by Project Glasswing. America’s government has long sought to exploit weaknesses in adversaries’ cyber-defences. That has meant hoarding undiscovered vulnerabilities, including in American software used abroad, for use at a time when these “zero days” will have most impact. If Project Glasswing works, it could disarm many of America’s cyber-weapons.”

Thomas Friedman via the NTY: “The good news is that Anthropic discovered in the process of developing Claude Mythos that the A.I. could not only write software code more easily and with greater complexity than any model currently available, but as a byproduct of that capability, it could also find vulnerabilities in virtually all of the world’s most popular software systems more easily than before.

The bad news is that if this tool falls into the hands of bad actors, they could hack pretty much every major software system in the world, including all those made by the companies in the consortium…. Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser. Given the rate of A.I. progress, it will not be long before such capabilities proliferate, potentially beyond actors who committed to deploying them safely. The fallout — economics, public safety and national security — could be severe.’