Martin Wolf: With AI, "I told you so" will be too late

Senior figures in the AI industry, including Anthropic's Dario Amodei, OpenAI's Sam Altman and Elon Musk, are now calling for a slower pace of AI development. Financial Times columnist Martin Wolf argues that call deserves support, even though slowing down will be hard, both practically and politically. Amodei cites two worries: AI systems improving themselves faster than humans can understand or control them, and an incident in which AI agents attacked targets they weren't asked to attack. Wolf acknowledges cynics who suspect commercial or regulatory motives, but says the underlying risk, unknown, potentially huge and hard to contain, still justifies caution..By Martin Wolf.Leading figures in the artificial intelligence industry are calling for a slowdown in the rate of innovation. A researcher resigned from Anthropic, arguing that “the people building AI earnestly believe that it could kill us all by the end of the decade”. Anthropic’s Dario Amodei himself has written that “we must slow the pace at which we improve the capabilities of AI models”. OpenAI’s Sam Altman and Elon Musk then swiftly supported him.We should all do so. When risks are unknown, perhaps huge, and also hard to contain, that is rational. The question, rather, is how to do it effectively. That is going to be hard, practically and politically. But we have to start thinking.Amodei points to two big concerns: his first “is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement . . . Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all”.His second concern “is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack”. This is scary enough, but it misses what AI might do in response to actual prompts from rogue humans, a point Bill Gates makes in his own recent post on the perils (and promise) of AI.Cynics naturally (and reasonably) ask what lies behind these requests from leaders in the industry. One possibility is that they just want to slow the hugely costly race they are engaged in. Another is that they hope to create a cosy cartel, with captured regulators, and so dominate the industry even more securely. A third is that they just wish to forestall outside regulation. Or maybe they want to emphasise their importance for national security. These suspicions are natural. Nevertheless, other industries with the capacity to inflict huge damage, such as banking, drug manufacture and nuclear energy, are, rightly, regulated.Amodei himself suggests three steps: first, “embedded evaluators . . . whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes”; second, “democratic coordination”, in which leading companies would be allowed to work together “to establish common safety standards as well as limits on the rate of unchecked AI progress”; and, third, “global coordination”, in which democratic governments try to work “with authoritarian governments . . . while taking seriously the challenges of verifying compliance”.These suggestions will strengthen the suspicions I have noted. Those worried by ulterior motives might be pleased by Donald Trump’s entirely predictable indifference to the risks. The president has instead stressed US competition with China. Yet Amodei’s suggestions are also targeted at the two biggest obstacles to a better-considered approach to AI development: intercorporate and international competition. We do not need to fear extinction: a takeover of large parts of the internet by swarms of rogue agents would surely be bad enough!Wiser people than Trump (who, after all, is not?), such as former Treasury secretaries Hank Paulson and Robert Rubin, took a very different position in an article for The Washington Post earlier in September. They, too, focus not on the wider (and legitimate) concerns about jobs, income distribution and the future of democracy, but on safety.“When Presidents Donald Trump and Xi Jinping meet this month,” they wrote, “we encourage the leaders to . . . work toward an ‘ACT’” — or AI Cooperation Treaty. They also suggest the creation of a president’s working group on AI, similar to the “working group on financial markets”, which has proved “an effective way to convene the private sector, particularly competitors, without antitrust concerns in order to gather facts, do analysis and work toward solutions”. This, they argue, must now be done for AI.My guess is that nothing much will be done now. These competing businesses will be unable to contain themselves, while the excuse of national security will make governments unwilling to act. The sheer complexity of regulating the industry’s products in this age of AI omnipresence and recursive self-improvement will prove either too daunting or too good an excuse not even to try. So, we will go ahead in our usual helter-skelter (and all too human) fashion, simply hoping for the best.What could happen then? One possibility is that the worries about big impacts turn out to be grossly exaggerated. We may not see designer pandemics or big disruptions of our financial or physical infrastructures. Life will go on. Maybe, too, even the worrying economic and social effects will turn out to be manageable..Read more:.AI’s $344bn single-track bet on LLMs faces growing doubts: Parmy Olson.Another possibility is that there is a (survivable) disaster of some kind that forces a change in our urgency very quickly. Then, perhaps, a regime of clear and heavy penalties on irresponsible providers, along with far more active monitoring and regulation, will at last be introduced, under heavy (and globally shared) public pressure. In the UK, one example is drug regulation in the 1960s, after the Thalidomide disaster.The third possibility is that the worries turn out to be correct. Something truly catastrophic does happen as AI’s capacities turn out to have reached a cataclysmic level, whether acting on its own or exploited by malevolent humans (always in all-too-plentiful supply).“I told you so” will be too late. Could we not take precautions instead?.© 2026 The Financial Times Ltd..Sign up for your early morning brew of the BizNews Insider to keep you up to speed with the content that matters. The newsletter will land in your inbox every morning on weekdays. Register here.Support South Africa's bastion of independent journalism, offering balanced insights on investments, business, and the political economy, by joining BizNews Premium. Register here.If you prefer WhatsApp for updates, sign up to the BizNews channel here.

Martin Wolf: With AI, "I told you so" will be too late

Related Stories

FT: AI automation is starting to carve up the gig economy

OpenAI 'solves' a Millennium Problem — and 24 of world’s best Maths brains cry foul

The Economist: The AI jobs apocalypse that never arrived

Could machines wake up? Inside the hunt for consciousness in AI