Ivo Vegter: Paid to create panic: Who's funding the AI apocalypse?

In this superb column for The Daily Friend, Ivo Vegter poses an important question: Who pays to keep you frightened of AI? Sabine Hossenfelder was offered sponsorship money to tell her YouTube audience that artificial intelligence will kill us all. She said no. One PhD computer scientist read the research instead. Her verdict: AI that improves itself is real, but it has limits. Humans still choose the game and keep the score. Follow the money behind the Centre for AI Safety and you find Effective Altruism funders, Anthropic investors and a marriage. The motive? A regulatory moat. The real threat is closer to home: AI slop made to farm clicks, poisoning politics. South Africans remember Bell Pottinger. We know the script..*By Ivo Vegter for The Daily Friend.Influencers are being paid to warn you that a self-improving AI will wipe out humanity. The research says otherwise. The real danger is already in your Facebook feed.There is a strange cognitive dissonance at the heart of the artificial intelligence business.To lenders and investors, the model builders and hyperscalers pitch their product as the next big thing, deserving of unprecedented torrents of capital. To regulators and the public, the same companies warn that it might escape human control by improving itself, faster and faster, until it ends the human race.Those claims make sense together only as sales pitches, aimed at different buyers.Recently, I argued that when an industry begs to be regulated, you should find out who it wants kept out. This time, the question is who is paying to keep the public frightened.Paid to panicOn 5 September, the German physicist and science YouTuber Sabine Hossenfelder published a video titled I Was Offered Money to Tell You AI Will Kill Us. An organisation warning that AI is an existential threat offered to sponsor her, provided she read the sentences it wrote and quoted the references it chose. She said no.The Center for AI Safety (CAIS), she said, hands out grants to creators to make videos warning of the coming apocalypse. “Would you know it, the apocalypse comes with a marketing budget,” she says, with dry sarcasm.She is not alone. In August, the software developer and YouTuber Ben Awad posted that he had received his “2nd anti-ai sponsor request”.On 22 September, the computer scientist who runs the House of El channel posted a video whose title tipped its hat to Hossenfelder: Like Sabine Hossenfelder, I was offered money to tell you AI will kill us. She, too, turned the money down.In fairness, Hossenfelder notes that the pro-AI side plays the same game, citing a nonprofit part-funded by OpenAI president Greg Brockman that offered a journalist $5,000 for a single TikTok video.CAIS, and its warning about an extinction-level threat, has the open support of a Who’s Who of the tech industry, from Sam Altman and Dario Amodei, to Bill Gates and Bill McKibben, to Bruce Schneier and Ray Kurzweil.Signing CAIS’s campaign statement, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” is of course largely performative. Who wouldn’t claim publicly that they’re against causing human extinction?(Schneier later walked his support back, saying: “I actually don’t think that AI poses a risk to human extinction. I think it poses a similar risk to pandemics and nuclear war—which is to say, a risk worth taking seriously, but not something to panic over. Which is what I thought the statement said.”)Reading the papersThe mononymous El, who has a PhD in computer science, did something a lot more uncommon than just refusing money for regurgitating propaganda.She worked her way through about a dozen papers to find out whether anything supported the claim that AI will recursively improve itself beyond human control, and then exterminate us.El found that recursive self-improvement (RSI) is real. The Darwin Gödel Machine rewrote its own agent code and lifted its score on a standard coding benchmark from about 20% to 50%. Google DeepMind’s AlphaEvolve bettered a 56-year-old algorithm for matrix multiplication, and improved some of the infrastructure used to train AI. That is AI making AI better.But every one of those results had something outside the system keeping score.A survey of 1,250 papers on AI self-improvement finds that self-improvement is strongest where the objective is a formal proof or passing a test that tells the system whether it succeeded, and weakest where the system has to judge itself. At the very top of that hierarchy sits what the authors call the “research direction-setting” bottleneck: choosing what is worth working on at all. That is what keeps humans in the loop.A Princeton-led team tested this directly. It gave frontier AI agents the central questions of two unpublished NeurIPS conference papers, six days, and thousands of dollars in computing budget. The agents did all the engineering. They made no substantial progress on the questions. The original authors rejected both papers, unambiguously.OpenAI’s own report this month says high-level planning remains “a minimal fraction” of its agents’ output, and that more than half of successful four-to-eight-hour tasks still needed a human to step in. It admits it does “not yet know how to safely get all the way to aligned, full RSI”.Bounded self-improvementThat AI models self-improve within set boundaries is key to thinking about the threat of RSI.A chess engine can become far better at chess than any human. An AI engine that tinkers with its own ability to play chess can become even better. But it has never once decided it would rather play Go.Give an AI a score and it will chase that score relentlessly, sometimes by cheating (El describes research agents that gamed their own experiments). What it does not do is choose the game. People set the task, build the scoreboard, and decide what counts as progress.Even if that changed, a runaway explosion would not follow.Nine economists who modelled the economics of recursive self-improvement found that its feedback loops are not currently strong enough to sustain themselves, although they appear to be strengthening. Smarter software still needs compute, which needs chips, fabs and power stations. As El puts it, you still can’t download a semiconductor fab.AI agents can do real mischief in the digital world. OpenAI’s agents hacked Hugging Face, as I noted before. But extinction is an extreme claim. It doesn’t mean AI can be used to kill a lot of us. The runaway recursive self-improvement scenario is that AI will choose to kill all of us, including the ones who live in the woods and have never touched a computer.Every extinction scenario is a long chain of “and then”, in which each link is a separate, doubtful claim. If you flip a coin on each one of a dozen links, the chance that all of them come up tails is very small.El does not conclude that AI is safe. She concludes that “plausible”, “forecast” and “demonstrated” are different things, and policy should not treat them as one. Watch her video. It is long, very interesting, and unsponsored.Follow the moneySo who funds the fear?CAIS was founded in 2022 by Dan Hendrycks and Oliver Zhang. Its 2023 statement that extinction risk from AI should be a global priority was signed by 695 AI scientists and industry luminaries. Hendrycks himself advises Elon Musk’s xAI and Scale AI, for a symbolic dollar a year.All the world’s major media ran with the doom story, each doing their best to out-sensationalise the other. I haven’t read all the coverage, but I’m going to declare The Guardian the provisional winner, with: “Five ways AI might destroy the world: ‘Everyone on Earth could fall over dead in the same second’.”According to a tally by Nirit Weiss-Blatt, a researcher who is an outspoken critic of AI doomerism, CAIS has received about $12.5 million from Open Philanthropy, $7.5 million from Jaan Tallinn’s Survival and Flourishing Fund, and $6.5 million from Sam Bankman-Fried’s FTX Future Fund, of which it later returned $5.2 million to the FTX bankruptcy estate.Bankman-Fried, now serving 25 years for fraud, was the poster boy and chief financier of the Effective Altruism movement. He was also an early backer of Anthropic, leading its Series B funding round.Open Philanthropy, since rebranded as Coefficient Giving, is Effective Altruism’s biggest bank. It was co-founded by Facebook billionaire Dustin Moskovitz, who invested in Anthropic, and Holden Karnofsky, who now works at Anthropic and is married to its co-founder and president, Daniela Amodei.To be fair to CAIS, it says it has not taken Open Philanthropy money “for years”, and now calls Effective Altruism “controlled opposition”. Perhaps. But the web of money, people and marriages that built it remains wound tightly around the very companies it warns us about, and whose leaders signed its campaign statement.Hidden motiveA doom lobby that is so closely connected to the AI industry itself (much like the hedge fund billionaires with massive oil portfolios that fund climate activist groups) must have a motive it does not advertise.That motive is not hard to discern. Tell investors your product is the most powerful thing ever built. Tell legislators it is so dangerous that only a handful of responsible, licensed incumbents should be permitted to build it.Both messages sell the same thing, from a different perspective. The apocalypse rhetoric is the marketing that buys a regulatory moat around the product, and investors love world-changing products surrounded by moats.The real apocalypseNone of this means AI is harmless. It is a tool, and like every tool, it will be used for evil. That alone is no great reason to fear it. That we’re getting better at doing things means the bad guys are also getting better at doing things, but that’s no reason to fear progress.The financial side of AI – trillions in capital chasing margins that open-weight models are eroding – is a bigger worry.But the truly apocalyptic impact of AI is on the world’s political sanity.In April, I wrote that we are barrelling into an epistemic crisis, in which we will no longer be able to tell what is real. A video published last week by the journalist Christophe shows what that looks like in practice.The Institute for Strategic Dialogue found 128 Facebook pages and groups, run from Sri Lanka, pumping AI-generated anti-migrant content at British audiences. Many videos depicted different people, in different locations, with different accents, shouting exactly the same scripted lines into the camera. None of them were real.These networks are not trying to swing elections. A Pakistani tutor told Christophe that US and UK audiences earn creators between $6 and $30 per thousand views, compared with 40 cents for a Pakistani audience. These hustlers don’t care about Western immigration policy. They have simply A/B-tested their way to whatever makes rich-world viewers angriest.“Immigrants out” gets more clicks than “Welcome immigrants”. It’s really that simple, and yet millions of people forward this AI slop to everyone they know, in search of the next dopamine hit of self-righteous outrage.Add the geostrategic disinformation agencies of Russia and China, who actually do want to destabilise Western democracies by radicalising the people at their political poles, and you get the mad polarisation we see around us.The centre cannot holdIntelligent people on the far left and the far right furiously repeat hateful talking points that were generated by AI operators who couldn’t care less whether they are true.Right-wingers shout about uncivilised immigrants and trans people. Left-wingers shout about greedy landlords and police brutality. Each side is targeted not only by trained Russian and Chinese content producers, but by entrepreneurs in India and Indonesia who know an easy mark when they see one..Read more:. Google’s AI overhaul sparks misinformation chaos: Parmy Olson.South Africans have seen this story before. In 2016, the Guptas paid the London PR firm Bell Pottinger £100,000 a month to stoke racial resentment over “white monopoly capital”, using fake bloggers and paid trolls. Ten years later, we’re still dealing with the political consequences of the anger that it unleashed.Today, a freelancer in Colombo with a chatbot can do the same job for free in half an hour, and get paid per thousand views by Meta.The real danger of AI is not that it will decide to kill us. It is the death of truth, and with it, the death of trust.The true apocalypse will come when the centre of normal, human politics no longer holds..*Ivo Vegter is a freelance journalist, columnist and speaker who loves debunking myths and misconceptions, and addresses topics from the perspective of individual liberty and free markets..This article was first published by The Daily Friend and is republished with permission.Sign up for your early morning brew of the BizNews Insider to keep you up to speed with the content that matters. The newsletter will land in your inbox every morning on weekdays. Register here.Support South Africa's bastion of independent journalism, offering balanced insights on investments, business, and the political economy, by joining BizNews Premium. Register here.If you prefer WhatsApp for updates, sign up to the BizNews channel here.

Ivo Vegter: Paid to create panic: Who's funding the AI apocalypse?

Related Stories

Ivo Vegter : Trump’s America - an empire consumed by ignorance, avarice and division

OpenAI 'solves' a Millennium Problem — and 24 of world’s best Maths brains cry foul

Empathy isn't a "bug" in Western civilisation, it's the foundation: Ivo Vegter

Ivo Vegter: What the DA's boring manifesto could teach America's Democrats