The Economist: AI is killing the web. Can anything resurrect it?

Key topics:AI chatbots are diverting traffic from traditional websites and search enginesContent creators are losing ad revenue and suing AI firms for fair compensationNew models like bot paywalls and revenue sharing are emerging to save the web.Sign up for your early morning brew of the BizNews Insider to keep you up to speed with the content that matters. The newsletter will land in your inbox at 5:30am weekdays. Register here.Support South Africa’s bastion of independent journalism, offering balanced insights on investments, business, and the political economy, by joining BizNews Premium. Register here.If you prefer WhatsApp for updates, sign up to the BizNews channel here.The auditorium doors will open for BNIC#2 on 10 September 2025 in Hermanus. For more information and tickets, click here..From The Economist, published under licence. The original article can be found on www.economist.com© 2025 The Economist Newspaper Limited. All rights reserved..The Economist.For companies that sell advertising or subscriptions, lost visitors means lost revenue. “We had a very positive relationship with Google for a long time…They broke the deal,” says Neil Vogel, head of Dotdash Meredith, which owns titles such as People and Food & Wine. Three years ago its sites got more than 60% of their traffic from Google. Now the figure is in the mid-30s. “They are stealing our content to compete with us,” says Mr Vogel. Google has insisted that its use of others’ content is fair. But since it launched its AI overviews, the share of news-related searches resulting in no onward clicks has risen from 56% to 69%, estimates Similarweb. In other words, seven in ten people get their answer without visiting the page that supplied it.“The nature of the internet has completely changed,” says Prashanth Chandrasekar, chief executive of Stack Overflow, best known as an online forum for coders. “AI is basically choking off traffic to most content sites,” he says. With fewer visitors, Stack Overflow is seeing fewer questions posted on its message boards. Wikipedia, also powered by enthusiasts, warns that AI-generated summaries without attribution “block pathways for people to access…and contribute to” the site.To keep the traffic and the money coming, many big content producers have negotiated licensing deals with AI companies, backed up by legal threats: what Robert Thomson, chief executive of News Corp, has dubbed “wooing and suing”. His company, which owns the Wall Street Journal and the New York Post, among other titles, has struck a deal with OpenAI. Two of its subsidiaries are suing Perplexity, another AI answer engine. The New York Times has done a deal with Amazon while suing OpenAI. Plenty of other transactions and lawsuits are going on. (The Economist’s parent company has not taken a public position on whether it will license our work.)Yet this approach has limits. For one thing, judges so far seem minded to side with AI companies: last month two separate copyright cases in California went in favour of their defendants, Meta and Anthropic, both of which argued that training their models on others’ content amounted to fair use. President Donald Trump seems to accept Silicon Valley’s argument that it must be allowed to get on with developing the technology of the future before China can. He has appointed tech boosters as advisers on AI, and sacked the head of the US Copyright Office soon after she argued that training AI on copyrighted material was not always legal..AI companies are more willing to pay for continuing access to information than training data. But the deals done so far are hardly stellar. Reddit, an online forum, has licensed its user-generated content to Google for a reported $60m a year. Yet its market value fell by more than half—over $20bn—after it reported slower user-growth than expected in February, owing to wobbles in search traffic. (Growth has since picked up and Reddit’s share price has recovered some lost ground.).Read more:. Legal storm brews as AI chatbots face copyright challenges.The bigger problem, however, is that most of the internet’s hundreds of millions of domains are too small to either woo or sue the tech giants. Their content may be collectively essential to AI firms, but each site is individually dispensable. Even if they could join forces to bargain collectively, antitrust law would forbid it. They could block AI crawlers, and some do. But that means no search visibility at all.Software providers may be able to help. All of Cloudflare’s new customers will now be asked if they want to allow AI companies’ bots to scrape their site, and for what purpose. Cloudflare’s scale gives it a better chance than most of enabling something like a collective response by content sites that want to force AI firms to cough up. It is testing a pay-as-you-crawl system that would let sites charge bots an entry fee. “We have to set the rules of the road,” says Mr Prince, who says his preferred outcome is “a world where humans get content for free, and bots pay a tonne for it”.An alternative is offered by Tollbit, which bills itself as a paywall for bots. It allows content sites to charge AI crawlers varying rates: for instance, a magazine could charge more for new stories than old ones. In the first quarter of this year Tollbit processed 15m micro-transactions of this sort, for 2,000 content producers including the Associated Press and Newsweek. Toshit Panigrahi, its chief executive, points out that whereas traditional search engines incentivise samey content—“What time does the Super Bowl start?”, for example—charging for access incentivises uniqueness. One of Tollbit’s highest per-crawl rates is charged by a local newspaper.Another model is being put forward by ProRata, a startup led by Bill Gross, a pioneer in the 1990s of the pay-as-you-click online ads that have powered much of the web ever since. He proposes that money from ads placed alongside AI-generated answers should be redistributed to sites in proportion to how much their content contributed to the answer. ProRata has its own answer engine, Gist.ai, which shares ad revenue with its 500-plus partners, which include the Financial Times and the Atlantic. It is currently more of an exemplar than a serious threat to Google: Mr Gross says his main aim is to “show a fair business model that other people eventually copy”.Meanwhile, content producers are rethinking their business models. “The future of the internet is not all about traffic,” says Mr Chandrasekar, who has built up Stack Overflow’s private, enterprise-oriented subscription product, Stack Internal. News publishers are planning for “Google zero”, deploying newsletters and apps to reach customers who no longer come to them via search, and moving their content behind paywalls or to live events. Audio and video are proving legally and technically harder for AI engines to summarise than text. The site to which answer engines refer search traffic most often, by far, is YouTube, according to Similarweb.Not everyone thinks the web is in decline—on the contrary, it is in “an incredibly expansionary moment”, argues Robby Stein of Google. As AI makes it easier to create content, the number of sites is growing: Google’s bots report that the web has expanded by 45% in the past two years. AI search lets people ask questions in new ways—for instance, taking a photo of their bookshelf and asking for recommendations on what to read next—which could increase traffic. With AI queries, more sites than ever are being “read”, even if not with human eyes. An answer engine may scan hundreds of pages to deliver an answer, drawing on a more diverse range of sources than human readers would.As for the idea that Google is disseminating less human traffic than before, Mr Stein says the company has not noticed a dramatic decline in the number of outbound clicks, though it declines to make the number public. There are other reasons besides AI why people may be visiting sites less. Maybe they are scrolling social media. Maybe they are listening to podcasts.The death of the web has been predicted before—at the hands of social networks, then smartphone apps—and not come to pass. But AI may pose the biggest threat to it yet. If the web is to continue in something close to its current form, sites will have to find new ways to get paid for content. “There’s no question that people prefer AI search,” says Mr Gross. “And to make the internet survive, to make democracy survive, to make content creators survive, AI search has to share revenue with creators.”

The Economist: AI is killing the web. Can anything resurrect it?

Key topics:

From The Economist, published under licence. The original article can be found on www.economist.com

© 2025 The Economist Newspaper Limited. All rights reserved.

Related Stories