‘This is coming for everyone': A new kind of AI bot takes over the web
People are replacing Google search with artificial intelligence tools like ChatGPT, a major shift that has unleashed a new kind of bot loose on the web.
To offer users a tidy AI summary instead of Google's '10 blue links,' companies such as OpenAI and Anthropic have started sending out bots to retrieve and recap content in real time. They are scraping webpages and loading relevant content into the AI's memory and 'reading' far more content than a human ever would.
Subscribe to The Post Most newsletter for the most important and interesting stories from The Washington Post.
According to data shared exclusively with The Washington Post, traffic from retrieval bots grew 49 percent in the first quarter of 2025 from the fourth quarter of 2024. The data is from TollBit, a New York-based start-up that helps news publishers monitor and make money when AI companies use their content.
TollBit's report, based on data from 266 websites - half of which are run by national and local news organizations - suggests that the growth of bots that retrieve information when a user prompts an AI model is on an exponential curve.
'It starts with publishers, but this is coming for everyone,' Toshit Panigrahi, CEO and co-founder of TollBit, said in an interview.
Panigrahi said that this kind of bot traffic, which can be hard for websites to detect, reflects growing demand for content, even as AI tools devastate traffic to news sites and other online platforms. 'Human eyeballs to your site decreased. But the net amount of content access, we believe, fundamentally is going to explode,' he said.
A spokesperson for OpenAI said that referral traffic to publishers from ChatGPT searches may be lower in quantity but that it reflects a stronger user intent compared with casual web browsing.
To capitalize on this shift, websites will need to reorient themselves to AI visitors rather than human ones, Panigrahi said. But he also acknowledged that squeezing payment for content when AI companies argue that scraping online data is fair use will be an uphill climb, especially as leading players make their newest AI visitors even harder to identify.
Debate around the AI industry's use of online content has centered on the gargantuan amounts of text needed to train the AI models that power tools like ChatGPT. To obtain that data, tech companies use bots that scrape the open web for free, which has led to a raft of lawsuits alleging copyright theft from book authors and media companies, including a New York Times lawsuit against OpenAI. Other news publishers have opted for licensing deals. (In April, The Washington Post inked a deal with OpenAI.)
In the past eight months, as chatbots have evolved to incorporate features like web search and 'reasoning' to answer more complex queries, traffic for retrieval bots has skyrocketed. It grew 2.5 times as fast as traffic for bots that scrape data for training between the fourth quarter of 2024 and the first quarter of 2025, according to TollBit's report.
Panigrahi said TollBit's data may underestimate the magnitude of this change because it doesn't reflect bots that AI companies send out on behalf of AI 'agents' that can complete tasks on a user's behalf, like ordering takeout from DoorDash.
The start-up's findings also add a new dimension to mounting evidence that the modern internet - optimized for Google search results and social media algorithms - will have to be restructured as the popularity of AI answers grows.
'To think of it as, 'Well, I'm optimizing my search for humans' is missing out on a big opportunity,' he said.
Installing TollBit's analytics platform is free for news publishers, and the company has more than 2,000 clients, many of which are struggling with these seismic changes, according to data in the report.
Although news publishers and other websites can implement blockers to prevent various AI bots from scraping their content, TollBit found that more than 26 million AI scrapes bypassed those blockers in March alone. Some AI companies claim bots for AI agents don't need to follow bot instructions because they are acting on behalf of a user.
Mark Howard, chief operating officer for the media company Time, a TollBit client, said the start-up's traffic data has helped Time negotiate content licensing deals with AI companies including OpenAI and the search engine Perplexity.
But the market to fairly compensate publishers is far from established, Howard said. 'The vast majority of the AI bots out there absolutely are not sourcing the content through any kind of paid mechanism. … There is a very, very long way to go.'
Related Content
Field notes from the end of life: My thoughts on living while dying
He's dying. She's pregnant. His one last wish is to fight his cancer long enough to see his baby.
The U.S. granted these journalists asylum. Then it fired them.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
12 minutes ago
- Yahoo
Vancouver company celebrating reusable food container success
started with just five stores but is now in institutions across Canada and the U.S. As Chad Pawson explains, local waste management officials say it's the type of private-sector ingenuity needed to address the region's garbage problem.
Yahoo
18 minutes ago
- Yahoo
Why Mondelez's Dividend Stands Strong in Uncertain Markets
Mondelez International, Inc. (NASDAQ:MDLZ) is one of the best dividend stocks for a bear market. Mondelez International, Inc. (NASDAQ:MDLZ) ranks among the top global snack makers and belongs to the consumer staples sector. During economic downturns, consumers typically continue buying food at consistent levels, and even tend to favor discounted options more, making large food companies like Mondelez appealing, especially when spending declines in more discretionary areas. A colourful array of products like candies, chocolates and gums on a supermarket shelf. Another reason investors are drawn to Mondelez International, Inc. (NASDAQ:MDLZ) is its strong dividend and reliable cash flow. In the latest quarter, the company generated $1.1 billion in operating cash flow and $0.8 billion in free cash flow, while returning $2.1 billion to shareholders through dividends and buybacks. Mondelez International, Inc. (NASDAQ:MDLZ) has also garnered attention on the dividend front. The company has been rewarding shareholders with growing dividends for the past 11 consecutive years. It offers a quarterly dividend of $0.47 per share and has a dividend yield of 2.83% The company has expressed confidence in its future earnings potential, which may support the strength and stability of its dividend payments. It has reaffirmed its outlook for the year, expecting organic net revenue to grow by around 5%. However, adjusted earnings per share are projected to decline by roughly 10% on a constant currency basis, mainly due to an unusual surge in cocoa prices. Despite this, the company anticipates generating over $3 billion in free cash flow for 2025. Since recessions often go hand in hand with falling stock markets, investors with shorter time horizons may find value in holding companies that offer steady or growing dividend yields. While we acknowledge the potential of MDLZ as an investment, we believe certain AI stocks offer greater upside potential and carry less downside risk. If you're looking for an extremely undervalued AI stock that also stands to benefit significantly from Trump-era tariffs and the onshoring trend, see our free report on the best short-term AI stock. READ NEXT: and Disclosure. None. Error while retrieving data Sign in to access your portfolio Error while retrieving data Error while retrieving data Error while retrieving data Error while retrieving data
Yahoo
18 minutes ago
- Yahoo
This Under-the-Radar Healthcare Stock Could Be a Solid Income Play
CVS Health Corporation (NYSE:CVS) is one of the best dividend stocks for a bear market. Even during economic downturns, people continue to rely on medications, essential consumer products, and affordable local healthcare. CVS Health Corporation (NYSE:CVS) serves as a convenient healthcare and retail destination within communities. A row of shelves in a retail pharmacy, demonstrating the variety of drugs and over-the-counter products. The company's overall business remains solid, thanks to its diversified operations and multiple sources of revenue. In recent years, it has expanded its presence in primary care and launched a subsidiary called Cordavis to focus on developing and marketing biosimilar drugs. Its broad reach across communities and wide range of services are key advantages. Lately, higher Medicare usage and increased post-pandemic healthcare costs have impacted the company's revenue and earnings growth. However, CVS Health Corporation (NYSE:CVS) remains profitable and maintains a solid cash position. In the most recent quarter, it reported $4.6 billion in operating cash flow. Looking ahead to 2025, the company has raised its full-year operating cash flow forecast from around $6.5 billion to approximately $7.0 billion. In addition, CVS Health Corporation (NYSE:CVS) appears to have significant room to grow its dividend. With a cash payout ratio of just 30%, even doubling that figure would still leave it within a sustainable range. Due to this strong cash generation, CVS Health Corporation (NYSE:CVS) has maintained its payouts since 1997. Currently, it offers a quarterly dividend of $0.665 per share and has a dividend yield of 3.96%, as of June 17. While we acknowledge the potential of CVS as an investment, we believe certain AI stocks offer greater upside potential and carry less downside risk. If you're looking for an extremely undervalued AI stock that also stands to benefit significantly from Trump-era tariffs and the onshoring trend, see our free report on the best short-term AI stock. READ NEXT: and Disclosure. None.
