‘This is coming for everyone': A new kind of AI bot takes over the web
People are replacing Google search with artificial intelligence tools like ChatGPT, a major shift that has unleashed a new kind of bot loose on the web.
To offer users a tidy AI summary instead of Google's '10 blue links,' companies such as OpenAI and Anthropic have started sending out bots to retrieve and recap content in real time. They are scraping webpages and loading relevant content into the AI's memory and 'reading' far more content than a human ever would.
Subscribe to The Post Most newsletter for the most important and interesting stories from The Washington Post.
According to data shared exclusively with The Washington Post, traffic from retrieval bots grew 49 percent in the first quarter of 2025 from the fourth quarter of 2024. The data is from TollBit, a New York-based start-up that helps news publishers monitor and make money when AI companies use their content.
TollBit's report, based on data from 266 websites - half of which are run by national and local news organizations - suggests that the growth of bots that retrieve information when a user prompts an AI model is on an exponential curve.
'It starts with publishers, but this is coming for everyone,' Toshit Panigrahi, CEO and co-founder of TollBit, said in an interview.
Panigrahi said that this kind of bot traffic, which can be hard for websites to detect, reflects growing demand for content, even as AI tools devastate traffic to news sites and other online platforms. 'Human eyeballs to your site decreased. But the net amount of content access, we believe, fundamentally is going to explode,' he said.
A spokesperson for OpenAI said that referral traffic to publishers from ChatGPT searches may be lower in quantity but that it reflects a stronger user intent compared with casual web browsing.
To capitalize on this shift, websites will need to reorient themselves to AI visitors rather than human ones, Panigrahi said. But he also acknowledged that squeezing payment for content when AI companies argue that scraping online data is fair use will be an uphill climb, especially as leading players make their newest AI visitors even harder to identify.
Debate around the AI industry's use of online content has centered on the gargantuan amounts of text needed to train the AI models that power tools like ChatGPT. To obtain that data, tech companies use bots that scrape the open web for free, which has led to a raft of lawsuits alleging copyright theft from book authors and media companies, including a New York Times lawsuit against OpenAI. Other news publishers have opted for licensing deals. (In April, The Washington Post inked a deal with OpenAI.)
In the past eight months, as chatbots have evolved to incorporate features like web search and 'reasoning' to answer more complex queries, traffic for retrieval bots has skyrocketed. It grew 2.5 times as fast as traffic for bots that scrape data for training between the fourth quarter of 2024 and the first quarter of 2025, according to TollBit's report.
Panigrahi said TollBit's data may underestimate the magnitude of this change because it doesn't reflect bots that AI companies send out on behalf of AI 'agents' that can complete tasks on a user's behalf, like ordering takeout from DoorDash.
The start-up's findings also add a new dimension to mounting evidence that the modern internet - optimized for Google search results and social media algorithms - will have to be restructured as the popularity of AI answers grows.
'To think of it as, 'Well, I'm optimizing my search for humans' is missing out on a big opportunity,' he said.
Installing TollBit's analytics platform is free for news publishers, and the company has more than 2,000 clients, many of which are struggling with these seismic changes, according to data in the report.
Although news publishers and other websites can implement blockers to prevent various AI bots from scraping their content, TollBit found that more than 26 million AI scrapes bypassed those blockers in March alone. Some AI companies claim bots for AI agents don't need to follow bot instructions because they are acting on behalf of a user.
Mark Howard, chief operating officer for the media company Time, a TollBit client, said the start-up's traffic data has helped Time negotiate content licensing deals with AI companies including OpenAI and the search engine Perplexity.
But the market to fairly compensate publishers is far from established, Howard said. 'The vast majority of the AI bots out there absolutely are not sourcing the content through any kind of paid mechanism. … There is a very, very long way to go.'
Related Content
Field notes from the end of life: My thoughts on living while dying
He's dying. She's pregnant. His one last wish is to fight his cancer long enough to see his baby.
The U.S. granted these journalists asylum. Then it fired them.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
42 minutes ago
- Yahoo
Intel to outsource marketing to Accenture and AI, resulting in more layoffs
When you buy through links on our articles, Future and its syndication partners may earn a commission. Employees at Intel's marketing division were informed that many of their roles will be handed over to Accenture, which will use AI to handle tasks traditionally done by Intel staff, reports OregonLive. The decision is part of a company-wide restructuring plan that includes job cuts, automation, and streamlining of execution. The marketing division has been one of Intel's key strengths since the company began communicating directly with end users with the launch of its "Intel Inside" campaign in 1991. However, it looks like the company will drastically cut its human-driven marketing efforts going forward, as it plans to lay off many of its marketing employees, believing that Accenture's AI will do a better job connecting Intel with customers. The number of positions affected was not disclosed, but Intel confirmed changes will significantly alter team structures, with only 'lean' teams remaining. Workers will be told by July 11 whether they will remain with the company. Among other things, the aim of the restructuring is to free up internal teams to focus on strategic, creative, and high-value projects, rather than routine functions. Therefore, Intel intends to use Accenture's AI in various aspects of marketing, including information processing, task automation, and personalized communications. Intel has acknowledged the shift to Accenture and explained that this will not only cut costs but will modernize its capabilities and strengthen its brand. How exactly the usage of AI instead of real people can reinforce the brand hasn't been explained yet. "As we announced earlier this year, we are taking steps to become a leaner, faster and more efficient company," a statement by Intel published by OregonLive reads. "As part of this, we are focused on modernizing our digital capabilities to serve our customers better and strengthen our brand. Accenture is a longtime partner and trusted leader in these areas and we look forward to expanding our work together." In messages to staff published by OregonLive, Intel indicated that part of the restructuring may involve existing employees training Accenture contractors by explaining how Intel's operations work. This knowledge transfer would occur during the transitional phase of the outsourcing plan, although it is unclear how long this phase will take. Follow Tom's Hardware on Google News to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button.
Bloomberg
an hour ago
- Bloomberg
Malaysia Says US Agrees to Finalize Trade Talks Before Deadline
Malaysia and the US have agreed to finalize trade negotiations before the tariff reprieve expires, the Malaysia's Ministry of Investment, Trade and Industry said. 'Both Malaysia and the US representatives expressed their intent to finalize the negotiations before the expiry of the 90-day pause on tariff implementation and agreed to intensify efforts to reach an agreement that would be mutually acceptable by key stakeholders from both countries,' according to a statement from the ministry on Sunday.
Yahoo
an hour ago
- Yahoo
2 world-class growth shares to consider buying during a stock market crash
It's always a smart idea to have a shopping list of shares at hand. This is what I had in April when the proverbial hit the fan, saving me valuable time deciding which stocks to buy when many suddenly went on sale. Two on my list were Shopify and Nvidia, and I took my chance when things quickly went south. Those positions are currently up 40% and 51% respectively since early April. Here are two shares that look overvalued, but which I think investors should consider putting on a shopping list for the next bear market or crash. The first is Axon Enterprise (NASDAQ: AXON). As well as owning the Taser brand, the company sells body-cams, dash-cams, and various software products to law enforcement customers. It's also innovating in drones, VR training, and powerful artificial intelligence (AI)-powered tools. Last year, revenue jumped 33% to $2.1bn, up from $681m in 2020. Over $1bn of that is now annual recurring revenue, while total future contracted bookings rose to $10.1bn. The company continues to scale up very impressively. This rapid progress is reflected in the share price, which has vaulted 700% in three years. The key risk now then is valuation, with the shares trading at 27 times sales. If Axon's growth underwhelms, the stock could sell off heavily. But I think this is definitely one to consider picking up during any such sell-off. The enterprise growth opportunity's very large. Take body-cams, for example. In an attempt to reduce attacks on staff, Walmart's piloting black-and-yellow body-cams in some stores (Axon's signature colours). Management points out that Walmart has 2.1m retail workers, far in excess of the 900,000 cops in the US. What about airlines? Cabin crew are no strangers to abusive behaviour. Cameras can provide evidence, reduce false claims, and deter aggression. They would integrate with Axon's cloud-based evidence system, adding to the recurring revenue. Meanwhile, the company's sitting on a library of video data that's roughly 40 times larger than that belonging to Netflix (NASDAQ: NFLX). Axon's using this vast data trove to train AI models and power a growing suite of tools, embedding it ever deeper into customers' daily workflows. Looking ahead, the international opportunity in Europe, Latin America and Asia remains largely untapped. Returning to Netflix, I think the global streaming leader is worthy of consideration. But perhaps not right now around $1,220. This price puts the stock at 13 times sales and 49 times forward earnings. Again, this stretched valuation leaves very little margin for error, particularly if subscriber growth unexpectantly slows. Long term though, I'm bullish on Netflix. The firm possesses an incredible brand and streaming still has plenty of room to grow globally. It also offers fantastic value for money. At £12.99, my Netflix subscription's probably one of the last discretionary things I would cut. The new ad-supported tier is just £5.99 a month — less than the price of a pint in London nowadays! I think Netflix can keep increasing prices for years without losing too many subscribers. The firm has set a goal of reaching a $1trn market-cap by 2030, up from $520bn today. I think that's achievable, especially if cutting-edge generative AI helps it make content for less money. The post 2 world-class growth shares to consider buying during a stock market crash appeared first on The Motley Fool UK. More reading 5 Stocks For Trying To Build Wealth After 50 One Top Growth Stock from the Motley Fool Ben McPoland has positions in Axon Enterprise, Nvidia, and Shopify. The Motley Fool UK has recommended Axon Enterprise, Nvidia, Shopify, and Walmart. Views expressed on the companies mentioned in this article are those of the writer and therefore may differ from the official recommendations we make in our subscription services such as Share Advisor, Hidden Winners and Pro. Here at The Motley Fool we believe that considering a diverse range of insights makes us better investors. Motley Fool UK 2025 Error while retrieving data Sign in to access your portfolio Error while retrieving data Error while retrieving data Error while retrieving data Error while retrieving data
