‘This is coming for everyone': A new kind of AI bot takes over the web
People are replacing Google search with artificial intelligence tools like ChatGPT, a major shift that has unleashed a new kind of bot loose on the web.
To offer users a tidy AI summary instead of Google's '10 blue links,' companies such as OpenAI and Anthropic have started sending out bots to retrieve and recap content in real time. They are scraping webpages and loading relevant content into the AI's memory and 'reading' far more content than a human ever would.
Subscribe to The Post Most newsletter for the most important and interesting stories from The Washington Post.
According to data shared exclusively with The Washington Post, traffic from retrieval bots grew 49 percent in the first quarter of 2025 from the fourth quarter of 2024. The data is from TollBit, a New York-based start-up that helps news publishers monitor and make money when AI companies use their content.
TollBit's report, based on data from 266 websites - half of which are run by national and local news organizations - suggests that the growth of bots that retrieve information when a user prompts an AI model is on an exponential curve.
'It starts with publishers, but this is coming for everyone,' Toshit Panigrahi, CEO and co-founder of TollBit, said in an interview.
Panigrahi said that this kind of bot traffic, which can be hard for websites to detect, reflects growing demand for content, even as AI tools devastate traffic to news sites and other online platforms. 'Human eyeballs to your site decreased. But the net amount of content access, we believe, fundamentally is going to explode,' he said.
A spokesperson for OpenAI said that referral traffic to publishers from ChatGPT searches may be lower in quantity but that it reflects a stronger user intent compared with casual web browsing.
To capitalize on this shift, websites will need to reorient themselves to AI visitors rather than human ones, Panigrahi said. But he also acknowledged that squeezing payment for content when AI companies argue that scraping online data is fair use will be an uphill climb, especially as leading players make their newest AI visitors even harder to identify.
Debate around the AI industry's use of online content has centered on the gargantuan amounts of text needed to train the AI models that power tools like ChatGPT. To obtain that data, tech companies use bots that scrape the open web for free, which has led to a raft of lawsuits alleging copyright theft from book authors and media companies, including a New York Times lawsuit against OpenAI. Other news publishers have opted for licensing deals. (In April, The Washington Post inked a deal with OpenAI.)
In the past eight months, as chatbots have evolved to incorporate features like web search and 'reasoning' to answer more complex queries, traffic for retrieval bots has skyrocketed. It grew 2.5 times as fast as traffic for bots that scrape data for training between the fourth quarter of 2024 and the first quarter of 2025, according to TollBit's report.
Panigrahi said TollBit's data may underestimate the magnitude of this change because it doesn't reflect bots that AI companies send out on behalf of AI 'agents' that can complete tasks on a user's behalf, like ordering takeout from DoorDash.
The start-up's findings also add a new dimension to mounting evidence that the modern internet - optimized for Google search results and social media algorithms - will have to be restructured as the popularity of AI answers grows.
'To think of it as, 'Well, I'm optimizing my search for humans' is missing out on a big opportunity,' he said.
Installing TollBit's analytics platform is free for news publishers, and the company has more than 2,000 clients, many of which are struggling with these seismic changes, according to data in the report.
Although news publishers and other websites can implement blockers to prevent various AI bots from scraping their content, TollBit found that more than 26 million AI scrapes bypassed those blockers in March alone. Some AI companies claim bots for AI agents don't need to follow bot instructions because they are acting on behalf of a user.
Mark Howard, chief operating officer for the media company Time, a TollBit client, said the start-up's traffic data has helped Time negotiate content licensing deals with AI companies including OpenAI and the search engine Perplexity.
But the market to fairly compensate publishers is far from established, Howard said. 'The vast majority of the AI bots out there absolutely are not sourcing the content through any kind of paid mechanism. … There is a very, very long way to go.'
Related Content
He's dying. She's pregnant. His one last wish is to fight his cancer long enough to see his baby.
The U.S. granted these journalists asylum. Then it fired them.
'Enough is enough.' Why Los Angeles is still protesting, despite fear.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
20 minutes ago
- Yahoo
Charter upgraded to Peer Perform from Underperform at Wolfe Research
Wolfe Research upgraded Charter (CHTR) to Peer Perform from Underperform without a price target President Trump's 'Big Beautiful Bill' seems likely to reinstate 100% bonus depreciation, the analyst tells investors in a research note. Wolfe estimates the five major connectivity providers could cumulatively gain $10.5B of cash tax relief in 2025 alone . Easily unpack a company's performance with TipRanks' new KPI Data for smart investment decisions Receive undervalued, market resilient stocks right to your inbox with TipRanks' Smart Value Newsletter Published first on TheFly – the ultimate source for real-time, market-moving breaking financial news. Try Now>> See today's best-performing stocks on TipRanks >> Read More on CHTR: Disclaimer & DisclosureReport an Issue Charter price target raised to $425 from $400 at UBS Nucor, Charter, Senseonics, Shake Shack, Altimmune: Trending by Analysts Boeing upgraded, PayPal initiated: Wall Street's top analyst calls Charter downgraded to Market Perform from Outperform at Bernstein Charter Communications: Hold Rating Amidst Growth and Competitive Challenges Sign in to access your portfolio
Gizmodo
22 minutes ago
- Gizmodo
The GoPro Hero 13 Black Is the Best 5.3K60 Camera, and Now Hits an All-Time Low on Amazon
GoPro is the leading brand for action cameras and the Hero 13 Black is the latest to carry that flag (released in early 2025). This camera is the ultimate in what an action cam is capable of in a tiny rugged package for adventurers and content creators who want the newest features. From extreme sports to travel vlogs to family moments, this camera is built to keep pace with your most ambitious endeavors. You can currently find the GoPro Hero 13 Black on Amazon for $329, discounted from its initial retail price of $429. This all-time low price is a substantial discount and makes the Hero 13 Black a fantastic deal if you're looking to invest in top-tier action camera technology. See at Amazon Technically, the GoPro Hero 13 Black is a monster: It records ultra-high-definition video in 5.3K resolution at 60 frames per second so your videos are smooth, crisp, and detailed. In terms of capturing photos, it records 27-megapixel stills so each photo is packed with color and detail. The camera is waterproof all the way down to 33 feet without an external housing, making it ideal for snorkeling, surfing, or any water adventure. You will also love the Hero 13 Black's compatibility with HB-series lenses: This lets you expand your creative options by attaching different lenses for special effects, whether you're shooting wide landscapes or close-up action shots. The camera also features high-end stabilization technology which keeps your footage steady even when you're moving fast or over rough terrain. The Hero 13 Black is packed with creator-specific modes and settings: It includes various shooting modes, including time-lapse, slow motion, and live streaming, to take your adventures and film them in new and different ways. With the intuitive touchscreen interface, modes are easy to switch and setting changes are simple on the fly. For individuals who want to share their videos in a haste, the camera supports fast wireless transfers to your smartphone or tablet, and it's Wi-Fi and Bluetooth capable. The Hero 13 Black is designed for long recording sessions and it supports fast charging so that you can get back to filming as quickly as possible. The camera is also voice-controlled, allowing you to begin and stop recording hands-free which is convenient when you are moving. Its compact size and lightweight design make it easy to mount on helmets, bikes, or even your wrist, allowing you the liberty to shoot from any angle. For content creators who want nothing but the best, the GoPro Hero 13 Black is a natural choice. Make sure you don't miss this deal. See at Amazon
Yahoo
25 minutes ago
- Yahoo
Aflac discloses cyber intrusion linked to wider crime spree targeting insurance industry
This story was originally published on Cybersecurity Dive. To receive daily news and insights, subscribe to our free daily Cybersecurity Dive newsletter. Major insurance provider Aflac Inc. said Friday that it was the target of a cyberattack on June 12 that is linked to a major cybercrime spree focusing on the industry. The company said it was able to contain the attack within hours and confirmed its systems remain operational. 'We continue to serve our customers as we respond to this incident and can underwrite policies, review claims and otherwise service our customers as usual,' the company said in a Securities and Exchange Commission filing. The incident is part of a larger crime wave targeting the insurance industry that researchers have linked to a collective known as Scattered Spider. The group recently conducted a weeks-long attack campaign against retailers in the U.S. and the U.K. Erie Insurance Group last week disclosed that it was the target of a cyberattack that began on June 7. The company said Tuesday that it has regained control over its systems and sees no further evidence of malicious activity. Erie is working with third-party forensic experts to restore full access to customers, agents and employees. Researchers from Google Threat Intelligence Group on Monday warned that the same hackers targeting the retail sector had pivoted toward the insurance industry. Google has not attributed the attacks to any actor but said they show the hallmarks of Scattered Spider, the notorious threat group linked to the 2023 MGM Resorts and Clorox hacks. "Given this actor's history of focusing on a sector at a time, the insurance industry should be on high alert, especially for social engineering schemes which target their help desks and call centers,' John Hultquist, chief analyst at GTIG, told Cybersecurity Dive in a statement. The retail sector intrusions began in April, with U.K. retailer Marks and Spencer and the Harrods department store chain among the major victims. In the U.S., the hacking spree hit Victoria's Secret and United Natural Foods, the largest supplier for Whole Foods, the grocery chain owned by Amazon. Aflac has begun a process of reviewing files that may have been accessed. The review is still in its early stages and Alfac said it cannot immediately determine how many people were affected. The files contain claims information, health records, Social Security numbers and other personal data related to customers, employees, beneficiaries, agents and other individuals. The company plans to notify regulators and will send breach letters to affected individuals and provide credit monitoring and identity-theft services. (Adds comment from Google) Sign in to access your portfolio
