Cloudflare dropped one of the more revealing AI-web reports of the year on July 13, 2026, and the headline should bother anyone who still thinks this is just a temporary product transition.
According to Cloudflare's latest Content Independence Day report, 52% of crawler requests are now for AI training, up from 22% in spring 2025. Mixed-use crawlers account for more than 36% of activity. Pure search crawling is the shrinking minority.
That is not a small classification change. That is the shape of the new internet economy showing up in public data.
I wrote earlier this month in Cloudflare Just Told AI Crawlers to Pick a Job that bundling search, training, and agent behavior into one crawler identity was always a loophole. This new report explains why Cloudflare bothered splitting those categories in the first place. The problem is not just bot labeling. The problem is that the old web bargain is getting strip-mined.
If the machine reads the page, extracts the value, and never sends the person back, that is not discovery. That is unpaid supply.
For years, websites made a rough deal with search engines: crawl the page, index the page, and in return send people back. The bargain was never perfect, but it was legible. Publishers got traffic. Search engines got content. Users got links.
That is not the deal anymore.
This is not a search economy anymore
Cloudflare's July 13 report says people now spend only about 15 minutes of each online hour on the open web when searching for information. The rest is being absorbed into more closed, summarized, AI-mediated surfaces. It also says some heavily crawled categories have seen human traffic decline by as much as 40% in less than a year.
Read that slowly. The human visit is no longer the center of gravity. The answer layer is.
That means a growing share of the web is being used as raw material for outputs that complete themselves somewhere else. The user never reaches the original page. The publisher still pays hosting costs, research costs, editorial costs, and maintenance costs. The platform captures the session. The model captures the learning. The website captures the bill.
This is why the cheerful AI-search framing has started to annoy me. We are not just talking about a nicer interface for finding information. We are talking about a system that increasingly consumes information without preserving the economic loop that produced it.
That is also why phrases like agentic internet can hide more than they reveal. Agentic sounds innovative and productive. In practice, a lot of that activity is just software doing more extraction at higher speed.
Training won because the referral loop lost
The most important thing in the Cloudflare data is not the raw bot percentage by itself. It is what that percentage implies about incentives.
If most crawling is now about training rather than search visibility, then the dominant machine relationship to the web is no longer "find this so a human can visit." It is "take this so the model gets better."
That is a massive shift in incentives.
Once training becomes the main purpose, every high-quality page starts looking less like a destination and more like a free upstream input. If you are an AI company, that is great. The open web becomes a distributed R&D subsidy. If you are a publisher, retailer, software company, forum owner, or anyone else producing original information, it starts looking like you are funding the improvement of systems that reduce your own traffic.
Cloudflare's July 1 search initiative makes this even clearer. The company says more than half of good-bot crawl traffic appears to be re-fetching pages that have not changed. That means site owners are not just losing traffic in the downstream sense. They are also paying upstream costs for a lot of bot activity that produces nothing new.
So the web gets hit twice.
- First, sites pay to serve the crawlers.
- Second, they lose the human traffic that used to justify being crawled in the first place.
- Third, the extracted value helps improve systems that may send back even less traffic later.
That is what makes this a free-rider economy. The extraction keeps scaling faster than the compensation or reciprocity.
Publishers are being asked to fund their own replacement
Cloudflare crossed another line in the same report when it said more than 50% of internet traffic is now non-human. I do not think that should be treated like a trivia stat. It is the kind of threshold that changes the assumptions underneath the whole stack.
The web was built around the idea that attention came from people. Ads assumed people. Analytics assumed people. funnels assumed people. Conversion paths assumed people. Even crawling, historically, was justified by pointing back to people.
Now a bigger share of the activity is software talking to software, and a lot of the human value is being intercepted before the visit happens.
That creates a brutal asymmetry. The publisher still has to act like a publisher. They still have to make the page worth reading. But the machine interacting with that page does not have to preserve the publisher's economics. It just has to preserve enough utility for the answer layer to stay convincing.
I have written before in There Is Now One Web for People and Another for Agents that the internet is splitting into separate layers for humans and software. This week makes the economic side of that split harder to ignore. The agent-facing layer is not just different in interface. It is different in who captures value.
And if that keeps going, the open web becomes the part that does the expensive original work while the AI layer becomes the part that monetizes the convenience.
Taxonomy helps, but it does not solve the real problem
Cloudflare deserves credit for at least naming the categories honestly. On July 1, it rolled out separate controls for Search, Agent, and Training crawlers, and said that beginning September 15, 2026, new ad-supported domains on Cloudflare will block Training and Agent traffic by default while still allowing Search.
That is directionally right. Search, training, and live agent fetches are not morally or economically equivalent. Pretending they are equivalent was a convenience mostly for the companies running blended crawlers.
But separation is table stakes, not the finish line.
A cleaner label on the extractor does not magically fix the underlying deal. Even a well-behaved training crawler is still asking for something that has a real cost. Even a polite answer engine can still starve the source if it keeps the user in its own interface. Transparency matters, but transparency without compensation mostly just means the losing party gets better reporting.
That is why I think the bigger story is not the crawler taxonomy by itself. It is Cloudflare admitting, in product form, that the old default assumptions no longer work.
The next fight is payment, not politeness
Cloudflare's answer is to move from defense into market design. It is experimenting with ideas like Pay Per Crawl shifting toward Pay Per Use, plus newer reporting for answer-engine visibility. In theory, that is the right direction. If AI systems are going to use the web as a knowledge substrate, they should pay for the value they actually derive instead of pretending a bot visit is neutral.
But I am still skeptical about how quickly this gets real at scale.
The pressure to extract first and negotiate later is still huge. Every frontier model company wants better outputs. Every answer engine wants fresher content. Every agent platform wants broader grounding. Meanwhile, most content creators are fragmented, underpowered, and still dependent on platforms for discovery. That is not exactly the setup for a balanced market to appear on its own.
So yes, payment infrastructure matters. Better crawl signals matter. Smarter refresh logic matters. But none of that changes the core reality: the agentic internet is already behaving like a free-rider system, and the fix will require more than cleaner dashboards and nicer language.
It will require real scarcity, real permissions, and real compensation.
The old bargain is not coming back by accident
The web is not going to snap back to ten blue links and publisher-friendly traffic flows just because people complain loudly enough. That era is over.
The real choice now is what replaces it.
One path is a future where independent sites keep publishing, AI systems keep ingesting, and almost all of the economic upside concentrates in the answer layer. That path is very convenient for users in the short term and very corrosive for the web in the long term.
The other path is messier but healthier: separate the purposes, expose the economics, pay for use, reduce pointless crawl load, and stop pretending that "helping users" automatically justifies infinite unpaid extraction.
Cloudflare's July 13 report matters because it makes the direction harder to deny. Training is no longer the side activity. It is the main event. The open web is no longer just being indexed. It is being harvested.
If AI companies want that arrangement to remain socially and economically stable, they are going to have to do better than calling it innovation and hoping the supply stays free.
Because right now the agentic internet does not look like a balanced future. It looks like a free-rider economy with better branding.