Soon, We’ll All Be Able to Charge AI Bots for Scraping Data

In 2025, an online small business forum went offline. It had been operating for 15 years and generated just enough ad revenue to cover the hosting costs. Until it didn’t. The problem was bots. So many bots. And not the spamming kind, but the AI kind. LLMs were mining them for training data first and then actively pinging them all day every day to tap into live conversations taking place in the small business owner community. And so the forum eventually added Cloudflare as a protective measure to try and slow the velocity of the traffic. Enough AI bot traffic still got through, however, and the strain on the bandwidth pushed the hosting costs beyond the revenue.

It was a good deal for the LLMs because when people asked their local chatbot a question, it had oddly specific conversational exchanges between business owners to draw from. Eventually, the forum owner realized that their real customer wasn’t advertisers or even small business owners anymore, but LLMs that wanted to mine what small business owners talked about on a live basis. If only there had been a way for a little small forum owner to charge that customer.

Now, there will be. On July 1, 2026, Cloudflare announced its new Monetization Gateway that will give Cloudflare customers “the ability to charge for any asset protected by Cloudflare: web pages, datasets, APIs, or MCP tools.”

It works as such: When an AI bot pings a url it is met with an http 402 code. Under the 402 protocol, the bot is presented with a price in real time of how much it will cost to access that data. If accepted, the bot pays in stablecoins, and they are passed through to the endpoint. This would all take place in less than a second with Cloudflare managing all the payments and settlements on blockchain rails. By using stablecoins, micropayments become highly feasible. Cloudflare uses an an example of a $0.001 base fee for example, 1/10th of 1 cent for a single call.

“An agent does not look at ads or need to maintain a monthly subscription to all the tools it wants to access,” Cloudflare said. “It reads a page or consumes a data feed once, takes what it needs, and moves on. Across the web, AI crawlers already request content anywhere from a hundred to tens of thousands of times for every visitor they send back.”

“This reality demands a new model: usage-based pricing for everything. If attention and e-commerce are moving from websites to AI harnesses and AI-written software, then agents should pay for the inputs they need — training data, inference content, developer tooling, and API usage.

Historically, usage-based billing was difficult to implement. Businesses needed to effectively become payments companies, running their own accounting to track internal usage in a robust and auditable way. Tracking this usage required significant overhauls of backend systems. Many instead chose per-seat pricing because it is simpler and frequently more profitable.

Agents flip this dynamic. A single agent can do the work of an entire team around the clock, making a flat one-time fee disconnected from actual consumption. At the same time, an agent can make thousands of micropayments without friction, while asking a person to approve each payment would be impossibly burdensome. Usage-based price points are where agents live and where stablecoin-based micropayments shine. That’s because stablecoins (such as Open USD and USDC) allow buyers to transfer tiny sums across the Internet, incurring negligible fees and settling in less than a second. This is not feasible with other payment rails today.”

– Cloudflare

“There is an enormous amount of value moving across the Internet today that goes unmonetized or undermonetized, not because no one would pay for it, but because the tools to charge for it have never existed,” Cloudflare further said.

And so while that forum remains offline, AI bots still ping the urls daily hoping it has come back. If in the near future the companies behind those bots decide the data is worth paying for, then perhaps the forum will eventually have a path for a return, and many other web-based businesses are finally able to charge who their real customers are, AI bots.