Cloudflare launches marketplace to monetise AI web scraping
Cloudflare’s new marketplace allows websites to earn revenue by licensing access to AI bots scraping their content.
Cloudflare has announced the launch of a new marketplace that enables website operators to charge AI companies for access to their content, targeting large language models (LLMs) and other bots that scrape web data. The announcement, made on 1 July 2025, marks a major development in the ongoing debate over data ownership, content scraping, and compensation in the AI era.
The initiative comes amid growing concerns among publishers, artists, and platforms about AI companies using web data to train models without permission or compensation.
New standard for content licensing and payment
At the core of Cloudflare’s marketplace is a standardised system that allows websites to identify whether a bot is accessing content for AI training purposes. If the bot is registered through Cloudflare's new "Humanity" protocol, a label that AI bots are encouraged to adopt—publishers can choose to block, allow, or monetise the access.
Cloudflare said that it will act as an intermediary between websites and AI companies, negotiating terms, tracking bot behaviour, and processing payments. The company claims this could be the beginning of a more formal “permission and compensation” ecosystem between AI developers and the broader web.
This service builds on Cloudflare’s previously announced efforts to block known AI crawlers, such as those from OpenAI’s GPTBot or Google’s AI crawler. Now, instead of simply blocking access, websites can opt in to licensing agreements that let them earn revenue.
OpenAI, Google among early discussions
According to Cloudflare, some of the major AI companies including OpenAI, Google, and Anthropic are either in discussions or have already begun implementing the Humanity protocol. These bots will reportedly present a verified identity when crawling sites, making it easier to enforce access rules.
According to TechCrunch, Cloudflare CEO Matthew Prince said the company’s role is not to decide what AI companies can or cannot access, but to enable transparency and enforceable rules that give content creators and web publishers greater control.
“Web scraping for AI is not going away,” Prince said. “But it shouldn’t be a free-for-all either.”
This model attempts to draw a parallel to the way Google previously established licensing agreements with news publishers, but now applied at scale for AI training data.