Cloudflare will grade verified bots on whether they slow down after an HTTP 429

Cloudflare has started scoring the bots on its Verified list by how they behave, not only by who they are, and one of the first tests is simple: when a website answers with HTTP 429, "too many requests", does the bot slow down or keep going at the same pace? The company described the system in an October 16 blog post and said it plans to let every customer choose how much trust a bot needs before its traffic gets through.

Until now, a bot's place on Cloudflare's list rested largely on proving its identity. The new approach treats that as the starting point and keeps checking afterwards.

From identity to a running record

Cloudflare's Verified bot rules ask for three things: that a bot identifies itself accurately and securely, does only what it says it does, and avoids abusive behavior. The first is handled by cryptographic request signing, known as Web Bot Auth, and a public catalogue called BotBase. The new system is aimed at the third.

The company draws a line between risk and trust. Risk, in its framing, is the chance that a single request is harmful, and it can shift from moment to moment. Trust, by contrast, accumulates slowly as a bot behaves the same way again and again. Its post sums up the idea in four words: "Verify, but continuously re-verify."

Each bot gets what Cloudflare calls a trust band, an explainable rating of how consistently it honors site preferences, along with the observed behavior behind it. Cloudflare says the band is not a permanent judgment and can move.

The two first signals

The first signal is the robots exclusion file, the plain text list of rules a site publishes to tell crawlers where not to go. Cloudflare says it watches for a habit of ignoring those rules rather than one-off slips, judged against RFC 9309, the standard behind the file. It also reads the rules strictly: if the file does not explicitly allow something, a bot should not assume it is permitted.

The second is the 429 response. A bot that consistently slows down or pauses when told it is sending too many requests moves its band up; one that repeatedly ignores the signal moves it down. Cloudflare notes that many problems with known bots come from misconfiguration or bugs rather than bad intent, which is part of why it wants a running record instead of a one-time check.

A bot that keeps falling short can be removed from the Verified list.

What is not public yet

The system is already used internally, and Cloudflare describes the current phase as building up history. It has not published how many bots it covers, where the band thresholds sit or any compliance rates.

Two steps are planned without dates: sharing the scores publicly on Cloudflare Radar, and offering the system to all customers, who would set their own trust threshold for how Verified bot traffic is handled on their sites.