Cloudflare's AI code reviews got 68% cheaper, but found 64% fewer key performance issues

Cloudflare says it cut the average cost of a review by its internal AI code reviewer from $2.43 to $0.77, a 68% drop, and made the median review finish in 1 minute 32 seconds instead of 4 minutes 9 seconds. In a post on October 19, the company also reported where quality slipped: as rated by blind judges, the performance reviewer surfaced roughly 64% fewer important issues.

The reviewer is a coordinator agent that hands each code change to specialist reviewers for code quality, security, performance, documentation and internal standards.

The biggest saving was a caching mistake

Costs had roughly doubled between April and August, from $1.19 to about $2.46 per review, according to Cloudflare. Part of that came from moving reviewers to more capable and pricier models. Tokens per review also more than tripled, to 2.99 million from 0.91 million.

The single largest fix was not a model swap. The first request of each session reused a cached prompt prefix in just 14% to 26% of cases, even though 11,000 to 21,000 tokens of tool definitions and prompts were identical across repositories. The tool's one cache breakpoint sat after a block that included the working directory and the date, so nothing could be shared. Splitting the prompt raised first-request cache hits to between 91% and 99.5%.

Smaller leaks added up. Sub-reviewers in 57% to 76% of sessions tried to call a posting tool they were not allowed to use, wasting a call. And of 446 cancelled jobs Cloudflare sampled, 358 carried on to the end, because the CI system never told them to stop. Jobs now check their own status every 15 seconds.

Models and effort did the rest

Of the $1.67 drop, code changes account for about $0.65 and model and effort settings for about $1.06, Cloudflare estimates. Moving from Sonnet 5 to Sonnet 5.5 at the same per-token price made runs 47% to 62% cheaper, mainly because the newer model made fewer calls. The company notes this split is based on before-and-after comparisons, not a controlled experiment, apart from two randomized tests.

What got worse

Cloudflare had two models from different providers grade 657 findings blind. Important findings per review held roughly steady, and findings rated more severe than they deserved fell 70%. But the performance reviewer lost about 64% of its important findings, which the company describes as about one fewer every 24 full-tier reviews. Security findings fell by about a quarter, a change the company says is not statistically significant.

At medium effort, the code quality reviewer led to roughly 25% as many findings that authors went on to fix, compared with high effort, so Cloudflare moved it back to high, at about 2 cents more per review. Performance and security still run at medium effort, and the team plans to test them at high next.