Meta's LLM-written 'catching' tests helped flag 74 serious privacy bugs before they landed

Meta says its Catching JiTTests, tests generated by large language models to fail on risky code changes, played a key role in stopping 74 serious privacy bugs before they reached production in the first half of 2026. In a post on October 29, the company said privacy incidents of that severity fell 20% over the same period.

The bigger shift is in precision. In a paper posted in January, Meta reported that only 8 of 41 candidate bugs flagged to engineers were confirmed as real. Engineers using the production system now report a 60% true-positive rate, according to the company.

Tests that are supposed to fail

Ordinary tests are written to pass and guard against future breakage. Catching tests flip that idea: they are generated for a specific code change, or diff, and are meant to fail if the change alters behavior in a way nobody intended. Meta argues this finds bugs that an LLM code reviewer reading the diff might miss, because the test shows a concrete behavioral difference rather than an opinion.

The system runs overnight on spare computing capacity and looks only at diffs from the previous day that a risk model marks as likely to cause problems. Two workflows then write tests. An intent-aware one guesses what the engineer was trying to do, imagines ways that could go wrong and writes tests aimed at those failure points. The other, which Meta calls the "dodgy diff" workflow, simply assumes every change is a bug and generates tests that expose any difference from the previous version.

Filtering the noise before a human sees it

Most failing tests are false alarms, which was the main obstacle in Meta's earlier research. The company now passes each failure through two filters: an ensemble of several frontier LLMs that votes on whether it looks real, and a deterministic rule-based checker called RubFake. Meta notes that the LLM judges are non-deterministic and hard to verify, which is why it pairs them with fixed rules.

Only high-ranked failures reach an engineer. Meta calls this human-on-the-loop: people set the criteria and review outcomes in aggregate rather than approving every result. The aim, the company says, is that engineers "rarely need to read test code" and only confirm whether a behavior change was intended.

What the numbers leave out

According to Meta, 589 engineers have engaged with the system, which it says is 64% of those it contacted. That also means roughly a third did not respond. The 60% figure is based on what those engineers reported, not an independent audit.

Meta credits the tests with a key role in the 20% drop in privacy incidents but does not present a controlled comparison, and it gives no figures on how many diffs were analyzed, how many tests were generated or what the overnight runs cost. The January paper, presented at FSE 2026, analyzed 22,126 generated tests and said rule-based and LLM assessors cut human review load by 70%. Whether the approach holds up outside Meta's own codebase has not been shown.