Meta's AI Test Generator Caught 74 Serious Bugs in Six Months
Meta has moved its JiTTests — a system that uses LLMs to automatically generate and run software tests against code changes — from research prototype to production infrastructure. Unlike traditional AI code review that just flags suspicious lines, JiTTests generates executable tests designed to fail if a code change introduces a real bug. The system uses two workflows: one that treats every change as potentially buggy ('dodgy diff') and another that infers the developer's intent to predict likely failure points. A second layer of 'LLM-as-Judge' ensembles and a rule-based assessor (RubFake) filter out false positives before human engineers are ever notified. Meta reports that in the first half of 2026, the system helped catch 74 serious (SEV) privacy bugs before they reached production, contributing to a 20% reduction in privacy SEVs during that period. About 64% of Meta's 589 engaged engineers reported actionable findings, with a reported 60% true-positive rate. The system runs overnight on spare capacity to avoid slowing down day-to-day development.
Meta has moved its JiTTests — a system that uses LLMs to automatically generate and run software tests against code changes — from research prototype to production infrastructure.
Unlike traditional AI code review that just flags suspicious lines, JiTTests generates executable tests designed to fail if a code change introduces a real bug. It uses two workflows: one that treats every change as potentially buggy ('dodgy diff') and another that tries to predict where errors might hide by inferring a developer's intent from code titles and summaries.
To keep engineers from drowning in false alarms, Meta layers on an ensemble of LLMs that judge whether each test failure is a real bug or a false positive, alongside a rule-based assessor called RubFake. Human engineers only get pulled in when the system's confidence is high enough — a 'human-on-the-loop' approach rather than manual review of every output.
The impact, according to Meta, shows up where it counts most. In the first half of 2026, JiTTests helped catch 74 serious (SEV) privacy bugs before they reached production, contributing to a 20% reduction in privacy SEVs during that period. About 64% of the 589 engineers who engaged with the system reported actionable findings, and Meta claims a 60% true-positive rate.
For builders, the trade-off is clear: the system requires extra computational steps to generate and run tests for every code change, but it may catch bugs that standard tests miss. The system runs overnight on spare capacity to avoid slowing down day-to-day development, and engineers who use it don't have to read test code — they just confirm whether a flagged behavioral change was intended or not.
Source: Meta EngineeringMeta Engineering