Meta's AI Agents Ran 1,097 Experiments in a Month for a 0.23% Search Gain
Meta researchers deployed autonomous AI agents to run 1,097 machine-learning experiments over 31 days, consuming roughly 360,000 GPU-hours, to improve the company's search-suggestion system by 0.23% in a live A/B test. The results, detailed in a November 18 preprint, offer a rare concrete look at the real-world returns of large-scale agentic AI deployment in a production environment [1].
The experiment
The target was Nullstate, Meta's search-suggestion ranker that determines the recommendations users see when opening the search bar before typing a query. The system handles more than a billion ranking requests per day [2].
Traditionally, improving such models requires engineers to hypothesize changes, implement them, train models, and evaluate results—a slow, serial process. Meta's "MLE agents" automated much of this loop. Using Anthropic's Claude Opus 4.8, the agents proposed architectural changes, wrote code, launched distributed training jobs on 64-GPU trainers, and evaluated offline metrics. Engineers retained control over the objective, strategic direction, and final model review [2].
The agents ran nominally seven candidates in parallel per iteration, plus a variance probe, across 131 iterations. Each training-and-evaluation run took a median of 3.54 hours. Of 1,097 completed candidate experiments, 325 were accepted—a 29.6% acceptance rate, up from 15.6% in an earlier, sequential version of the system [2].
Offline gains, online validation
The accepted changes compounded into a -0.0224 improvement in a weighted normalized-entropy objective (lower is better), driven mainly by better prediction of downstream engagement [2]. Several accepted changes were architectural ideas the human team had never tried, including Random Fourier Feature gating, learned-sigmoid attention gating, and a zero-init gated distillation bypass [2].
Meta then tested the consolidated model in a live A/B test over one week. Search-session volume increased 0.23% relative to the control, with a 95% confidence interval of ±0.15 percentage points. Downstream engagement metrics moved in a positive direction [2].
Trade-offs and undisclosed costs
The experiment exposed a practical trade-off. The new model slowed the search interface's time-to-render, a latency regression the researchers disclosed but did not quantify [2]. The agents had optimized against a calibrated offline objective tied to engagement; rendering speed was monitored separately and not part of the agent's reward function. The authors acknowledged that an agent optimizing a pure quality objective does not surface such trade-offs on its own [2].
Meta estimated the campaign consumed approximately 360,000 GPU-hours, including 155 variance probes and 2,798 failed or retried attempts [2]. The company did not disclose the financial cost, nor did it run a matched comparison against a conventional engineering team pursuing the same objective. The paper leaves permanent rollout and that human-only benchmark unresolved [2].
What this shows—and doesn't
The work demonstrates that autonomous agents can perform meaningful ML engineering on a production-scale system, clearing an industry-level bar far higher than academic benchmarks [2]. The calibrated objective, structured context (including a "what didn't work" blocklist), and parallel evaluation loop were more decisive than the underlying LLM, which was held fixed throughout [2].
But the results also illustrate diminishing returns: the best model was found well before the campaign's end, and later iterations added nothing. The agents also exhibited exploration collapse (an 0-for-8 streak on optimizer tuning before human redirection) and unreliable self-correction [2]. The offline-online gap remains: not every offline winner transfers, and the agent has no mechanism beyond the calibrated objective to predict which will.
For a platform of Meta's scale, a 0.23% lift in search sessions is statistically significant and operationally meaningful. Whether it justifies the compute, the latency cost, and the engineering overhead of the agent harness is a question the paper does not answer.
Sources
- Meta's AI agents ran over 1000 experiments in a month but improved ... (cryptoslate.com)
- MLE Agents: Autonomous ML Engineering in Large-scale ... (arxiv.org)
- 3 Unstoppable AI Stocks to Buy Hand Over Fist Now That the Midterms Are ... (theglobeandmail.com)
- NEAR Protocol (NEAR) Historical Prices (coinlore.com)