METR Report Finds Anthropic's AI-Assisted Research Nearly Doubles Model Improvement Rate, But Warns of Evaluation Gaps

Independent AI safety organization METR published an assessment November 5 concluding that Anthropic's use of AI in its own research and development has nearly doubled the pace of model improvement, while flagging that current evaluation methods may not catch emerging risks from automated research.

Acceleration Measured at 1.9x With Wide Uncertainty

From August 12 to October 29, 2026, METR staff conducted a high-access evaluation at Anthropic, reviewing internal benchmark scores, training compute data, algorithmic progress estimates, code contributions, and interviews with roughly 15 researchers [1]. Their headline finding: since October 2025, Anthropic's models have improved on benchmarks at an estimated 1.9 times the pre-October 2025 rate, with a 95% confidence interval of 1.5–2.8x [1]. In practical terms, METR says Anthropic made roughly two years of historical progress in the past year — approximately the distance from Claude 2.1 to Sonnet 4.5 [1].

The measurement uses AECI, Anthropic's internal version of the Epoch Capabilities Index, which aggregates scores across multiple benchmarks. Robustness checks varying benchmark selection, model dating, and statistical assumptions produced acceleration estimates ranging from 1.1x to 2.9x [1]. METR emphasizes high uncertainty: "We would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate" [1].

Most Acceleration Attributed to AI Use, But Evidence Redacted

METR's subjective estimate is that approximately 80% of the current acceleration stems from Anthropic's extensive use of AI in R&D, though with "substantial uncertainty" [1]. Three evidence streams support this:

• Researcher interviews centered on roughly 2x speedup in their areas [1]
• Internal Slack polls showed self-reported uplift reaching 12x by October 2026, though METR calls this "only very loose evidence" [1]
• "Signs of life" suggest fully autonomous AI contributions could grow soon, though AI currently assists rather than leads the most important training work [1]

Critically, METR states that "a majority of the evidence we found most informative for these conclusions" was redacted by Anthropic [1]. Anthropic described the redactions as "aimed at removing confidential information that could accelerate other AI developers" [1]. The redacted material included detailed acceleration estimates for specific pipeline stages, algorithmic progress breakdowns, bottleneck analyses, and forecasts for future models [1]. As a result, METR notes the public "has to rely on METR's judgment much more than would be ideal" [1].

Compute Scaling Ruled Out; Bottlenecks Unclear

METR says it received "convincing evidence" that the acceleration was not driven by faster growth in training compute, though those details were also redacted [1]. The assessment found no clear bottlenecks in algorithmic progress, training compute, experiment compute, or data that would constrain progress in the next 12 months — but METR's understanding of bottlenecks "changed several times as we got more detailed information, so we are not confident in our current assessment" [1].

Scope Changes and Process Concerns

The original engagement included verification of Anthropic's compute allocation between capabilities and safety research. Anthropic canceled that component mid-project, stating it wanted to keep the engagement focused on risk assessment [1]. METR worked on Anthropic devices using Anthropic-provided tokens, took no payment, and had unilateral rights to disclose the existence of redactions and whether findings depended on them [1].

Implications for Safety Thresholds

The findings bear on Anthropic's Responsible Scaling Policy, which lists AI R&D acceleration as a key risk indicator [1]. An EA Forum analysis by a METR colleague (writing personally) quotes the RSP's "dramatically accelerated progress due to AI" threshold as: observing or expecting overall capabilities progress at twice the fastest sustained rate before AI contributed significantly to R&D, plausibly attributable to AI automation [5]. That analysis argues this threshold has likely been crossed [5].

Limitations Flagged by METR

METR lists several ways its estimate could be wrong. Acceleration could be higher if: recent growth has further increased (possibly hyperbolic), automation is closer than estimated, or AECI understates real-world impact — citing resolution of Millennium Prize Problems as an example of sudden threshold effects [1]. Acceleration could be lower if: non-AI factors explain most gains, progress is limited to well-specified benchmarks, or Anthropic's 2023–2025 historical rate was unusually low [1]. Notably, when including non-Anthropic frontier models, the estimated acceleration drops to roughly 1.4x, suggesting Anthropic may have been "subfrontier in 2025" [1].

Broader Context

Anthropic CEO Dario Amodei published a "We Must Pace the Frontier" essay in mid-September during the assessment period [4]. The company is also preparing to go public [4][6]. Anthropic has separately published its own "R&D Automation Index" showing Claude "leads" 26% of Anthropic's AI R&D work, with the share of work at or above the "AI collaborates" level above 90% [8], though METR's assessment used different methodology and deeper access.

The report arrives as frontier labs increasingly open their doors to third-party evaluators. Anthropic has established three tiers of evaluator access, with METR operating at the highest tier with "employee-like access" [7]. However, the redaction pattern in this assessment illustrates the tension between commercial secrecy and public accountability in AI safety oversight.