METR Finds AI Accelerates Vulnerabilities But Not Optimization

METR published an empirical research note August 14 measuring whether LLMs have bent discovery curves across three domains: cyber vulnerabilities, mathematics, and algorithmic optimization. The finding is mixed. Vulnerabilities accelerate. Math accelerates weakly. Algorithmic optimization shows no detectable acceleration across seven benchmarks. Authors Tom Cunningham and Nate Rush measured slope changes in public data using agents for collection and analysis.

Vulnerability acceleration is unambiguous. Reported rates in 2026 rose across cURL, OpenSSL, Firefox, and Microsoft. On cURL and OpenSSL, most new disclosures carry AI attribution. On Firefox, Microsoft, and aggregates, AI credits account for a small share—the bulk unlabeled. Databases tracking actively exploited vulnerabilities show significantly lower year-over-year growth than databases of known vulnerabilities. AI finds more flaws than adversaries weaponize in public records. Higher-severity categories show less acceleration than lower tiers, but all accelerate.

Anthropic mathematician Levent Alpöge disproved the Jacobian conjecture—Problem 16 on Stephen Smale's 1998 list of open problems—on July 19 using Claude. He posted the disproof on X on July 20. The counterexample is a three-variable polynomial map with a constant Jacobian determinant that maps three distinct inputs to one output, proving global non-invertibility. Independently verified by several mathematicians within hours. Two additional resolutions followed from Ben Green's list: Problem 44 and part of Problem 100. arXiv submissions doubled in some areas in under 12 months. METR grades the math evidence weak—historical baselines for open-problem resolutions are too thin for confident slope estimates.

The optimization finding weighs most for ML platform teams. Seven benchmarks—CIFAR-10, Hutter compression, Gurobi mixed-integer programming, MIPLIB, nanoGPT speedrun, Stockfish, and matrix-multiplication exponent—show no slope change since January 2026. Two series including LLM contributions show no acceleration. METR flags a puzzle: Yuksekgonul et al. reported in January 2026 that a simple model advanced the frontier on five optimization problems. The result hasn't appeared in aggregate trend lines.

METR explains the domain split through verifiability. Vulnerability discovery and mathematical counterexamples share a property: progress is cheap to verify. A patch tests in seconds; a counterexample verifies by hand in minutes. Optimization benchmarks require expensive training runs with noisy signals. If AI closes the evaluation loop cheaply per attempt, exploration scales. If each trial costs hundreds of GPU-hours, the advantage compresses. METR also raises a possibility: AI labs may discover algorithmic gains internally, not reflected in public benchmarks. The optimization gap may be a measurement artifact, not a ceiling.

The note frames the domain split as relevant to recursive self-improvement. If AI cannot yet accelerate algorithmic optimization measurably in public data, the conditions for a self-reinforcing R&D loop aren't met. METR acknowledges mistakes likely remain in its data and invites corrections.

Public evidence supports acceleration of vulnerability discovery and, weakly, mathematical search over large spaces. It does not support expecting algorithmic efficiency gains on public benchmarks. The gap between lab announcements and aggregate trend lines is worth tracking.