A benchmark from Kent State, George Mason, and Binghamton Universities tests whether LLMs can recover semantic information C/C++ compilers lack, then use it to produce faster, provably correct code.
SeGaBench, posted to arXiv on August 4, contains 120 test cases—100 synthetic, 20 from real source code—across three problem classes: low-level assumptions (aliasing, alignment, overflow), data-structure invariants, and high-level semantic lifting. Every case includes hidden enabling semantics, an oracle artifact representing maximum achievable optimization, a correctness validator, a semantic validator, and a reproducible performance protocol. Architects can clone it and run it today.
The evaluation covers five LLMs, each tested five times per case. The strongest produces correct artifacts in 94.8% of responses, achieves at least 1.05× speedup on 83.3%, and succeeds on 93.3% of cases. The catch: correct artifacts often close only part of the oracle gap. Correctness and full performance recovery are not the same.
That distinction matters for inference stack owners. The compiler lacked semantic context—pointer provenance, container invariants, type narrowing—that lives outside the function under analysis. SeGaBench asks the LLM to supply that context as a contract-preserving annotation or rewrite. When the model errs, the validator catches it. When the model is right but conservative, the performance delta against the oracle reveals remaining upside. Both outcomes signal what a production pipeline can act on.
KernelBench (Stanford, ICML 2025) measures full kernel synthesis with one-shot success rates of 12% for DeepSeek R1, 10% for o1, and 10% for Claude 3.5 Sonnet on Level 1 tasks. SeGaBench is narrower—semantic annotation and contract recovery, not full synthesis—which explains the 93.3% case success rate. The scope is intentionally bounded to where LLMs have a plausible advantage over static analysis.
Meta's production KernelEvolve deployment uses LLMs in a closed-loop optimization loop across NVIDIA, AMD, and MTIA hardware, achieving 2× to 14× speedups over torch.compile on real models including Llama-3.1-8B, Whisper, and Stable Diffusion. The system relies on validated outputs; the LLM proposes, an execution harness decides. SeGaBench codifies the same contract: LLMs as speculative semantic proposers, validators as the gate. The benchmark formalizes what production deployments already do informally.
Even at 94.8% response-level correctness, partial oracle gap closure is the norm. A model that annotates aliasing correctly may fail to express the full invariant the optimizer needs to fire the most aggressive transformation. That residual gap is not random noise; it signals which semantic classes are hard for current models. The three-category taxonomy makes it actionable rather than a single pass/fail score.
For architects wiring LLM calls into an offline compilation or kernel synthesis pipeline, SeGaBench is now the baseline. Run it against your chosen model before shipping; compare response-level correctness and oracle gap closure, not just pass rate.