OpenAI fixes 18-year-old GNU libunwind race condition via 'epidemiological' debugging

OpenAI engineers debugged and fixed an 18-year-old race condition in GNU libunwind's _Ux86_64_setcontext function that was causing mysterious crashes in Rockset, the C++ data infrastructure service powering ChatGPT's search and data plugins. The breakthrough came not from analyzing individual crash cases—which presented contradictory symptoms—but from applying 'epidemiological debugging': building a pipeline to automatically analyze every production core dump from the past year, then searching for population-level patterns. The team had ChatGPT write a script that extracted registers from each core file, filtered known false positives, and labeled crashes as return-to-null, misaligned-stack, or other, running it in parallel across a year of Rockset dumps.

What appeared as one bug turned out to be two: hardware corruption (a single Azure physical host silently producing incorrect CPU results) and a genuine libunwind race condition. The root cause in libunwind: during C++ exception unwinding, the function updates the stack pointer (%rsp) before finishing the read of the instruction pointer from the struct on the old stack. If a signal arrives in that picosecond window—roughly 100 picoseconds at modern clock speeds—the kernel builds its signal frame on top of the struct, corrupting the instruction pointer. OpenAI's Rockset uses SIGUSR2 signals every few milliseconds for per-query accounting, creating far more signal delivery events than typical applications and turning a theoretically possible race into a production crash. The fix reorders instructions so %rip is read before %rsp is updated, eliminating the race window entirely. OpenAI upstreamed the fix and reproducer to GNU libunwind.

For practitioners debugging production crashes with contradictory symptoms: this case study demonstrates that 'complete, labeled data across the entire population of failures' is faster than deeper analysis of individual cases. The team's own conclusion: 'The most important step was not clever assembly reading or deep knowledge of the details. It was building a high-quality data set.' Population-level observation revealed what case-by-case reasoning could not: two unrelated phenomena being conflated. Watch for this pattern in your own incident analysis and consider whether your logging and crash capture enable population-level debugging.