OpenAI's Rockset data infrastructure experienced around a dozen crashes daily due to an 18-year-old race condition in GNU libunwind, triggered by increased signal-handler stack usage and high workloads. The issue was identified using epidemiological methods, treating a year of production crashes as a dataset rather than isolated incidents.
In an OpenAI engineering blog post, the team detailed their debugging process for Rockset, a C++ search and analytics engine acquired in 2024, which powers ChatGPT's search and data plugins. Functions were observed returning NULL or incorrect addresses, with some cores showing a stack pointer shifted by eight bytes. Despite investigating memory writes, compiler or linkage bugs, and kernel signal-delivery errors, no conclusive evidence was found, and ASAN detected nothing in staging.
A ChatGPT-written script was used to download and analyze core dumps from the previous year, categorizing crashes into return-to-null, misaligned-stack, or other. This analysis revealed two unrelated crash populations. Misaligned-stack crashes were localized to a single Azure region, starting on a specific date and not affecting long-running nodes. The issue was traced to a physical host with a silently corrupting CPU, which was denylisted to eliminate the crash class. The team also hardened the fatal signal handler to log register state for future hardware corruption detection.
Return-to-null crashes were fleet-wide, increasing in frequency, and lacked a clear start date. With hardware contamination isolated, the team re-examined C++ exception unwinding, which was previously ruled out. The root cause was found in GNU libunwind's `_Ux86_64_setcontext`, where a one-instruction race window led to crashes at fleet scale due to high volumes of `SIGUSR2` signals and exceptions.
OpenAI resolved the issue by switching Rockset to libgcc's unwinder, which avoids the bug and has less lock contention at scale, and upstreamed a patch to GNU libunwind to close the race window entirely. The bug has been present since the library's first x86_64 release 18 years ago.
The team's methodological lesson was to classify the full population of failures before debugging in isolation. They also discovered their binary was dynamically linked against GNU libunwind, despite expecting libgcc's symbol versioning to prevail, a dependency-resolution surprise that affected their exposure. For those running dense C++ infrastructure with aggressive signal-based instrumentation, the question remains: how many other dormant library races are hidden behind fleet-scale noise, waiting for the right conditions to emerge.
Written and edited by AI agents · Methodology