Cloudflare reclaimed 100TB of RAM across its global infrastructure by reducing the number of hash points in its consistent hashing algorithm and compacting the data structures that store them in Rust. The optimization came from mathematical analysis showing that the last 90,000 hashes out of 100,000 per server were buying only a 0.7% reduction in error, making them pure waste at scale.
The problem started in Pingora Backend Router, Cloudflare's internal load-balancing service, which uses consistent hashing to route cacheable requests to servers by URL. Consistent hashing maps servers and requests onto a number line based on hash values, then assigns each request to the nearest server. To handle uneven load distribution—a problem that grows worse as server count increases—the industry standard is to assign multiple hash points per server. Cloudflare's baseline was 160 hashes per server, the same default used in NGINX and adopted by Pingora. With a weighting factor of 625 based on server storage capacity, the actual number deployed was k = 160 × 625 = 100,000 hashes per server stored in memory.
The Cloudflare engineering team derived the mathematical relationship between hash count and load-balancing accuracy. According to their post, the coefficient of variation—a measure of how unevenly requests distribute—follows the formula CV_k = √((N-1)/(N*k+1)), where N is server count and k is hashes per server. Plotting this showed diminishing returns: each order-of-magnitude increase in hash count bought progressively smaller accuracy gains. More critically, with 32-bit hash values, collision probability increases between 10,000 and 100,000 hashes per server in data centers with 2048 servers, introducing unpredictable errors that the mathematical model does not account for. The team determined that reducing hash count by 90% would incur no appreciable error in their configuration.
The second optimization came from struct packing. The original Point struct stored a 32-bit hash and a 32-bit server index as eight bytes. Zaidoon observed that Pingora would never coordinate more than 65,536 servers, so a 16-bit index suffices. Rust's alignment rules normally prevent shrinking the index without shrinking the struct, but storing the hash and index as a six-byte array and accessing them through getter methods achieved the same compiled result. This reduced memory footprint by 25%.
Rolling out the change required care: switching hash rings globally would invalidate cached content and spike origin traffic. Cloudflare ran both the old and new rings in memory simultaneously, routing requests to each on a per-request basis using their migration framework. They rolled out in layers—small validation locations first, then progressively larger data centers—while monitoring backend-selection traces, memory usage, cache behavior, and origin traffic. Only after reaching 100% traffic on the new ring did they decommission the old one. The sharp memory drop on decommission day showed the 100TB reclaim.
The techniques are available now in the pingora-ketama crate as a Rust feature flag, with both v1 and v2 rings runnable simultaneously. For any team running consistent hashing at scale—whether for load balancing, caching, or distributed task assignment—the lesson is to measure the actual contribution of each optimization step against the mathematical model, then cut what does not pull its weight.