Databricks cuts network-config latency 98.5% with async snapshots

Databricks' serverless platform starts tens of millions of VMs daily across AWS, Azure, and GCP. Each one requires network configuration: storage destinations, private endpoints, and Unity Catalog permissions. Billions of config requests per day hit this bottleneck at startup. Synchronous answering no longer scales.

The original design chained services on the critical path: the network configuration service called Unity Catalog, network policy stores, and Delta Sharing, aggregated results, and returned the assembled config. With multiple services in series, p99 RPC latency hit 5,000 ms. Availability degraded with each dependency. Multi-service aggregation repeated per workspace per startup, duplicating work as tenant count grew.

The re-architecture splits the problem into two paths. The management path runs asynchronous: upstream services emit change events to a message queue when customers modify a network policy, add a Unity Catalog connection, or change Delta Sharing configuration. An event processor consumes those events, determines affected workspaces, and fans out per-workspace update notifications. A local event manager fetches details, recomputes the full workspace config, and writes it to a partition-local snapshot store. A reconciler independently re-syncs all workspaces in the background as a safety net, ensuring eventual consistency even when events are missed or replayed out of order.

The serving path reads from storage. When a cluster starts, the network config service pulls the pre-computed snapshot with zero upstream calls. That decoupling—async computation, synchronous-read serving—raised availability to 99.99% and cut upstream call volume 86%. On latency, the Databricks post presents two figures: the narrative body states p99 fell from 5,000 ms to 125 ms (97.5% reduction), while the introductory callout states 75 ms (98.5% reduction). This likely reflects different measurement windows or rollout stages. The order-of-magnitude improvement stands either way.

Three design decisions matter most for teams building analogous systems. First: events carry only workspace and resource identifiers, not the config payload itself. Lightweight events free of sensitive data are safe to queue, replay, and reorder without introducing a data-exfiltration surface. Second: snapshot stores are partition-local, co-located with the workspaces they serve. This contains blast radius during incidents and eliminates cross-partition dependencies on the hot path. Third: static stability—during upstream outages, the serving path returns the last-known-good snapshot rather than failing open or closed. Clusters launch; workloads run; the incident stays contained to the management plane.

The reconciler is the difference between "eventually consistent with finite drift" and "consistent only if no events are ever dropped." Periodic full re-sync is operationally expensive but cheap compared to silent inconsistency bugs that surface only under failures. Databricks treats the reconciler as mandatory, not optional.

For ML platform leads, the pattern applies directly. Model serving endpoints require the same config at startup: which feature store endpoints are reachable, what auth policies apply, which model versions route to this replica. Synchronous calls to a model registry on the hot inference path is the same antipattern Databricks eliminated. Snapshot pre-computation with event-driven invalidation backed by a reconciler is the appropriate architecture for config that changes infrequently but must be served at high request rates with low latency and high availability. The 86% reduction in upstream call volume is the number to bring to capacity-planning conversations.