DoorDash has shipped SafeChat, a synchronous multi-model content moderation system that screens every chat message, image, and voice call before delivery. The platform processes 4 million chat messages, 400,000 voice calls, and 200,000 images daily. The constraint is severe: a message cannot be delivered until classified as safe, requiring pipeline completion in a fraction of a second instead of the 2–10 seconds a raw LLM call requires.
The architecture rests on a foundational observation: only single-digit percentages of messages are unsafe. That distribution shapes every cost and latency choice downstream.
Phase 1 deployed a three-layer cascade. A free moderation API at very low threshold auto-cleared 90% of messages. Remaining traffic moved to a fast, low-cost LLM with higher precision, pushing 99.8% of that to safe. The remaining 0.2% went to a precise, higher-cost LLM that scored messages 0–5 across three categories: profanity, threats, and sexual content. Messages scoring 4 or higher in at least two categories triggered Dasher-protective actions, including delivery cancellation.
| Layer | Model Type | Threshold / Precision | Traffic Cleared | Scoring Output |
|---|---|---|---|---|
| 1 | Free moderation API | Very low threshold | ~90% of total traffic | Pass / Fail |
| 2 | Fast, low-cost LLM | Higher precision | 99.8% of remaining traffic | Pass / Fail |
| 3 | Precise, high-cost LLM | Maximum precision | Remaining ~0.2% | Score 0–5 across profanity, threats, sexual content |
Phase 1 accumulated 10 million labeled data points. DoorDash trained an internal classifier on that corpus—no per-call cost, deployed on its own infrastructure. Phase 2 replaced the first two external layers with this model. It produces safeScore and unsafeScore summing to 1. High safeScore clears immediately; high unsafeScore routes to the precise LLM. The internal model now handles 99.8% of chat traffic at under 300 milliseconds. The expensive precise LLM receives less than 0.2% of messages; flagged messages may take up to 3 seconds, adding latency only where a message is likely unsafe.
Image moderation took a different path. DoorDash shadow-tested an external computer vision API for two weeks before production rejected it: false positive rate too high. The team evaluated additional CV models and tuned thresholds through iterative human review. The production pipeline now handles over 200,000 images daily at latency compatible with live interactions.
Voice moderation is newest and hardest. Unlike chat, voice cannot be blocked before words are spoken. The pipeline streams calls in real time, analyzing tone, transcribed words, and conversational context across multiple languages. DoorDash rolled out voice moderation in observe-only mode first—automated enforcement did not activate until confidence thresholds validated against observed data. Available actions include call termination and communication restrictions; severe or repeated violations escalate to human safety agents and trigger account suspension.
| Channel | Daily Volume | Pipeline Approach | Latency Target | Enforcement Actions |
|---|---|---|---|---|
| Chat | 4 million messages | Internal classifier cascade → precise LLM | <300ms (99.8% of msgs); up to 3s (<0.2%) | Delivery cancellation |
| Images | 200,000 images | CV model with iteratively tuned thresholds | Live-interaction-compatible | Image block |
| Voice | 400,000 calls | Real-time stream: tone + transcription + conversational context | Real-time (observe-only rollout) | Call termination; communication restrictions; account suspension |
Enforcement is graduated by severity and recurrence across all channels. The team built no-code configuration workflows and a backtesting harness so policy changes validate against historical traffic before going live. This matters for organizations needing non-engineers to own moderation policy without touching model code.
SafeChat has cut low- and medium-severity safety incidents by 50% since deployment.
For architects building any real-time AI decision pipeline: measure the actual distribution first, then build an aggressive cheap gate for the easy majority. Use production traffic to label data for a cost-free internal model. Reserve the expensive frontier model for irreducible hard cases. This cascade math reduces expensive inference calls by 99.8% with no compromise to recall.
