Google DeepMind released SL2T (Sign Language-to-Text), a multilingual model enabling sign-language transcription on consumer phones for the first time. The model powers new sign-to-text dictation in Gboard and Live Transcribe for Pixel 11, launching with American Sign Language (ASL) to English conversion. Users can now sign instead of typing to search the web, draft messages, or interact with Gemini, with additional languages coming soon.
SL2T was trained on over 100,000 hours of data across 50+ sign languages (roughly 25% ASL). Unlike earlier approaches that used intermediate annotations called "glosses," SL2T translates directly from body-pose coordinates to text, capturing non-manual markers and spatial constructions that glosses miss. The model achieves a zero-shot BLEURT score of 70 on the FLEURS-ASL benchmark — significantly higher than any previously reported result for ASL-to-English translation.
The engineering addressed real-world constraints: streaming latency reduction, hallucination prevention on non-signing inputs, fairness for left-handed signers (10% of the population), and performance on one-handed signing (used while holding a phone). Privacy is preserved by using on-device pose-landmark detection (MediaPipe Holistic) rather than raw video — only geometric coordinates are sent to the server for translation, and the original video is immediately discarded.