Netflix consolidated five separate artwork-ranking models into one by encoding images with CLIP embeddings, then built two additional production systems—query-aware artwork ranking and video preview personalization—using the same embedding infrastructure. The shift from ID-based to content-aware asset representation solved the cold-start problem for new titles and canvases while delivering measurable gains in discovery and streaming hours.
The artwork system works by concatenating a CLIP image embedding (768 dimensions) with a learned ID embedding, then passing both through an MLP layer to score assets against members. Because CLIP embeddings are largely invariant to crop and resize, the same artwork rendered across five different canvases—billboard, vertical-box, horizontal-panel, short-panel, landscape-panel—maps to nearly identical vectors. This invariance let Netflix replace five canvas-specific models with a single unified model that pools interaction signal across all canvases. The per-canvas models had treated each cropped rendering as a separate asset ID with no way to transfer learning between them; the unified model immediately applies a member's affinity learned on high-traffic canvases to sparse ones.
Training the unified model across disparate canvases introduced a data-mixing problem. Raw impression counts would let the highest-volume canvas dominate, leaving low-data canvases unhelped. Netflix solved this with reward-based weighting, where each training example is weighted by the long-term reward score of its interaction type rather than by impression volume. A canvas contributes in proportion to the long-term value of the interactions it drives, not to how many times it was shown. This rebalancing happened automatically, with no hand-tuned hyperparameters per canvas.
The ablation tested three variants against the prior five-model system: image embeddings alone on the old per-canvas models, a unified model with only ID embeddings, and both together. Offline, measured with inverse propensity scoring on exploration traffic, both ideas helped on data-starved canvases but neither moved the needle significantly on its own. Online, across a four-week A/B test on all device platforms, only the combined approach—unified model plus image embeddings—delivered statistically significant lift. The effects compounded: the short-panel lift from the combined approach exceeded V1 and V2 added together, because mature canvases taught the shared model how CLIP embeddings map to preference, and that mapping transferred directly to sparse canvases. The real test came when Netflix redesigned its TV home screen to make short-panel the dominant canvas overnight. Netflix shipped the unified model ahead of the launch and held back a control group on the old per-canvas system for a month. The unified model absorbed the shift immediately with statistically significant gains on both discovery metrics and streaming hours, larger than the steady-state ablation because a sudden canvas shift is exactly where content-aware embeddings help most.
Query-aware artwork ranking, a separate production system, leverages the same CLIP embeddings without additional modeling. Because CLIP projects text and images into one shared embedding space, Netflix measures how well a search query matches a candidate artwork by computing cosine similarity between the CLIP text embedding of the query and the CLIP image embedding of the asset. This alignment term is blended with the personalization score through a mixing weight tuned via A/B testing, letting the ranker balance "what we think you like" against "what you just asked for."
For video previews, Netflix built a distinct third system. SeqCLIP averaged CLIP embeddings of individual frames, capturing what a preview looked like but missing motion, pacing, dialogue, and soundtrack. MediaFM, Netflix's in-house multimodal foundation model trained on 80 million shots, fuses visual (via SeqCLIP), audio (via a pretrained speech and audio encoder), and text (via captions) into a single embedding per shot. In a five-week online A/B test, MediaFM outperformed SeqCLIP, which outperformed ID-only baselines, with the largest gains on TV. The audio and text signals that visual-only encoders cannot capture added real value.
To avoid running expensive end-to-end trials for every new embedding candidate, Netflix gates the funnel with a linear probe trained on exploration data. For a fixed set of titles, the system finds the debiased popularity winner—the asset with the highest interaction rate after adjusting for how often it was shown—and trains a linear classifier to predict that winner from the embedding alone, with no title or metadata. If the embedding captures the semantic drivers of popularity, the probe beats random guessing; if not, it does not. All three signals—linear probe accuracy, offline IPS lift, and online A/B results—ranked MediaFM ahead of SeqCLIP, validating the proxy task as a cheap screening step before full evaluation.
The Netflix Embedding Store decouples foundation-model updates from personalization-model deployments. A new embedding is registered, backfilled across the catalog, and validated independently, without touching training or serving code in downstream models. The same embeddings are served at training time and online inference time, eliminating skew. This infrastructure let Netflix swap CLIP into artwork, stand up query-aware ranking on the same vectors, and roll MediaFM through video previews as independent changes rather than coordinated migrations. For teams building recommendation systems with vision and language components, the lesson is that content-aware embeddings solve cold-start not by gathering more data but by transferring learned preferences across assets that share semantic properties, and that multimodal signals compound their benefits when unified in a shared embedding space.