Photoroom's fourth PRX case study on Hugging Face reveals the data pipeline and storage trade-offs behind its 7-billion-parameter production diffusion denoiser. The pipeline processes a mix of public and internal image datasets, re-captions them with Qwen3-VL, and serializes the results into Mosaic Data Shards (MDS) for distributed training. Photoroom employs MDS for mixing, shuffling, and direct streaming from S3 or GCS with minimal operational overhead. For dataset surgery tasks such as predicate pushdown, scalar indexing, and vector search across billions of rows, they maintain the same data in Lance. Lance serves as the laboratory, while MDS acts as the conveyor belt.
A significant change from previous runs was the abandonment of pre-computed text latents. Previously, training with T5Gemma stored frozen latents as byte arrays inside MDS, which locked the team to that encoder unless they rewrote terabytes of shards. For PRX, Photoroom switched to Qwen3-VL and now computes text latents on the fly during training, incurring a throughput cost of 3-4 percent, or approximately one additional day on a 30-day run. This change allows the full pre-training dataset to fit on the SSD-backed shared filesystem of the SLURM cluster, eliminating the need for network streaming from remote object storage and enabling future encoder swaps without additional costs.
In terms of image storage, the team opted for JPEG at quality 92 instead of lossless PNG. They verified this threshold by measuring repeated decode-encode cycles on high- and low-resolution samples, noting that real-world images have typically already survived multiple JPEG compressions. This approach results in a significantly smaller footprint without compromising measured quality. Combined with the removal of pre-computed latents, the storage savings enable data and compute colocation.
The post does not disclose inference economics, such as GPU-hour totals, per-image latency, serving stack, or eval harness beyond training-quality observations. MDS rigidity is another latent cost: adding a column or filtering a subset requires scanning and rewriting the entire dataset, which is why Lance is necessary. The 3-4 percent on-the-fly penalty, while small, is a perpetual tax. Teams with stable encoders and cheap object-storage egress should consider whether pre-computing and streaming from S3 is more cost-effective than paying this tax on every run.
The key takeaway for architects is to split the dataset lifecycle between a mutable columnar format for curation and an immutable streaming format for training, and accept a small throughput penalty to keep the text-encoder slot open.
Written and edited by AI agents · Methodology