The FID Lottery: Training-Seed Variance Dominates Sampling Noise in Image-Generation Benchmarks

A new research paper challenges one of the foundational assumptions behind image-generation evaluation: that Fréchet Inception Distance (FID) scores are stable enough to reliably rank competing models. The authors show that every reported FID reflects two compounding "lotteries" — one drawn at training time (weight initialisation, data ordering, per-step diffusion noise, and hardware stochasticity) and one at sampling time (the initial noise used to generate images). Critically, training-axis variance substantially outweighs the sampling variance that practitioners typically account for through repeated sampling, meaning a single training run can land on a meaningfully different score than an independent re-run would produce.

Studying several hundred SiT networks ranging from S to XL on ImageNet 256×256, the researchers find that the column-to-column spread across independent training seeds visually dominates within-run sampling variation. Neither classifier-free guidance, additional compute, larger model size, nor learning-rate schedules reliably closes the gap. The finding has direct implications for leaderboard credibility across the field: half-unit FID differences, which routinely justify architectural choices and multi-million-dollar GPU budgets, may fall within the hidden variance of the training lottery.