Know, then Draw: Hybrid AR-Diffusion Paradigm Trains 3B Text-to-Image Model in 6,600 GPU-Hours

Researchers from UC San Diego and collaborating institutions have proposed "Know, then Draw," a two-stage learning paradigm for text-to-image generation that separates semantic understanding from pixel rendering. The approach uses autoregressive (AR) pretraining to acquire high-level conceptual knowledge, followed by diffusion-based post-training for high-fidelity image synthesis — a division of labor the authors argue is more compute-efficient than training either paradigm alone from scratch.

Key findings include that AR pretraining outperforms pixel diffusion as a pretraining objective for downstream rendering quality, and that overall AR generation quality does not strongly predict final image fidelity — semantic quality matters more. This insight motivated a purely semantic tokenizer trained with language supervision alone, dispensing with image reconstruction. Guided by compute-optimal scaling laws derived from controlled experiments, the team trained a 3-billion-parameter model achieving scores of 86 on GenEval and 87 on DPG-Bench using only 6,600 H100 GPU-hours — a notably lean training budget for a model at this scale.