JVT: A Simplified DiT Transformer Tackles Pixel-Space Video Diffusion Without Autoencoders
Researchers have introduced JVT (Just Video Transformer), a DiT-style Transformer designed for pixel-space video diffusion that removes the dependency on pretrained variational autoencoders, cascaded stages, U-Net skip connections, or auxiliary detailers. By using large spatiotemporal patches under a rectified-flow objective, JVT makes direct modeling of raw video tractable, achieving generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings.
A key finding from the work is that patch embedding represents a significant bottleneck for generation quality and efficiency, while standard adaptive layer normalization (AdaLN) remains an effective conditioning design. The pixel-space formulation also unlocks native joint generation of pixel-aligned modalities — the paper demonstrates camera-conditioned RGB-disparity generation on RealEstate10K, improving camera-following accuracy and yielding explicit scene geometry at inference without an additional reconstruction model.