Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation

 
 

Sucheng Ren1,   Qihang Yu2,   Ju He2,   Xiaohui Shen2,   Alan Yuille1,   Liang-Chieh Chen2,  

 1 Johns Hopkins University  2 ByteDance
[Paper]  [Code & Models (Coming soon)]  [BibTeX]
 

🔥 Highlights

 
  • We generalize next-token prediction to next-X prediction.
  • We find that next-cell prediction yields the best performance by capturing richer spatial-semantic relationships.
    •
  • We propose Noisy Context Learning (NCL) by deliberately exposing the model to noisy contexts during training to address the problems in teacher forcing that during inference, errors accumulate over time.
    •
  • Our base model xAR-B (172M) outperforms the large DiT-XL and SiT-XL (675M) while achieving 20$\times$ faster inference. Additionally, our largest model, xAR-H (1.1B), sets a new state-of-the-art with an FID of 1.24 on ImageNet-256, without relying on vision foundation models (e.g., DINOv2) or extra guidance interval sampling.
    •
     
     

    Abstract

     

  • Autoregressive (AR) modeling, known for its next-token prediction paradigm, underpins state-of-the-art language and visual generative models. Traditionally, a ``token'' is treated as the smallest prediction unit, often a discrete symbol in language or a quantized patch in vision. However, the optimal token definition for 2D image structures remains an open question. Moreover, AR models suffer from exposure bias, where teacher forcing during training leads to error accumulation at inference. In this paper, we propose xAR, a generalized AR framework that extends the notion of a token to an entity X, which can represent an individual patch token, a cell (a $k\times k$ grouping of neighboring patches), a subsample (a non-local grouping of distant patches), a scale (coarse-to-fine resolution), or even a whole image. Additionally, we reformulate discrete token classification as \textbf{continuous entity regression}, leveraging flow-matching methods at each AR step. This approach conditions training on noisy entities instead of ground truth tokens, leading to Noisy Context Learning, which effectively alleviates exposure bias. As a result, xAR offers two key advantages: (1) it enables flexible prediction units that capture different contextual granularity and spatial structures, and (2) it mitigates exposure bias by avoiding reliance on teacher forcing. Our base model, xAR-B, outperforms DiT-XL/SiT-XL with 20$\times$ faster inference, while our largest model, xAR-H, achieves a state-of-the-art 1.26 FID on the ImageNet-256 benchmark without relying on additional vision foundation modules (e.g., DINOv2) or advanced guidance interval sampling. Codes will be publicly released.
    •  
     

    Next-X Prediction

     

  • xAR: Autoregressive (AR) Image Generation with Next-X Prediction. The proposed xAR adopts a general next-X prediction framework, where X is a flexible prediction entity that can correspond to: (a) an individual image patch (as in vanilla AR), (b) a cell (a group of spatially contiguous tokens), (c) a subsample (a non-local grouping), (d) an entire image (as in flow-matching), or (e) a scale (coarse-to-fine resolution, similar to VAR). We use red, green, blue, yellow to illustrate the first four AR prediction steps for each entity example. The gray tokens represent the remaining tokens.
    •  
     

    Noisy Context Learning

     

  • Conditioning Mechanism Comparison between Vanilla AR vs xAR. During training, vanilla AR conditions on all preceding ground truth tokens (i.e., Teacher Forcing), whereas xAR conditions on all previous noisy entities, each with different noises (e.g., Noisy Context Learning). At inference, vanilla AR suffers from exposure bias, as errors accumulate over AR steps due to its exclusive training on ground truth tokens, leaving it unprepared for imperfect predictions. In contrast, xAR, trained to handle noisy inputs, reduces reliance on ground truth signals and improves robustness to prediction errors.
    •  
     

    Experiment Results

     

  • Out best variant, xAR-H, achieves a new state-of-the-art-performance of 1.24 FID, outperforming the GAN-based StyleGAN-XL, masked-prediction-based MaskBit, AR-based RAR, VAR, MAR, and flow-matching-based REPA.
  • Our most lightweight variant, xAR-B (172M), outperforms DiT-XL (diffusion-based), SiT-XL (flow matching-based), and MAR (autoregressive-based), while achieving a 19.6$\times$ speedup. Additionally, xAR-L surpasses the recent state-of-the-art model REPA, running 5.3$\times$ faster. Finally, our largest model, xAR-H, achieves 1.24 FID on ImageNet-256, setting a new state-of-the-art, while still running 2.2$\times$ faster than REPA.
    •  
     

    Visualization

     

     

    Bibtex

                    @article{ren2025nexttokennextxpredictionautoregressive,
                    title={Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation}, 
                    author={Sucheng Ren and Qihang Yu and Ju He and Xiaohui Shen and Alan Yuille and Liang-Chieh Chen},
                    year={2025},
                    eprint={2502.20388},
                    archivePrefix={arXiv},
                    primaryClass={cs.CV},
                    url={https://arxiv.org/abs/2502.20388}, 
                            }