---

 

Method

 

There exists recent work in computer vision, named VAR, that proposes a new autoregressive paradigm for image generation. Diverging from the vanilla next-token prediction, VAR structurally reformulates the image generation into a coarse to fine next-scale prediction. In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into intra-scale modeling, which captures local spatial dependencies within each scale, and inter-scale modeling, which models cross-scale relationships progressively from coarse-to-fine scales. This decoupling structure allows to rebuild VAR in a more computationally efficient manner. Specifically, for intra-scale modeling --- crucial for generating high-fidelity images --- we retain the original bidirectional self-attention design to ensure comprehensive modeling; for inter-scale modeling, which semantically connects different scales but is computationally intensive, we apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. We term this new framework M-VAR. Extensive experiments demonstrate that our method outperforms existing models in both image quality and generation speed. For example, our 1.5B model, with fewer parameters and faster inference speed, outperforms the largest VAR-d30-2B. Moreover, our largest model M-VAR-d32 impressively registers 1.78 FID on ImageNet 256X256 and outperforms the prior-art autoregressive models LlamaGen/VAR by 0.4/0.19 and popular diffusion models LDM/DiT by 1.82/0.49, respectively.

 

---

 

Decoupled Attention

 

We note that intra-scale attention scores account for 79.6% of the total attention scores in 256X256 image generation and 77.1% in 512X512 image generation. This dominance of intra-scale attention suggests that capturing fine-grained details within the same scale is crucial for high-quality image synthesis. More interestingly, we observe that the associated computation cost presents a contrasting scenario --- despite intra-scale attention contributing the most to the attention scores, it only consumes 23.92% of the computation cost for 256X256 images and 30.32% for 512X512 images; In stark contrast, inter-scale attention, which accounts for a much smaller portion of the attention scores (20.4% for 256X256 and 22.9% for 512X512 images), is responsible for the majority of the computation cost, \ie, 76.1% and 69.7% respectively. The disparity between the attention scores and computation cost highlights an inefficiency in the current attention configuration in VAR.

 

---

 

Method

 

We propose a novel approach to optimize the computational efficiency of the scale-wise autoregressive image generation models. Specifically, we retain the standard attention mechanisms for intra-scale interactions, but, importantly, employ Mamba for inter-scale interactions. The main motivation of this change is that Mamba is designed to handle long-range interactions efficiently that scales linearly with the sequence length, as opposed to the quadratic scaling of traditional attention mechanisms. This property makes Mamba particularly suitable for modeling inter-scale relationships, where the computational cost is otherwise prohibitive. We name this new method M-VAR.

Formally, as illustrated in Figure, M-VAR models inputs in the following two steps. First, given an image with multiple scales $S=[s_1, ..., s_n]$, % M-VAR aims to model both intra-scale and inter-scale representations effectively while optimizing computational. we apply the standard attention mechanism independently to each scale for capturing the fine-grained details and local dependencies: \begin{equation} S^{'}=[s^{'}_1, ...,s^{'}_n]=[Attn(C), Attn(Upsample(s_1)), ..., Attn(Upsample(s_{n-1}))] \end{equation} Here, $Attn$ represents the standard attention, which produces the intra-scale representation, and $C$ is the condition token. All attention share the same parameters but process each scale independently. This design choice ensures consistency across scales and reduces the overall model complexity. For efficient implementation, we adopt FlashAttention~\citep{flash1,flash2} to perform the intra-scale attention in parallel. After obtaining the intra-scale representations $S^{'}$, modeling the relation between different scales becomes crucial for ensuring global coherence and coarse-to-fine consistency in the generated images. However, as previously discussed, traditional attention mechanisms are computationally expensive for inter-scale interactions due to their quadratic complexity, and we adopt the Mamba model with linear complexity, as \begin{equation} S^{''} =[s^{''}_1, ...,s^{''}_n]= Mamba(Concat([s^{'}_1, ..., s^{'}_{n}])) \end{equation} By concatenating $s^{'}$ from all scales into a single sequence, Mamba efficiently processes the combined representations, capturing the essential inter-scale interactions without incurring heavy computational burden.

 

---

 

Experiment Results

 

M-VAR consistently achieves lower FID scores and higher Inception Scores (IS), indicating superior image quality and diversity. M-VAR-d24 attains an FID of 1.93 and an IS of 320.7 with 1.5 billion parameters, which already surpasses the largest VAR model, VAR-d30, with 25% fewer parameters and 14% faster inference speed. M-VAR also shows compelling scalability — our largest model, M- VAR-d32, achieves state-of-the-art performance with an FID of 1.78 and an IS of 331.2, utilizing ∼3 billion parameters.

 

Bibtex

                @article{ren2024mvardecoupledscalewiseautoregressive,
                    title={M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation}, 
                    author={Sucheng Ren and Yaodong Yu and Nataniel Ruiz and Feng Wang and Alan Yuille and Cihang Xie},
                    year={2024},
                    eprint={2411.10433},
                    archivePrefix={arXiv},
                    primaryClass={cs.CV},
                    url={https://arxiv.org/abs/2411.10433}, 
              }