Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers

Sucheng Ren¹, Qihang Yu², Ju He², Alan Yuille¹, Liang-Chieh Chen²,

¹ Johns Hopkins University

² Independent Researcher

Using HunyuanVideo for text-to-video generation, GRAT delivers a 15.8× improvement in attention efficiency over full attention and generates a 5-second 720p video 3.8× faster.
Full Attention	GRAT

🔥 Highlights

The majority of attention mass is concentrated in a query token's local spatial neighborhood.

Preserving a small set of long-range dependencies is sufficient to maintain large receptive fields and global coherence.

Abstract

Diffusion-based Transformers have demonstrated impressive generative capabilities, but their high computational costs hinder practical deployment—for example, generating an $8192\times8192$ image can take over an hour on an A100 GPU. In this work, we propose GRAT (GRouping first, ATtending smartly), a training-free attention acceleration strategy for fast image and video generation without compromising output quality. The key insight is to exploit the inherent sparsity in learned attention maps (which tend to be locally focused) in pretrained Diffusion Transformers and leverage better GPU parallelism. Specifically, GRAT first partitions contiguous tokens into non-overlapping groups, aligning both with GPU execution patterns and the local attention structures learned in pretrained generative Transformers. It then accelerates attention by having all query tokens within the same group share a common set of attendable key and value tokens. These key and value tokens are further restricted to structured regions, such as surrounding blocks or criss-cross regions, significantly reducing computational overhead (e.g., attaining a 35.8$\times$ speedup over full attention when generating $8192\times8192$ images) while preserving essential attention patterns and long-range context. We validate GRAT on pretrained Flux and HunyuanVideo for image and video generation, respectively. In both cases, GRAT achieves substantially faster inference without any fine-tuning, while maintaining the performance of full attention. We hope GRAT will inspire future research on accelerating Diffusion Transformers for scalable visual generation.

GRAT

Query tokens are shown in red, and their corresponding attended regions (key and value tokens) are highlighted in light blue. (a) Full Attention: each query attends to the entire feature map. (b) Neighborhood Attention: each query attends only to its local spatial neighborhood. (c) Criss-cross Attention: each query attends to tokens in the same row and column. (d–f) In this work, we propose GRAT (GRouping first, ATtending smartly), which first partitions the feature map into non-overlapping groups (d) (each group has $2 \times 2$ tokens in this example). Query tokens within the same group share a common set of key and value tokens, which are restricted to structured regions—such as surrounding blocks (e) or criss-cross patterns (f).

Comparison of Attention Schemes

The comparison is based on Flux with various attention mechanisms, including Full Attention, Neighborhood Attention, and the proposed GRAT-B and GRAT-X. FLOPs Sparsity measures the theoretical reduction in compute relative to Full Attention (0\% indicates no reduction). Inference Speedup reflects real-world speedup on an A100 GPU, relative to Full Attention (1$\times$ means no speedup). Farthest Token Distance denotes the maximum distance over which a query can attend—representing the effective receptive field. As shown, GRAT-B achieves the same FLOPs sparsity as Neighborhood Attention (NA) but delivers higher inference speedup. Conversely, GRAT-X maintains comparable speedup while offering a much larger receptive field. Both variants outperform NA in GenEval scores, with GRAT-X notably matching the quality of Full Attention while running 12$\times$ faster.

Experiment Results

Visualization

Videos

Bibtex

                @article{ren2025grat,
                        title={Grouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers},
                        author={Ren, Sucheng and Yu, Qihang and He, Ju and Yuille, Alan and Chen, Liang-Chieh},
                        journal={arXiv preprint arXiv:2505.14687},
                        year={2025}}