HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

Dongyeun Lee1, Amir Zandieh2, Vahab Mirrokni2, Junmo Kim1,†, Insu Han1,†
1KAIST   2Google Research
†Corresponding authors
NeurIPS 2026
Paper arXiv Code
Comparison of HunyuanVideo full attention and HyperVAttention on a generated mammoth video. HyperVAttention reduces latency from 27.00 seconds to 12.41 seconds while preserving visual quality.
HyperVAttention is a training-free sparse attention framework for video diffusion that achieves 2.13× speedup on HunyuanVideo while preserving the visual fidelity of full attention.

Abstract

Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation. However, their ability to produce long-duration videos is fundamentally constrained by the quadratic complexity of the self-attention mechanism. Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks: substantial clustering overhead and low CTA utilization caused by irregular cluster-induced blocks. We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly. To reduce clustering overhead, we introduce 3D local-window clustering, which exploits the spatio-temporal locality of video tokens to restrict centroid search to fixed local neighborhoods, and implement it with a custom Triton kernel for efficient execution. We further propose a hybrid clustering strategy that performs full clustering only at anchor steps and updates only subset tokens at intermediate steps, leveraging the temporal stability of cluster assignments across denoising steps. To improve CTA utilization, we present hardware-aware cluster merging that minimizes CTA-aligned execution cost through parallel agglomerative merging, improving block density and approximation fidelity by utilizing idle tile capacity. Together, these components reduce clustering overhead, avoid redundant updates, and better align sparse attention with the fixed tile structure of modern GPU kernels. Experiments on Text-to-Video generation show that HVA establishes a new Pareto frontier for training-free sparse attention in video diffusion, reducing end-to-end latency by up to 2.13× while improving fidelity over existing training-free sparse attention baselines.

HyperVAttention

Comparison of clustering-based sparse attention before and after hardware-aware cluster merging. Cluster merging fills unused CTA tile capacity and reduces thread underutilization.
Figure: Illustration of how existing clustering-based sparse attention methods lead to thread underutilization. (a) These methods form cluster-induced blocks, whose irregular sizes often misalign with fixed CTA tiles and leave padded regions idle. (b) Merging similar clusters better matches cluster-induced blocks to CTA tiles, reducing tiled execution cost. Darker regions show additional tokens placed in unused tile capacity, improving approximation fidelity without increasing CTA tiles.

3D Local-Window Clustering

Existing clustering-based sparse attention methods often rely on generic global procedures such as k-means, which compare every token against every centroid without exploiting the spatio-temporal locality of video tokens. As a result, clustering itself becomes a major computational bottleneck, even when highly optimized kernels are used. Our 3D local-window clustering instead restricts centroid search to fixed local neighborhoods by leveraging the intrinsic spatio-temporal structure of video tokens. Compared with generic global clustering, this local search converges faster and has lower complexity. We further implement the procedure with a custom Triton kernel to reduce clustering overhead and enable efficient execution.

Hybrid Clustering across Denoising Steps

Existing approaches also overlook the temporal stability of token assignments across adjacent denoising steps. This leads to redundant recomputation because they repeatedly update all token assignments even though most remain unchanged over time. Our hybrid clustering strategy exploits this stability by performing full clustering only at designated anchor steps. At intermediate steps, it updates only a subset of tokens rather than clustering every token again. Combining infrequent full clustering with partial intermediate updates avoids redundant work and further reduces the clustering overhead accumulated throughout the denoising process.

Hardware-Aware Cluster Merging

Clustering-based sparse attention uses token clusters as logical attention blocks, but these irregular blocks are executed after being padded to the fixed CTA tile sizes of modern attention kernels. When cluster sizes do not align with the CTA tile shape, threads assigned to padded regions remain underutilized, preventing theoretical sparsity from fully translating into actual speedup. Hardware-aware cluster merging addresses this mismatch by formulating tiled execution cost as a hardware cost minimization problem and approximately solving it with efficient parallel agglomerative merging. Merging clusters to reduce this cost improves CTA utilization and latency. It also places additional tokens in otherwise unused tile capacity, enabling a more accurate attention approximation without increasing the number of CTA tiles.

Video Results

Wan 2.2 Image-to-Video

Wan 2.2 Text-to-Video

Results

Quality–Efficiency Comparison

Table comparing the quality and efficiency of Full Attention, SpargeAttn, SVG, SVG2, HVA, and HVA-Flash on Wan 2.2 and HunyuanVideo.

Across Wan 2.2 image-to-video, Wan 2.2 text-to-video, and HunyuanVideo text-to-video, HVA achieves the strongest fidelity to full attention among the evaluated sparse methods. On HunyuanVideo, HVA reaches 33.97 PSNR and 0.046 LPIPS, compared with 32.66 PSNR and 0.056 LPIPS for SVG2. HVA-Flash prioritizes inference speed while retaining competitive fidelity, reaching up to 1.86× speedup on Wan 2.2 and 2.23× on HunyuanVideo.

Efficiency Analysis

Efficiency analysis showing end-to-end latency breakdown, clustering error versus latency, and theoretical versus CTA density before and after cluster merging.

The latency breakdown shows that hybrid clustering with 3D local-window clustering reduces clustering time from 83 s for SVG2 to 26 s. With 10 iterations, our clustering reaches an error comparable to 50 iterations of k-means while reducing clustering latency by 6.29×. Hardware-aware cluster merging also raises theoretical density from 39.70% to 42.24% within idle tile capacity while lowering actual CTA density from 50.42% to 46.88%, a 7.02% relative reduction.

BibTeX

@misc{lee2026hypervattention,
  title         = {HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion},
  author        = {Dongyeun Lee and Amir Zandieh and Vahab Mirrokni and Junmo Kim and Insu Han},
  year          = {2026},
  eprint        = {2607.03012},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2607.03012}
}