FAST-AR: NVIDIA Cracks the Video Diffusion Bottleneck — 10× Faster Generation with Constant Memory

Autoregressive video diffusion has a scaling problem. As generation progresses, the KV cache grows linearly — each new frame attends to every previous frame. Latency creeps up, GPU memory balloons, and multi-minute video generation becomes practically impossible. NVIDIA Research's FAST-AR, accepted at ICML 2026, solves this with three training-free attention tricks that deliver 5–10× end-to-end speedups while keeping memory flat.

The Bottleneck: Why Autoregressive Video Gets Slower Over Time

Unlike one-shot video generation (which denoises the entire clip at once), autoregressive models generate the next segment conditioned on all previous frames. This is what enables multi-minute generation and video world models — but it creates a fundamental scaling issue: every new frame must attend to the growing history.

The authors identify three sources of redundant computation:

1. Near-duplicate cached keys across frames — most visual content persists between consecutive frames. The KV cache stores essentially the same keys over and over.
2. Slowly evolving queries/keys — many attention computations produce near-identical results frame-to-frame, wasting compute on redundant dot products.
3. Cross-attention over long prompts — for text-to-video generation, only a tiny subset of the prompt tokens are relevant to any given frame. Computing full cross-attention over the entire prompt every frame is wasteful.

FAST-AR: Three Training-Free Compressions

FAST-AR is a drop-in replacement for attention layers in autoregressive video diffusion. No fine-tuning, no retraining — just smarter attention.

1. TempCache — Temporal KV-Cache Compression

The KV cache keeps growing because identical visual content across frames gets stored as separate key-value entries. TempCache identifies near-duplicate keys using attention-derived temporal correspondence: for each current-frame query, it finds the top-1 nearest neighbor key from previous frames. Keys that track the same content are grouped, and only the most recent representative stays in the cache. Result: the cache stops growing.

2. AnnCA — Approximate Nearest-Neighbor Cross-Attention

For text-to-video, the model computes cross-attention between every latent query and every prompt token. AnnCA projects both into a shared ANN (approximate nearest neighbor) search space and keeps only prompt tokens that have a nearby latent query. A frame showing a "sunset over mountains" doesn't need to attend to prompt tokens about "city traffic." The result: cross-attention computed over a dramatically smaller, frame-relevant prompt subset.

3. AnnSA — Approximate Nearest-Neighbor Self-Attention

Self-attention in video is highly structured: pixels mostly attend to semantically related regions (sky attends to sky, objects attend to themselves across time). AnnSA assigns tokens to ANN buckets representing semantic neighborhoods and restricts each query to attend only to keys within the same bucket(s). The result is a sparse attention pattern that remains high-recall while cutting dot-products and memory bandwidth by orders of magnitude.

Results: What Changes at Scale

Why This Matters

The implications go beyond faster video generation. FAST-AR unlocks three compute regimes that were previously impractical:

• Multi-minute video generation. With constant memory and stable latency, there's no practical ceiling on generation length. Think AI-generated films, not clips.
• Real-time video world models. Autonomous systems and game engines that simulate environments frame-by-frame can now run autoregressive diffusion as the rendering backbone without runaway costs.
• Interactive neural game engines. The paper specifically calls out this use case — diffusion models that serve as the visual core of interactive experiences, generating each frame conditioned on user input and world state.

Training-Free = Immediate Applicability

A critical detail: FAST-AR is training-free. It's a drop-in replacement for the attention modules in existing autoregressive video diffusion models. No fine-tuning, no retraining, no new model checkpoints. This means every existing pipeline — CogVideo, Sora-style architectures, video world models — can adopt these techniques immediately without the compute cost of retraining.

Relevance to Delta V

Efficient video generation at scale is increasingly relevant to our work in AI engineering and agent systems. As agentic architectures incorporate visual understanding and generation — security monitoring, simulation environments, sovereign media pipelines — the ability to run video diffusion with bounded compute and memory becomes a practical requirement, not a research curiosity.

We're tracking FAST-AR alongside other training-free acceleration techniques (FlashAttention, vLLM Semantic Router's sparse attention, TimesFM's time-series compression) as part of our broader thesis: the next wave of AI capability won't come from bigger models — it'll come from making existing models dramatically more efficient at inference time.