---
title: "Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention"
authors:
  - "Jingyang Yuan"
  - "Huazuo Gao"
  - "Damai Dai"
  - "Junyu Luo"
  - "Liang Zhao"
  - "Zhengyan Zhang"
  - "Zhenda Xie"
  - "Y. X. Wei"
  - "Lean Wang"
  - "Zhiping Xiao"
  - "Yuqing Wang"
  - "Chong Ruan"
  - "Ming Zhang"
  - "Wenfeng Liang"
  - "Wangding Zeng"
year: 2025
venue: "ACL 2025 (Best Paper)"
arxiv: "2502.11089"
doi: null
source: "https://arxiv.org/abs/2502.11089"
topics:
  - "sparse-attention"
  - "long-context"
  - "kernels"
  - "deepseek"
seed_rank: 990
seed_batch: "deepseek-lab-2026-08-13"
reviewed: "2026-08-13"
pool: "agents"
relevance_score: 10
lineage: deepseek
cites:
  - title: "Attention Is All You Need"
    url: "https://arxiv.org/abs/1706.03762"
    year: 2017
    arxiv: "1706.03762"
  - title: "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"
    url: "https://arxiv.org/abs/2205.14135"
    year: 2022
    arxiv: "2205.14135"
  - title: "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning"
    url: "https://arxiv.org/abs/2307.08691"
    year: 2023
    arxiv: "2307.08691"
  - title: "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision"
    url: "https://arxiv.org/abs/2407.08691"
    year: 2024
    arxiv: "2407.08691"
  - title: "FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling"
    url: "https://arxiv.org/abs/2603.05451"
    year: 2026
    arxiv: "2603.05451"
  - title: "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models"
    url: "https://arxiv.org/abs/2401.06066"
    year: 2024
    arxiv: "2401.06066"
  - title: "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning"
    url: "https://arxiv.org/abs/2501.12948"
    year: 2025
    arxiv: "2501.12948"
see:
  - "013-attention-is-all-you-need"
  - "089-flashattention-fast-and-memory-efficient-exact-attention-wit"
  - "080-flashattention-2-faster-attention-with-better-parallelism-an"
  - "073-flashattention-3-fast-and-accurate-attention-with-asynchrony"
  - "958-flashattention-4"
  - "975-deepseekmoe-towards-ultimate-expert-specialization"
  - "988-deepseek-r1-incentivizing-reasoning-via-rl"
---

# Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

## One-sentence takeaway

NSA is trainable hierarchical sparse attention — compressed block tokens, top-n selected blocks, and a sliding window, gated per query — implemented with Triton kernels aligned to GQA/MQA so a head group shares sparse KV; at 27B it matches or beats full attention and is up to 9×/6× faster than FA-2 Triton at 64K.

## Why it matters here

Sparse-attention ancestor of V3.2's DSA and V4's CSA/HCA. Hardware-aligned the same way FlashAttention is: cite the FA line and don't remint kernels. Long-context GRID agents need this, not naive scatter sparsity.

## Key ideas

- arXiv:2502.11089. ACL 2025 Best Paper. First author Yuan: PKU internship at DeepSeek-AI; Liang Wenfeng coauthor.
- Three gated branches per query; natively trained (not post-hoc inference sparsity). 27B GQA+DeepSeekMoE, 270B tokens at 8K then YaRN to 32K.
- Matches/exceeds full attention on general, LongBench, and AIME-distill; decode memory traffic ~11.6× less at 64K.
- Requires custom kernels; naive sparse attention will not hit the reported speedups. V3.2's DSA is the production cousin, not a drop-in NSA.

## Caveats

## Links

- arXiv: [2502.11089](https://arxiv.org/abs/2502.11089)