---
title: "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models"
authors:
  - "Xin Cheng"
  - "Rui Tian"
  - "Wangding Zeng"
  - "Damai Dai"
  - "Qinyu Chen"
  - "Bingxuan Wang"
  - "Zhenda Xie"
  - "Kezhao Huang"
  - "Xingkai Yu"
  - "et al."
year: 2026
venue: "arXiv:cs.CL"
arxiv: "2601.07372"
doi: null
source: "https://arxiv.org/abs/2601.07372"
topics:
  - "engram"
  - "conditional-memory"
  - "moe"
  - "deepseek"
seed_rank: 997
seed_batch: "deepseek-lab-2026-08-13"
reviewed: "2026-08-13"
pool: "agents"
relevance_score: 9
lineage: deepseek
cites:
  - title: "Attention Is All You Need"
    url: "https://arxiv.org/abs/1706.03762"
    year: 2017
    arxiv: "1706.03762"
  - title: "DeepSeek-V3 Technical Report"
    url: "https://arxiv.org/abs/2412.19437"
    year: 2025
    arxiv: "2412.19437"
  - title: "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models"
    url: "https://arxiv.org/abs/2401.06066"
    year: 2024
    arxiv: "2401.06066"
  - title: "mHC: Manifold-Constrained Hyper-Connections"
    url: "https://arxiv.org/abs/2512.24880"
    year: 2025
    arxiv: "2512.24880"
see:
  - "013-attention-is-all-you-need"
  - "942-deepseek-v3-technical-report"
  - "975-deepseekmoe-towards-ultimate-expert-specialization"
  - "996-mhc-manifold-constrained-hyper-connections"
---

# Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

## One-sentence takeaway

Engram is O(1) hashed N-gram memory as a sparsity axis complementary to MoE: a U-shaped allocation law (~20–25% memory / 75–80% MoE), iso-FLOP wins on knowledge and reasoning, and host-RAM offload of 100B tables at <3% throughput hit.

## Why it matters here

Conditional memory vs conditional compute. Agent-relevant as a native lookup primitive so the Transformer stops simulating a dictionary in early layers — the thing Anoptic/GRID would rather have as a table than as attention.

## Key ideas

- arXiv:2601.07372. Tokenizer-compressed suffix N-grams → multi-head hash into huge embedding tables → context-aware gate from hidden state → depthwise conv; residual into selected layers.
- Sparsity allocation \(\rho\) (fraction of inactive params kept as MoE experts): U-shape, optimum ~75–80% MoE / 20–25% Engram under iso-param/iso-FLOP.
- Engram-27B (reallocate 72→55 routed experts into 5.7B memory) beats MoE-27B on MMLU +3, BBH +5, HumanEval +3, MATH +2.4; long-context NIAH MQ 97 vs 84.
- Deterministic IDs → prefetch from host DRAM; 100B table offload ~2–3% throughput. Integrates with mHC (M=4). Research 27B/40B, not yet a V4 product module.

## Caveats

## Links

- arXiv: [2601.07372](https://arxiv.org/abs/2601.07372)
- GitHub: https://github.com/deepseek-ai/Engram