---
title: "Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning"
authors:
  - "Vincent Abbott"
  - "Gioele Zardini"
year: 2026
venue: "arXiv:cs.LG / math.CT"
arxiv: "2604.07242"
doi: null
source: "https://arxiv.org/abs/2604.07242"
topics:
  - "category-theory"
  - "deep-learning"
  - "broadcasting"
  - "string-diagrams"
  - "neural-circuit-diagrams"
seed_rank: 933
seed_batch: "user-add-2026-08-13"
reviewed: "2026-08-13"
pool: "languages"
relevance_score: 10
lineage: categorical-deep-learning
cites:
  - title: "Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures"
    url: "https://arxiv.org/abs/2402.05424"
    year: 2024
    arxiv: "2402.05424"
  - title: "FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness"
    url: "https://arxiv.org/abs/2412.03317"
    year: 2025
    arxiv: "2412.03317"
  - title: "pyncd: A Python package for the algebraic construction, compilation, and analysis of deep learning models"
    url: "https://github.com/mit-zardini-lab/pyncd"
    year: 2026
  - title: "tsncd: A TypeScript package for rendering Neural Circuit Diagrams from serialized categorical terms"
    url: "https://github.com/mit-zardini-lab/tsncd"
    year: 2026
  - title: "Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks"
    url: "https://doi.org/10.1007/978-3-032-00686-8_1"
    year: 2025
    doi: "10.1007/978-3-032-00686-8_1"
  - title: "Lectures on Quantum Tensor Networks"
    url: "https://arxiv.org/abs/1912.10049"
    year: 2019
    arxiv: "1912.10049"
  - title: "Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges"
    url: "https://arxiv.org/abs/2104.13478"
    year: 2021
    arxiv: "2104.13478"
  - title: "Applied Compositional Thinking for Engineering"
    url: "https://bit.ly/3qQNrdR"
    year: 2024
  - title: "Named Tensor Notation"
    url: "https://arxiv.org/abs/2102.13196"
    year: 2023
    arxiv: "2102.13196"
  - title: "Categorical Foundations of Gradient-Based Learning"
    url: "https://arxiv.org/abs/2103.01931"
    year: 2022
    arxiv: "2103.01931"
  - title: "Deep Learning with Parametric Lenses"
    url: "https://arxiv.org/abs/2404.00408"
    year: 2024
    arxiv: "2404.00408"
  - title: "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning"
    url: "https://arxiv.org/abs/2307.08691"
    year: 2024
    arxiv: "2307.08691"
  - title: "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"
    url: "https://arxiv.org/abs/2205.14135"
    year: 2022
    arxiv: "2205.14135"
  - title: "DeepSeek-V3 Technical Report"
    url: "https://arxiv.org/abs/2412.19437"
    year: 2025
    arxiv: "2412.19437"
  - title: "Backprop as Functor: A compositional perspective on supervised learning"
    url: "https://arxiv.org/abs/1711.10455"
    year: 2019
    arxiv: "1711.10455"
  - title: "Representable Markov Categories and Comparison of Statistical Experiments in Categorical Probability"
    url: "https://arxiv.org/abs/2010.07416"
    year: 2023
    arxiv: "2010.07416"
    doi: "10.1016/j.tcs.2023.113896"
  - title: "Position: Categorical Deep Learning is an Algebraic Theory of All Architectures"
    url: "https://arxiv.org/abs/2402.15332"
    year: 2024
    arxiv: "2402.15332"
  - title: "AI and Memory Wall"
    url: "https://arxiv.org/abs/2403.14123"
    year: 2024
    arxiv: "2403.14123"
  - title: "Deep Learning"
    url: "https://www.deeplearningbook.org/"
    year: 2016
  - title: "Deep Residual Learning for Image Recognition"
    url: "https://arxiv.org/abs/1512.03385"
    year: 2015
    arxiv: "1512.03385"
    doi: "10.1109/CVPR.2016.90"
  - title: "The Geometry of Tensor Calculus, I"
    url: "https://doi.org/10.1016/0001-8708(91)90003-P"
    year: 1991
    doi: "10.1016/0001-8708(91)90003-P"
  - title: "ImageNet classification with deep convolutional neural networks"
    url: "https://doi.org/10.1145/3065386"
    year: 2017
    doi: "10.1145/3065386"
  - title: "Formal Algorithms for Transformers"
    url: "https://arxiv.org/abs/2207.09238"
    year: 2022
    arxiv: "2207.09238"
  - title: "An Introduction to String Diagrams for Computer Scientists"
    url: "https://www.cambridge.org/core/books/an-introduction-to-string-diagrams-for-computer-scientists/"
    year: 2025
  - title: "Categorical foundations for CuTe layouts"
    url: "https://research.colfax-intl.com/download/categories-of-layouts/"
    year: 2025
  - title: "Einops: Clear and Reliable Tensor Manipulations with Einstein-Like Notation"
    url: "https://openreview.net/forum?id=oapKSVM2bcj"
    year: 2022
  - title: "A Survey of Graphical Languages for Monoidal Categories"
    url: "https://arxiv.org/abs/0908.3347"
    year: 2010
    arxiv: "0908.3347"
    doi: "10.1007/978-3-642-12821-9_4"
  - title: "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision"
    url: "https://arxiv.org/abs/2407.08691"
    year: 2024
    arxiv: "2407.08691"
  - title: "GLU variants improve transformer"
    url: "https://arxiv.org/abs/2002.05202"
    year: 2020
    arxiv: "2002.05202"
  - title: "Category Theory in Machine Learning"
    url: "https://arxiv.org/abs/2106.07032"
    year: 2021
    arxiv: "2106.07032"
  - title: "Attention Is All You Need"
    url: "https://arxiv.org/abs/1706.03762"
    year: 2017
    arxiv: "1706.03762"
  - title: "Graph Attention Networks"
    url: "https://arxiv.org/abs/1710.10903"
    year: 2018
    arxiv: "1710.10903"
  - title: "String diagrams for strictification and coherence"
    url: "https://lmcs.episciences.org/13982"
    year: 2024
    doi: "10.46298/lmcs-20(4:8)2024"
  - title: "FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling"
    url: "https://arxiv.org/abs/2603.05451"
    year: 2026
    arxiv: "2603.05451"
  - title: "Co-Design of Complex Systems: From Autonomy to Future Mobility Systems"
    url: "https://doi.org/10.3929/ethz-b-000620391"
    year: 2023
    doi: "10.3929/ethz-b-000620391"
see:
  - "934-neural-circuit-diagrams"
  - "935-flashattention-on-a-napkin"
  - "936-accelerating-ml-systems-via-category-theory"
  - "937-lectures-on-quantum-tensor-networks"
  - "938-geometric-deep-learning-grids-groups-graphs-geodesics"
  - "960-applied-compositional-thinking-for-engineering"
  - "939-named-tensor-notation"
  - "940-categorical-foundations-of-gradient-based-learning"
  - "941-deep-learning-with-parametric-lenses"
  - "080-flashattention-2-faster-attention-with-better-parallelism-an"
  - "089-flashattention-fast-and-memory-efficient-exact-attention-wit"
  - "942-deepseek-v3-technical-report"
  - "943-backprop-as-functor-compositional-supervised-learning"
  - "944-representable-markov-categories-statistical-experiments"
  - "945-categorical-deep-learning-algebraic-theory-of-architectures"
  - "946-ai-and-memory-wall"
  - "961-deep-learning"
  - "947-deep-residual-learning-for-image-recognition"
  - "948-the-geometry-of-tensor-calculus-i"
  - "949-imagenet-classification-with-deep-convolutional-neural-nets"
  - "950-formal-algorithms-for-transformers"
  - "951-an-introduction-to-string-diagrams-for-computer-scientists"
  - "952-categorical-foundations-for-cute-layouts"
  - "953-einops-clear-and-reliable-tensor-manipulations"
  - "117-a-survey-of-graphical-languages-for-monoidal-categories"
  - "073-flashattention-3-fast-and-accurate-attention-with-asynchrony"
  - "954-glu-variants-improve-transformer"
  - "955-category-theory-in-machine-learning"
  - "013-attention-is-all-you-need"
  - "956-graph-attention-networks"
  - "957-string-diagrams-for-strictification-and-coherence"
  - "958-flashattention-4"
  - "959-co-design-of-complex-systems-zardini-phd"
---

# Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning

## One-sentence takeaway

A categorical algebra for deep-learning architectures: broadcasting is an axis-stride / array-broadcasted category `[B;A]` over BorelStoch with affine indexes, Para hides weights, and weaves distinguish tiled vs target axes — implemented in mirrored Python (`pyncd`) and TypeScript (`tsncd`), with DeepSeek-V3 as the worked example.

## Why it matters here

This is the typed-array / notation-as-IR paper Broadside has been missing next to rank polymorphism. Neural circuit diagrams are a compositional blueprint language (human diagrams ↔ machine terms) in the same spirit as ano: an architecture is a term, not a pile of `view`/`expand` comments. Dual Python/TypeScript implementations are the kind of portable IR Anoptic wants if GRID COMMAND ever grows a typed tensor surface for agents.

## Key ideas

- Ad-hoc notation and pseudocode fail on *nonlinear broadcasting* and on the component-vs-composed-model relationship. The fix is a constructed term system with sound interpretation into a mathematical category.
- **Axis-stride** and **array-broadcasted** categories: objects carry axes; morphisms reindex. The structured form is `[B;A]` — BorelStoch (`B`, standard Borel spaces and measurable Markov kernels) for datatypes/probability, affine indexing category `A` for strides.
- **Weaves** (and integrated reindexings) separate *tiled* axes from *target* axes, which is how FlashAttention-style tiling and convolution/MoE broadcasting become the same algebra rather than special cases.
- **Para(`[B;A]`)** (Fong–Spivak–Tuyéras) wraps the symmetric monoidal category so learned weights are hidden parameters rather than extra wires.
- Operations recovered uniformly: Einstein (einsum) products, dropout/elementwise, embeddings, MoE, convolution.
- **DeepSeek-V3** is the large worked example (MLA + DeepSeekMoE expressed as terms). The stack then: algebraic construction → hypergraph ↔ diagram → PyTorch compilation (`pyncd`) and diagram rendering (`tsncd`).
- Continues Neural Circuit Diagrams (TMLR 2024) and *FlashAttention on a Napkin* (TMLR 2025): diagrams that are also a calculus.

## Caveats


## Links

- arXiv: [2604.07242](https://arxiv.org/abs/2604.07242)
- PDF: https://arxiv.org/pdf/2604.07242
- HTML: https://arxiv.org/html/2604.07242
- pyncd: https://github.com/mit-zardini-lab/pyncd
- tsncd: https://github.com/mit-zardini-lab/tsncd