---
title: "Formal Algorithms for Transformers"
authors:
  - "Mary Phuong"
  - "Marcus Hutter"
year: 2022
venue: "arXiv:cs.LG"
arxiv: "2207.09238"
doi: null
source: "https://arxiv.org/abs/2207.09238"
topics:
  - "transformers"
  - "formal-algorithms"
seed_rank: 950
seed_batch: "user-add-2026-08-13"
reviewed: "2026-08-13"
pool: "languages"
relevance_score: 8
lineage: categorical-deep-learning
cites:
  - title: "Attention Is All You Need"
    url: "https://arxiv.org/abs/1706.03762"
    year: 2017
    arxiv: "1706.03762"
see:
  - "013-attention-is-all-you-need"
---

# Formal Algorithms for Transformers

## One-sentence takeaway

Self-contained, mathematically precise algorithms for Transformer components and training — what the architecture is, not what the scores were.

## Why it matters here

The pseudocode Weaves wants to replace with terms. Useful as a checklist of ops any algebra must cover: attention, residuals, layer-norm, training loops, and the main variants.

## Key ideas

- arXiv:2207.09238; 16 pages, 15 algorithms.
- Covers architecture blocks, training, and prominent model variants — not empirical SOTA.
- Assumes MLP-level literacy; no category theory.
- Algorithms document, not a semantics. Complements Weaves rather than competing with it.

## Caveats

## Links

- arXiv: [2207.09238](https://arxiv.org/abs/2207.09238)
- PDF: https://arxiv.org/pdf/2207.09238