Tutorial
Build a transformer from scratch
Assemble a small decoder-only transformer and trace one complete forward, backward, and Adam update.
Read guideLearn
Build a working mental model from tensors and gradients through a complete decoder-only transformer.
Concept
Tensor Milestone
Understand contiguous storage, shapes, strides, scalar tensors, indexing, reshape, and backend transfer.
Read guideConcept
Tensor Operations
Learn the checked tensor-operation contracts, broadcasting rules, reductions, and relationship to autograd.
Read guideConcept
Autograd Milestone
Follow reverse-mode automatic differentiation from graph construction through gradient accumulation and custom gradients.
Read guideConcept
Neural-Network Primitives and Feed-Forward Layer
Understand embeddings, linear layers, normalization, activations, feed-forward composition, softmax, and cross-entropy.
Read guideConcept
Causal Multi-Head Self-Attention
Trace causal multi-head self-attention shapes, masking, materialized execution, and exact memory-linear execution.
Read guideConcept
Decoder-Only Transformer
Follow embeddings, pre-normalized residual blocks, final normalization, and the language-model head through a decoder-only transformer.
Read guideConcept
Training Loop
Understand how Python orchestrates seeded batches and metrics while C++ executes fresh graphs, all-position or time-range cross-entropy, backward, and Adam.
Read guide