Inside the transformer

Almost every language model in production is a decoder-only transformer — the architecture introduced in Attention Is All You Need (2017), with the encoder half discarded. Its dominance is not because it is the most intelligent design anyone could imagine; it is because it parallelises across a sequence during training, which is what let the field spend enormous compute usefully.

The path of a token

1. Tokenise. Text becomes integer IDs via a subword vocabulary, typically byte-pair encoding, usually 30k–200k entries.
2. Embed. Each ID indexes a learned vector of width d_model (thousands, in a large model).
3. Position. Attention is permutation-invariant, so order must be injected. Modern models mostly use rotary position embeddings (RoPE), which rotate the query and key vectors by an angle proportional to position, encoding relative distance rather than absolute index.
4. N identical blocks. Each contains attention and a feed-forward network, each wrapped in a residual connection and normalisation.
5. Project to vocabulary. A final linear layer produces one logit per vocabulary entry; softmax turns those into the next-token distribution.

What one block does

x = x + attention(norm(x))        # mix information across positions
x = x + feedforward(norm(x))      # process each position independently

That is the entire block, and the division of labour is the thing to remember: attention moves information between positions; the feed-forward network transforms it in place. A model with attention but no MLP could route but not compute; with MLP but no attention it could not see other tokens at all.

Why pre-norm

Normalisation sits inside the residual branch (x + f(norm(x))) rather than after it. This keeps a clean, unobstructed residual path from input to output, which is what makes very deep stacks trainable without careful warmup schedules. Post-norm transformers of 80 layers are painful to train; pre-norm ones are routine. RMSNorm — normalising by root-mean-square without re-centring — is the common choice, being cheaper and empirically as good.

The feed-forward network is where the parameters live

Typically two-thirds of a dense model's parameters sit in the MLPs. The standard form expands to roughly 4× d_model, applies a non-linearity, and projects back; gated variants (SwiGLU) use two parallel projections, one gating the other, and are now near-universal.

Mixture of experts

Instead of one MLP per block, hold many "experts" and route each token to a small number of them. Parameter count rises enormously while the compute per token stays roughly constant, because most experts are inactive for any given token. The costs are real: all experts must be resident in memory, routing can become unbalanced, and training is less stable. Several frontier models are believed to be sparse in this way.

Counting parameters and compute

Two approximations worth memorising:

• Parameters per block ≈ 12 × d_model² for a dense transformer with a 4× MLP.
• Training compute ≈ 6 × N × D FLOPs, for N parameters and D training tokens — roughly 2 for the forward pass and 4 for the backward.

The second gives you a back-of-envelope for any published training run, and it is the foundation of the scaling-law lesson later in this track.

Try it yourself

Take d_model = 4096, 32 layers, vocabulary 128k. Estimate the parameter count including embeddings, then estimate the FLOPs to train it on 2 trillion tokens. Convert that to GPU-hours at a plausible 400 teraFLOP/s sustained. The number you get is why frontier training is a capital-expenditure decision.