03 / Resources/Deep Learning · Transformers

Interactive Diagnostic Essay · Topic 16

The Transformer: Attention Is All You Need

A complete first-principles architectural breakdown: from subword Byte-Pair Encoding and 2D semantic token spaces, to sinusoidal positional encoding, two-layer feed-forward expansion, residual highways, 6-layer dual towers, and final softmax temperature sampling.

Read Time: 16 min•Vaswani et al. (NeurIPS 2017)•AIS Research Cohort

01 / Tokenization & Vocabulary

Byte-Pair Encoding (BPE) & Subword Representation

Transformers cannot ingest raw characters or arbitrary-length string words directly. Word-level vocabularies suffer from out-of-vocabulary (OOV) tokens, while character-level tokenization inflates sequence length by 10× (and self-attention scales quadratically $O(N^2)$ ). Byte-Pair Encoding (BPE) merges frequent character pairs into morphemic subwords:

SIMPLIFIED SUBWORD TOKENIZER

0 TOKENS EXTRACTED

Simplified demo: words longer than 7 characters are split after the 5th character and IDs are hashed into a 37,000-entry range. A real BPE tokenizer learns its merges from corpus statistics (the original paper used a shared ~37k-token BPE vocabulary for English–German).

02 / Vector Geometry

Tokens on the 2D Semantic Plane & Cosine Angular Geometry

Each token ID is projected into a dense continuous embedding space $d_{model} = 512$ . High-dimensional embeddings organize semantic meaning geometrically: tokens with similar contextual roles cluster together. On the 2D semantic plane below, observe semantic cluster hulls, select token pairs, and inspect vector alignment $\cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|}$ :

2D SEMANTIC PLANE SIMULATOR ·

\mathbb{R}^{512} \to \mathbb{R}^2

INTERACTIVE 2D VECTOR SPACE · EMBEDDING GEOMETRY

VECTOR A:

VECTOR B:

COSINE SIMILARITY: 0.999

\frac{u \cdot v}{\|u\| \|v\|}

03 / Harmonic Geometry

Sinusoidal Positional Encoding & Frequency Harmonics

Because self-attention operates across all positions in parallel with zero recurrence, it has no built-in notion of order: it is permutation-equivariant, so shuffling the input words simply shuffles the outputs the same way. Vaswani et al. solved this by adding sine and cosine waves of different frequencies directly into the embedding vectors:

PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)

HARMONIC SINE (CHARCOAL) & COSINE (TERRACOTTA) SPECTRUM

POSITIONS:20 POS

Dimension pairs 2i = 0, 32, 64, 128 (darker = lower dimension = higher frequency)Wavelengths grow from 2π (2i = 0) to ≈10,000·2π (2i = 510)

04 / Attention Mechanics

Scaled Dot-Product & Multi-Head Projections

Queries ( $Q$ ), Keys ( $K$ ), and Values ( $V$ ) are computed via learned projections. Scaling by $1/\sqrt{d_k}$ prevents softmax gradients from vanishing for large vector dimensions:

Scaled Dot-Product Attention

Scrollable →

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Multi-Head Projections ( $h=8$ )

Scrollable →

\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)W^O

INTERACTIVE COREFERENCE ATTENTION RESOLUTION

The:

2.0%

animal:

76.0%

didn't:

1.0%

cross:

3.0%

the:

1.0%

street:

4.0%

because:

2.0%

it:

5.0%

was:

2.0%

too:

1.0%

tired:

3.0%

05 / Deep Layers

Two-Layer Position-Wise Feed-Forward Networks as Associative Memories

While self-attention connects different tokens across sequence space, the Feed-Forward Network (FFN) processes each token independently. It projects the embedding dimension 4× from $d_{model} = 512$ up to $d_{ff} = 2048$ via $W_1$ with non-linear activation (ReLU or GELU), and contracts back down to $d_{model} = 512$ via $W_2$ :

\mathrm{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2 = \mathrm{ReLU}(xW_1 + b_1)W_2 + b_2

TWO-LAYER NEURAL ARCHITECTURE · W₁ EXPANSION & W₂ PROJECTION

2/3 OF MODEL WEIGHTS LIVE IN FFN

ACTIVATION:

INPUT MAGNITUDE (x):1.4x

DIMENSIONAL PROJECTION512 → 2048 → 512 (4×)

ACTIVE NEURONS (FIRING)460 / 2048 Neurons Active

RELU SPARSITY77.5% output exactly 0

Toy layer: a scalar input x feeds 2,048 hidden neurons with fixed random weights and slightly negative biases. ReLU zeroes every negative pre-activation exactly; GELU lets small negative values through, so it never produces exact zeros. The 13 drawn nodes are the first 13 of the 2,048.

06 / Stability & Backprop

Residual Connections & Layer Normalization Gradient Highways

Without residual skip connections, gradients vanish exponentially in deep architectures. By formulating each sub-layer as $x + \mathrm{Sublayer}(x)$ , the gradient derivative contains an unbroken identity term $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} \cdot \left(1 + \frac{\partial \mathrm{Sublayer}}{\partial x}\right)$ :

\mathrm{Output} = \mathrm{LayerNorm}\left(x + \mathrm{Sublayer}(x)\right) = \left[\frac{(x + \mathrm{Sublayer}(x)) - \mu}{\sqrt{\sigma^2 + \epsilon}}\right] \odot \gamma + \beta

SIGNAL RETENTION THROUGH 6 DEEP LAYERS

LAYER 1:

100%

LAYER 2:

99.2%

LAYER 3:

98.4%

LAYER 4:

97.6%

LAYER 5:

96.8%

LAYER 6:

96%

✓ Gradient Highway Active: the identity skip path keeps the signal close to full strength across all 6 layers.

07 / Canonical Paper Architecture

The Transformer: Vaswani et al. (2017) Dual-Tower Blueprint

The authentic architecture from “Attention Is All You Need” (Vaswani et al., 2017) consists of an Encoder stack that creates rich contextual embeddings and an Autoregressive Decoder stack that consumes target prefix tokens and cross-attends into the encoder states:

INTERACTIVE LAYER INSPECTOR

encoder TOWERX_enc ∈ ℝ^{N × 512} (h=8, d_k=64)

Encoder Multi-Head Attention

\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)W^O

All tokens in the source sequence attend to all other tokens bidirectionally. Every token integrates global context from across the entire prompt simultaneously.

Hyperparameters (Base Vaswani Model)

• Layers: N = 6

• d_model: 512

• Heads: h = 8

• d_k = d_v: 64

• d_ff: 2048

• Vocab: 37,000

08 / Generation Dynamics & Memory

Attention Matrix, Causal Masking & Incremental KV Cache

In autoregressive generation, each token attends only to prior positions. Naive inference re-computes all previous representations at each step in $O(N^2)$ time. The KV Cache eliminates redundant projections by storing past Key and Value matrices in GPU VRAM, turning each decoding step into an $O(N)$ operation:

AUTOREGRESSIVE GENERATION SEQUENCESTEP 3 OF 6

Context Window (Input to Decoder)

<BOS>Attentionis? → Next Token

Top Softmax Predictions

“all”97%

“a”2%

“the”1%

ATTENTION MATRIX, CAUSAL MASK & KV CACHE STUDIO

Click any cell $(i, j)$ in the 5×5 attention matrix to inspect how Query token $q_i$ takes the dot-product with Key token $k_j$ , scales by $1/\sqrt{d_k}$ ( $1/\sqrt{64} = 1/8$ ), and produces the final softmax attention weight:

Q \ K

The

neural

network

learns

fast

The

neural

network

learns

fast

CELL (2, 1) BREAKDOWNACTIVE ATTENTION

Query Token (i):“network” (row 2)

Key Token (j):“neural” (col 1)

Raw Dot Product

q_i \cdot k_j

:28.0

Scale Factor

\sqrt{d_k} = \sqrt{64}

:8.0

Scaled Score

S_{ij} / \sqrt{d_k}

:3.50

Softmax Weight

A_{ij}

:26.37%

Token "network" directs 26.4% of its attention context towards "neural".

09 / Output Head

Final Linear Projection, Softmax Probabilities & Temperature

The top decoder output vector $h_{final} \in \mathbb{R}^{d_{model}}$ is projected by a final linear weight matrix $W_v \in \mathbb{R}^{d_{model} \times |V|}$ into raw vocabulary logits $z \in \mathbb{R}^{|V|}$ . A calibrated softmax with Temperature ( $T$ ) converts logits into normalized probability distribution $P(w_i)$ :

P(w_i) = \frac{\exp\left(z_i / T\right)}{\sum_{j=1}^{|V|} \exp\left(z_j / T\right)}

INTERACTIVE SOFTMAX TEMPERATURE & TOP-K STUDIO

TEMPERATURE (T):0.80

0.1 (Greedy / Deterministic)1.0 (Standard)2.0 (High Entropy / Creative)

CANDIDATE TOKEN (LOGIT z_i)SOFTMAX PROBABILITY P(w_i)

“transduction”(z=6.8)

75.6%

“modeling”(z=5.4)

13.1%

“generation”(z=4.9)

7.0%

“translation”(z=4.2)

2.9%

“synthesis”(z=3.1)

0.7%

“learning”(z=2.4)

0.3%

“prediction”(z=1.8)

0.1%

“recognition”(z=0.9)

0.0%

💡 Notice: Lowering temperature to T=0.2 collapses entropy onto the highest logit (“transduction”), while raising to T=1.8 flattens the distribution across all candidates.

Byte-Pair Encoding (BPE) & Subword Representation

Tokens on the 2D Semantic Plane & Cosine Angular Geometry

Sinusoidal Positional Encoding & Frequency Harmonics

Scaled Dot-Product & Multi-Head Projections

Scaled Dot-Product Attention

Multi-Head Projections (h=8h=8h=8)

Two-Layer Position-Wise Feed-Forward Networks as Associative Memories

Residual Connections & Layer Normalization Gradient Highways

The Transformer: Vaswani et al. (2017) Dual-Tower Blueprint

Encoder Multi-Head Attention

Attention Matrix, Causal Masking & Incremental KV Cache

Final Linear Projection, Softmax Probabilities & Temperature

Multi-Head Projections ( $h=8$ )