AI Notes

Transformers - Revolutionary Attention Architecture

Master transformer architecture for AI. Learn attention mechanisms, self-attention, multi-head attention, BERT, GPT. Complete guide 2024.

Why Transformers Replaced RNNs

Before 2017, sequence modeling was dominated by RNNs and LSTMs. These models process tokens sequentially—each timestep depends on the previous—creating two fundamental problems: they cannot be parallelized during training (each step waits for the prior), and they struggle with very long-range dependencies despite gating mechanisms.

The Transformer architecture, introduced in "Attention Is All You Need" (Vaswani et al., 2017), eliminated recurrence entirely. Instead of processing sequences step-by-step, Transformers use self-attention to relate every position to every other position in constant number of operations. This enables massive parallelization and captures dependencies regardless of distance.

The impact was immediate and transformative: Transformers now power virtually all state-of-the-art models in NLP (BERT, GPT, T5), computer vision (ViT, DINO), speech (Whisper), protein folding (AlphaFold), and more.

Self-Attention: The Core Mechanism

Intuition

Consider translating "The animal didn't cross the street because it was too tired." What does "it" refer to? A human instantly knows "it" = "animal" (because animals get tired). Self-attention lets the model learn this by having each token compute relevance scores with every other token.

Scaled Dot-Product Attention

Q = X · W_Q (Queries	what am I looking for?)
K = X · W_K (Keys	what do I contain?)
V = X · W_V (Values	what information do I provide?)

Numerical Example

Sentence	"cat sat mat" (3 tokens, d=4 for simplicity)
"cat" attending	[1·1+0·1+1·0+0·0, 1·0+0·0+1·1+0·1, 1·1+0·0+1·1+0·0]
= [1, 1, 2]	"cat" attends most to "mat"
After softmax (÷√4=2)	weights = softmax([0.5, 0.5, 1.0])

Why Scale by √d_k?

Without scaling, dot products grow with dimension d_k, pushing softmax into regions with tiny gradients:

d_k = 64	dot products have variance ~64
softmax([32, -5, 2, ...])	[≈1, ≈0, ≈0, ...] (saturated)
After scaling: softmax([4, -0.6, 0.25, ...])	[0.7, 0.03, 0.08, ...]
Softer distribution	meaningful gradients

Multi-Head Attention

Instead of one attention function, use multiple "heads" that each attend to different aspects:

Each head	d_k = d_v = 512/8 = 64
Head 1	syntactic dependencies (subject-verb)
Head 2	positional proximity
Head 3	semantic similarity
Head 4	coreference resolution

Complete Transformer Architecture

Encoder Block

Input embeddings + Positional Encoding ↓ ┌────────────────────────────────┐ │ Multi-Head Self-Attention │ │ Add & LayerNorm (residual) │ │ Feed-Forward Network │ │ FFN(x) = ReLU(xW₁+b₁)W₂+b₂│ │ (d_ff = 2048, inner expand) │ │ Add & LayerNorm (residual) │ └────────────────────────────────┘ × N layers (N=6 in original) ↓ Encoder output (contextualized representations)

Decoder Block

Output embeddings + Positional Encoding ↓ ┌────────────────────────────────┐ │ Masked Multi-Head Attention │ ← causal mask │ Add & LayerNorm │ │ Cross-Attention │ ← attends to encoder output │ Q from decoder, K,V from │ │ encoder │ │ Add & LayerNorm │ │ Feed-Forward Network │ │ Add & LayerNorm │ └────────────────────────────────┘ × N layers ↓ Linear → Softmax → next token probabilities

Encoder-Decoder Interaction

The decoder's cross-attention layer connects decoder queries with encoder keys and values. This is how the model "looks at" the input while generating output—critical for translation, summarization, and sequence-to-sequence tasks.

Positional Encoding

Self-attention is permutation-invariant—without positional information, "dog bites man" and "man bites dog" would have identical representations. Positional encoding injects order information.

Sinusoidal Encoding (Original)

Code example

PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Properties:
- Deterministic (no learned parameters)
- Extends to unseen sequence lengths
- Relative positions computable via linear transformation
  PE(pos+k) is a linear function of PE(pos)

Learned Positional Embeddings

Modern models (BERT, GPT) learn position embeddings directly:

Example

pos_embedding = Embedding(max_seq_len, d_model) Simpler, performs comparably for fixed-length inputs Cannot extrapolate beyond training length

Residual Connections and Layer Normalization

Residual	output = LayerNorm(x + Sublayer(x))
- Enables very deep networks (GPT-3	96 layers)
Pre-norm variant	LayerNorm(x) before sublayer (more stable)

Training Details

Original Transformer (Machine Translation)

Model	d_model=512, h=8, N=6, d_ff=2048
Training	WMT English-German, English-French
Optimizer	Adam with warmup schedule
Warmup steps	4000
Label smoothing	0.1
Dropout	0.1 on attention weights and sublayer outputs

Transformer Variants

Model	Type	Key Innovation
BERT	Encoder-only	Bidirectional, MLM pre-training
GPT	Decoder-only	Causal, autoregressive generation
T5	Encoder-Decoder	Text-to-text framework
ViT	Encoder	Image patches as tokens
DALL-E	Decoder	Image generation from text

Computational Complexity

Self-attention	O(n² · d) — quadratic in sequence length
FFN layer	O(n · d²) — linear in sequence length
n=512, d=512	attention ≈ FFN cost
n=4096, d=512	attention 8× more expensive
- Sparse attention (Longformer)	O(n · k)
- Linear attention (Performers)	O(n · d)
- Chunked attention (BigBird)	O(n · √n)

Interview Questions

Q: Why "Attention Is All You Need"? What was removed? A: Recurrence (RNNs) and convolutions (CNNs) were both removed. Self-attention alone captures sequential patterns, positional encoding provides order, and the FFN provides per-position nonlinear processing.

Q: What's the difference between self-attention and cross-attention? A: In self-attention, Q, K, V all come from the same sequence. In cross-attention, Q comes from one sequence (decoder) while K and V come from another (encoder output). Cross-attention implements the "looking at input" mechanism.

Q: How does the Transformer handle variable-length inputs? A: Padding tokens fill sequences to equal length within a batch. An attention mask sets padding positions to -∞ before softmax, zeroing their attention weights. The model effectively ignores padding.

Exam Focus

Revise definitions, diagrams, examples, and short-answer points for Transformers - Revolutionary Attention Architecture.

Interview Use

Prepare one clear explanation, one practical example, and one common mistake for this Artificial Intelligence topic.

Search Terms

artificial-intelligence, artificial intelligence, artificial, intelligence, deep, learning, transformers, transformers - revolutionary attention architecture

Continue learning this concept

Deep LearningBERT - Bidirectional Encoder RepresentationsMaster BERT architecture for AI. Learn bidirectional transformers, pretraining, fine-tuning, token embeddings, attention. Interview preparation 2024.Deep LearningConvolutional Neural Networks (CNN) - Complete AI GuideMaster CNN architecture for artificial intelligence. Learn convolution operation, pooling, filters, image classification, PyTorch implementation. Technical interview 2024.Deep LearningDeep Learning Fundamentals - Complete GuideMaster deep learning for artificial intelligence. Learn neural networks, backpropagation, optimization, regularization, training. BTech notes 2024.Deep LearningGenerative AI - Complete TutorialMaster generative models for AI. Learn VAE, GAN, diffusion models, image generation, text generation. BTech guide 2024.Deep LearningGPT Models - Generative Pre-trained TransformersMaster GPT architecture for AI. Learn causal language modeling, in-context learning, scaling laws, GPT-3, GPT-4. Interview guide 2024.Deep LearningLSTM Networks - Advanced Sequence ModelingMaster Long Short-Term Memory networks. Learn LSTM architecture, gates, bidirectional LSTM, stacked LSTMs, applications. AI interview guide 2024.

PreviousRecurrent Neural Networks (RNN) - Sequence AI Models NextConversational AI - Chatbots