GPT Models - Generative Pre-trained Transformers

The Autoregressive Paradigm

GPT (Generative Pre-trained Transformer) represents a fundamentally different philosophy from BERT. Where BERT uses bidirectional context for understanding, GPT uses left-to-right (causal) attention for generation. The model predicts the next token given all previous tokens, making it naturally suited for text generation, dialogue, code completion, and any task that can be framed as sequence completion.

The GPT family—from GPT-1 (2018) through GPT-4 (2023)—demonstrated that scaling up autoregressive language models produces increasingly capable systems, eventually exhibiting emergent abilities like in-context learning, chain-of-thought reasoning, and instruction following without task-specific training.

Architecture: Decoder-Only Transformer

Structural Comparison

GPT Architecture Stack

Causal Attention Mask

The key mechanism that makes GPT autoregressive:

This ensures that during training, each position's prediction uses only past context—exactly matching the generation scenario where future tokens don't exist yet.

Training Objective

Next-Token Prediction (Causal Language Modeling)

Training Efficiency

Unlike BERT's MLM which only trains on 15% of tokens per example, GPT trains on every token in every sequence. This makes GPT significantly more sample-efficient per token processed, though it sees less context per prediction (only left context vs. bidirectional).

Scaling Laws: The Power of Size

Empirical Scaling Laws (Kaplan et al., 2020)

GPT Family Scaling

Each generation brought not just better performance on existing tasks, but qualitatively new capabilities (emergent abilities).

In-Context Learning

GPT-3's breakthrough discovery: large models can learn tasks from examples in the prompt without any parameter updates.

Few-Shot Learning

Types of In-Context Learning

Performance generally improves with more examples, up to context window limits.

Generation Strategies

Temperature Sampling

Top-k and Top-p (Nucleus) Sampling

From GPT-3 to InstructGPT and ChatGPT

The Alignment Problem

Raw GPT-3 is trained to predict internet text—it mimics the training distribution, which includes toxic content, hallucinations, and unhelpful patterns. Making it useful and safe requires alignment.

RLHF Pipeline (Reinforcement Learning from Human Feedback)

This pipeline produced InstructGPT and later ChatGPT—models that follow instructions, refuse harmful requests, and produce helpful responses.

GPT vs. BERT: When to Use Which

Emergent Abilities

Large GPT models exhibit capabilities not present in smaller versions:

These abilities appear suddenly at certain scales—they're not gradual improvements but phase transitions in capability.

Interview Questions

Q: Why causal attention instead of bidirectional? A: During generation, future tokens don't exist yet—you can't attend to what hasn't been produced. Causal masking at training time matches this generation constraint, ensuring consistent behavior between training and inference.

Q: How does in-context learning work mechanistically? A: The transformer's attention layers implement implicit gradient descent on the examples. The model effectively "trains" on the few-shot examples within its forward pass, using attention to implement a learning algorithm over the context.

Q: What are the limitations of autoregressive generation? A: Each token is committed once generated—no revision. The model can't "plan ahead" or consider global document structure. It's also sequential (each token depends on the previous), making generation inherently slower than parallel encoding.