BERT - Bidirectional Encoder Representations

What Makes BERT Revolutionary

BERT (Bidirectional Encoder Representations from Transformers), introduced by Google in 2018, fundamentally changed how we approach NLP. Before BERT, language models like GPT processed text left-to-right (or right-to-left), building context from only one direction. BERT's key innovation is bidirectional attention—every token attends to every other token simultaneously, building rich contextualized representations that capture meaning from both preceding and following context.

Consider the word "bank" in these sentences: "I went to the bank to deposit money" versus "I sat on the river bank." A left-to-right model processing "bank" hasn't yet seen "deposit" or "river," making disambiguation difficult. BERT sees the entire sentence at once, making contextual understanding natural.

BERT achieved state-of-the-art results on 11 NLP benchmarks simultaneously upon release, establishing the pretrain-then-finetune paradigm that dominates modern NLP.

Architecture Deep Dive

Model Variants

Input Representation

BERT's input is the sum of three embedding types:

[CLS] token: Its final hidden state serves as the aggregate sequence representation for classification tasks.

[SEP] token: Separates sentence pairs and marks sentence boundaries.

WordPiece Tokenization

BERT uses WordPiece with a 30,000 token vocabulary:

This handles rare words by breaking them into known subwords, eliminating out-of-vocabulary issues.

Pre-training Objectives

1. Masked Language Modeling (MLM)

MLM is BERT's primary training signal. Randomly mask 15% of input tokens and predict them:

Training Example Walkthrough

2. Next Sentence Prediction (NSP)

Binary classification: does sentence B follow sentence A in the original corpus?

NSP was later shown to be less important than MLM (RoBERTa removed it entirely with no performance loss), but it helps with tasks like question answering and natural language inference.

Fine-tuning BERT for Downstream Tasks

The pretrain-finetune paradigm:

1. Pre-train on massive unlabeled corpus (BookCorpus + Wikipedia, ~3.3B words)
2. Fine-tune on small task-specific labeled data (often just thousands of examples)

Task-Specific Architectures

Fine-tuning Hyperparameters

Attention Mechanism in BERT

Each transformer layer has multi-head self-attention:

With 12 heads in BERT-Base, each head learns different relationship patterns: some capture syntax, others semantics, coreference, or positional patterns.

BERT Variants and Successors

Practical Considerations

Computational Requirements

Limitations

1. Max 512 tokens: Cannot process long documents directly
2. Slow inference: Full transformer computation for every input
3. No generation: Encoder-only; cannot generate text autoregressively
4. Static after fine-tuning: Cannot learn from new data without retraining

Interview Questions

Q: Why bidirectional? What's the trade-off vs. autoregressive? A: Bidirectional context gives richer representations (sees full sentence). The trade-off: BERT cannot generate text token-by-token since it needs the full input. For generation tasks, autoregressive models (GPT) are necessary.

Q: Why does BERT mask only 15% of tokens? A: Masking too many tokens (say 50%) would make the context too sparse—predictions become unreliable. Masking too few (say 1%) makes training extremely slow. 15% balances training signal quality with efficiency.

Q: How does BERT handle words not in its vocabulary? A: WordPiece tokenization splits unknown words into known subword units. "Transformers" might become ["Transform", "##ers"]. The model learns subword compositions through pre-training.