Modern CPU Design

Introduction

A modern CPU is perhaps the most complex object ever created by humans — billions of transistors working together to execute billions of operations per second. But the core ideas that make it fast are understandable: do multiple things at once (superscalar), do them in any order that data allows (out-of-order execution), guess what comes next (branch prediction), and keep data close (caches). This page ties together all these techniques to show how a modern processor like Intel Core or AMD Ryzen actually achieves its performance.

The Performance Challenge

A single instruction takes multiple clock cycles if executed sequentially. But users want performance — how do we get more work done per second?

Modern CPUs focus on reducing effective CPI below 1.0 (executing multiple instructions per cycle).

Superscalar Execution

A superscalar processor issues multiple instructions per clock cycle:

Real CPI is higher than ideal due to dependencies, misses, and stalls. Modern desktop CPUs achieve IPC (instructions per cycle) of 4-6 on favorable code.

Execution Units

A modern core has multiple parallel execution units:

Out-of-Order Execution

Why In-Order Is Slow

In-order: Everything stalls behind the cache miss. Wasted cycles.

How Out-of-Order Works

The key insight: Instructions with ready operands execute immediately, regardless of program order. Instructions waiting for data sit in the scheduler until their inputs arrive.

Register Renaming

Architectural registers (16 in x86-64) are mapped to many physical registers (200+):

WAW hazard eliminated — both writes go to different physical registers. The second instruction can execute without waiting for the first.

Reorder Buffer (ROB)

The ROB maintains program order for retirement:

This ensures that exceptions and interrupts see a consistent architectural state.

Branch Prediction

Why Prediction Matters

In a 20-stage pipeline, a branch misprediction wastes ~15-20 cycles (flushing all wrong-path instructions). With 15-20% of instructions being branches:

Prediction accuracy is CRITICAL for deep pipelines.

Modern Prediction Techniques

1. Branch Target Buffer (BTB): Caches target addresses of recent branches
2. Return Address Stack: Predicts function return addresses (near 100% accurate)
3. TAGE Predictor: Uses multiple history tables with different history lengths — can learn complex patterns
4. Loop Detector: Identifies counted loops and predicts exact iteration count
5. Indirect Branch Predictor: For virtual function calls (polymorphism in C++)

Modern predictors achieve 97-99% accuracy on typical code.

Memory Hierarchy Design

Prefetching

Modern CPUs do not just wait for cache misses — they predict future accesses and fetch data proactively:

• Hardware stride prefetcher: Detects sequential/strided access patterns and fetches ahead
• Next-line prefetcher: Always fetches the next cache line (works for sequential code)
• Spatial prefetcher: Detects related accesses across pages
• Software prefetch instructions: Programmer/compiler hints (PREFETCH instruction)

Store Buffer

Stores do not immediately write to cache (would stall the pipeline):

Multi-Core Design

Why Multi-Core?

Power wall: P ∝ V² × f, and voltage must increase with frequency. Doubling frequency more than doubles power. Instead, use two cores at half frequency: same performance, less power.

Cache Coherence (MESI Protocol)

When multiple cores share data, caches must stay synchronized:

Putting It All Together

A modern instruction flows through:

1. Branch Predictor → predicts direction, provides fetch address
2. Instruction Cache → fetches 32+ bytes (multiple instructions)
3. Decoder → translates to micro-ops (4-6 per cycle)
4. Micro-op Cache → bypasses decoder for hot code
5. Rename/Allocate → maps to physical registers, allocates ROB entry
6. Scheduler → waits for operands, issues to execution units
7. Execute → ALU/Load/Store/Branch operations
8. Retire → commits results in program order

All happening simultaneously for hundreds of in-flight instructions.

Key Takeaways

1. Modern CPUs achieve IPC > 4 through superscalar execution with many parallel units
2. Out-of-order execution hides memory latency by executing independent instructions while waiting for cache misses
3. Register renaming eliminates false dependencies (WAR/WAW), exposing more parallelism
4. Branch prediction with 97%+ accuracy is essential — without it, deep pipelines would stall constantly
5. Multi-core design solves the power wall — parallel cores at lower frequency give better performance per watt
6. The combination of these techniques means a modern core effectively executes 4-6 instructions per cycle despite having a 20+ stage pipeline