Cache Memory — Computer Organization & Architecture

Introduction

Cache memory is the unsung hero of computer performance. It's a small, fast memory that sits between the CPU and main memory, storing copies of frequently accessed data. Without cache, modern processors would spend most of their time waiting for slow main memory — effectively wasting 99% of their computing power. Cache exploits the locality of reference to serve most memory requests at near-register speed.

Why Cache is Necessary

Consider a modern CPU running at 4 GHz (0.25 ns per cycle) accessing DRAM at 50 ns:

• Without cache: CPU waits 200 cycles for every memory access!
• With L1 cache (1 ns): CPU waits only 4 cycles for cache hits (95% of accesses)
• Effective speedup: massive

The "memory wall" — the growing gap between processor speed and memory speed — makes cache increasingly critical with each processor generation.

Cache Operation

Basic Flow

1. CPU requests data at address X
2. Cache checks if address X is stored (tag comparison)
3. Cache hit: Data found → return data to CPU (fast, 1-4 cycles)
4. Cache miss: Data not found → fetch from main memory (slow, 50-200 cycles)

• Load data into cache (for future accesses)
• Return data to CPU

Cache Structure

Each cache entry (line) contains:

• Valid bit: Is this entry currently holding valid data?
• Tag: Identifies which memory block is stored here
• Data block: The actual cached data (typically 64 bytes)

Address Breakdown

A memory address is split into fields for cache lookup:

• Block Offset: Which byte within the cache line (log₂(line_size) bits)
• Index: Which cache set to look in (log₂(number_of_sets) bits)
• Tag: Remaining upper bits — used to verify the correct block is cached

Cache Levels

L1 Cache (Level 1)

• Closest to CPU, on the same die
• Typically split: L1I (instructions) + L1D (data) — Harvard-style
• Size: 32-80 KB per core
• Access: 1-4 cycles
• Line size: 64 bytes

L2 Cache (Level 2)

• Per-core (in modern designs)
• Unified (instructions + data)
• Size: 256 KB - 2 MB per core
• Access: 10-12 cycles

L3 Cache (Level 3)

• Shared across all cores
• Size: 8-256 MB total
• Access: 30-40 cycles
• Acts as a "victim cache" for L2 evictions

Modern Example (Intel Core i7-13700K)

Cache Hits and Misses

Types of Cache Misses (The Three Cs)

Compulsory (Cold) Misses:

• First access to a block — data has never been in cache
• Unavoidable — every block must be loaded at least once
• Solution: Prefetching (predict and load data before it's needed)

Capacity Misses:

• Cache is too small to hold all needed data
• Working set exceeds cache size
• Solution: Larger cache (but slower and more expensive)

Conflict Misses:

• Multiple blocks map to the same cache set
• Data evicted even though cache has empty space elsewhere
• Solution: Higher associativity (more ways per set)

Miss Penalty

Time to fetch data from the next level:

• L1 miss → L2: ~10 cycles penalty
• L2 miss → L3: ~30 cycles penalty
• L3 miss → DRAM: ~100-200 cycles penalty

Write Policies

Write Hit Policies

Write-through: Write to BOTH cache and main memory immediately

• Memory always up-to-date
• Simple but generates lots of memory traffic
• Use a write buffer to avoid stalling CPU

Write-back: Write ONLY to cache; write to memory when evicted

• Less memory traffic (many writes absorbed by cache)
• More complex (need dirty bit to track modified lines)
• Dominant in modern systems

Write Miss Policies

Write-allocate: On write miss, load the block into cache, then write

• Works well with write-back
• Bet that you'll write (or read) this block again soon

No-write-allocate: On write miss, write directly to memory (skip cache)

• Works well with write-through
• Avoids polluting cache with write-only data

Cache Performance Metrics

Example:

• L1: 1 cycle hit, 5% miss rate
• L2: 10 cycles hit, 2% miss rate (of L1 misses)
• Memory: 100 cycles

Compare to 100 cycles without any cache!

Key Takeaways

1. Cache bridges the speed gap between CPU and main memory using SRAM technology
2. Cache works by exploiting locality — keeping recently/frequently used data close to the CPU
3. Addresses are split into tag, index, and offset for cache lookup
4. Modern systems use 2-3 cache levels (L1: fastest/smallest, L3: slowest/largest)
5. Three types of misses: compulsory, capacity, and conflict (the "3 Cs")
6. Write-back with write-allocate is the dominant policy in modern designs
7. Cache transforms a 100-cycle memory access into an effective ~2-6 cycle average