Multicore Processors

Introduction

Multicore processors represent the dominant approach to improving processor performance in the modern era. When increasing clock frequency hit physical limits around 2004 (due to power consumption scaling as frequency cubed and heat dissipation challenges), processor designers shifted strategy: instead of making one core faster, they put multiple independent cores on a single chip. A quad-core processor can execute four independent instruction streams simultaneously, potentially delivering four times the throughput of a single core for parallelizable workloads.

Today, desktop processors have 8-16 cores, server processors have 64-128 cores, and even mobile phone processors have 8 cores. Understanding multicore architecture is essential because virtually all modern software must be designed to exploit multiple cores to achieve good performance.

Physical Architecture

A multicore processor integrates multiple complete CPU cores on a single silicon die, sharing certain resources while keeping others private:

Private per core:

• Level 1 instruction cache (L1i): typically 32-64 KB
• Level 1 data cache (L1d): typically 32-64 KB
• Level 2 cache: typically 256 KB - 1 MB
• Execution units, register file, branch predictor

Shared across cores:

• Last-level cache (L3): typically 8-64 MB
• Memory controller and DRAM interface
• I/O interfaces (PCIe, USB controllers)
• Interconnect fabric (ring bus or mesh network)

This arrangement means each core can execute independently at full speed using its private caches, but all cores share the same main memory view — creating both opportunities and challenges.

Cache Coherence Problem

The central challenge in multicore design is cache coherence. When multiple cores cache the same memory location, what happens when one core modifies it?

Consider this scenario:

1. Core 0 reads variable X (value 5) into its L1 cache
2. Core 1 also reads X (value 5) into its L1 cache
3. Core 0 writes X = 10 (updates its L1 cache)
4. Core 1 reads X — does it see 5 (stale) or 10 (current)?

Without a coherence protocol, Core 1 would see the stale value 5, leading to incorrect program behavior. The hardware must ensure that all cores always observe a consistent view of memory.

The MESI Protocol

The MESI protocol is the most widely used cache coherence mechanism. Each cache line exists in one of four states:

Modified (M): This cache has the only valid copy, and it has been modified. Memory is stale. This cache must write back before any other cache can access the line.

Exclusive (E): This cache has the only copy, but it matches memory. Can be modified without notifying other caches (transitions to M).

Shared (S): Multiple caches may hold copies. All match memory. Must notify others before modifying (cannot silently transition to M).

Invalid (I): The cache line is not valid — effectively empty.

MESI State Transitions Example

1. Core 0 reads X (not cached anywhere): Core 0 fetches from memory → state = Exclusive
2. Core 1 reads X: Core 0 detects the snoop, downgrades to Shared; Core 1 gets copy in Shared
3. Core 0 writes X: Core 0 sends invalidation message to Core 1; Core 1's copy → Invalid; Core 0 → Modified
4. Core 1 reads X again: Core 0 must supply the data (write-back to memory or direct cache-to-cache transfer); both → Shared

The protocol uses a snooping mechanism where each core's cache controller monitors (snoops) the shared bus for memory transactions from other cores and takes appropriate action.

Simultaneous Multi-Threading (SMT)

Simultaneous Multi-Threading (Intel's brand name: Hyper-Threading) is a technique that makes one physical core appear as two logical cores to the operating system. The core maintains two sets of architectural state (registers, program counter) but shares the execution units, caches, and branch predictor.

When one thread stalls (on a cache miss, branch misprediction, or data dependency), the other thread can use the execution units that would otherwise sit idle. This improves utilization of the core's resources — a typical core uses only 30-40% of its execution unit capacity with a single thread due to stalls.

SMT provides roughly 20-30% improvement in overall throughput for typical workloads, at the cost of only about 5% additional die area for the duplicated register files and thread-selection logic.

Programming Challenges

Writing correct and efficient multicore software requires addressing several challenges:

Race Conditions

When multiple threads access shared data without proper synchronization, the result depends on the unpredictable timing of their execution — a race condition. The solution: mutual exclusion using locks (mutexes), ensuring only one thread accesses critical shared data at a time.

False Sharing

False sharing occurs when two threads modify different variables that happen to reside in the same cache line. Even though they are accessing independent data, the coherence protocol treats it as a conflict and bounces the cache line between cores. The solution: pad data structures so that different threads' data falls on different cache lines (64 bytes on most architectures).

Synchronization Primitives

Hardware provides atomic instructions that enable lock implementations:

• Compare-and-Swap (CAS): Atomically reads a value, compares it, and writes a new value only if the comparison matched
• Test-and-Set: Atomically reads a bit and sets it to 1
• Load-Linked/Store-Conditional (LL/SC): ARM and MIPS approach to atomic operations

These primitives are used to build higher-level synchronization: mutexes, semaphores, barriers, and lock-free data structures.

Heterogeneous Multicore (big.LITTLE)

Modern mobile processors use heterogeneous multicore designs with different types of cores:

• Performance cores (big): High clock speed, wide execution, out-of-order — for demanding tasks
• Efficiency cores (LITTLE): Lower clock speed, narrow execution, in-order — for background tasks

ARM's big.LITTLE (and Apple's equivalent) allows the scheduler to assign work to the appropriate core type: heavy workloads run on performance cores while light background tasks run on efficiency cores, optimizing the power-performance trade-off. Intel's 12th generation and later processors adopted this approach with P-cores and E-cores.

Scalability Limits

Multicore scaling faces fundamental limits:

• Amdahl's Law: If 10% of a program is inherently serial, maximum speedup with infinite cores is only 10x
• Memory bandwidth: All cores compete for the same DRAM bandwidth
• Coherence traffic: More cores mean more snooping and invalidation messages on the interconnect
• Software parallelism: Many applications have limited inherent parallelism

Key Takeaways

1. Multicore processors place multiple independent CPU cores on one die to increase throughput without increasing clock frequency
2. Each core has private L1/L2 caches but shares L3 cache and memory controller with other cores
3. Cache coherence (MESI protocol) ensures all cores observe consistent memory state through snooping and invalidation
4. SMT (Hyper-Threading) shares one core between two threads, improving utilization by 20-30%
5. Programming multicore systems requires careful synchronization using atomic operations, locks, and awareness of false sharing
6. Heterogeneous multicore (big.LITTLE) mixes performance and efficiency cores for power optimization
7. Amdahl's Law and memory bandwidth fundamentally limit multicore scaling