Shared Memory Systems

Introduction

Shared memory systems are multiprocessor architectures where all processors access a common physical address space. Any processor can read from or write to any memory location, and communication between processors happens implicitly through shared variables in this common memory. This programming model is intuitive — it mirrors how threads communicate within a single program — but the hardware that supports it must solve complex problems of memory access uniformity, coherence, and synchronization.

Shared memory architectures are the foundation of nearly all modern server and desktop systems. From a dual-core laptop to a 4-socket server with 256 cores, the shared memory model provides the abstraction that makes parallel programming tractable. Understanding the different flavors of shared memory (UMA vs. NUMA) and the synchronization mechanisms they require is essential for anyone working with parallel systems.

Uniform Memory Access (UMA)

In a UMA system, all processors have equal access time to all memory locations. The memory is physically equidistant from every processor, connected through a shared bus or crossbar switch. A memory access takes the same time regardless of which processor issues it and which memory module contains the data.

Characteristics of UMA

• All processors are symmetric (hence the name Symmetric Multiprocessor or SMP)
• Memory access latency is uniform — typically 50-100 nanoseconds to main memory
• A shared bus or crossbar connects processors to memory modules
• Cache coherence is maintained by snooping the shared bus
• Scalability is limited because the shared bus becomes a bottleneck (typically 2-8 processors)

UMA Architecture Diagram

In practice, modern UMA systems are single-socket multicore chips where all cores share a memory controller. The "bus" is now an on-chip ring or mesh interconnect, and the shared L3 cache reduces memory traffic.

Non-Uniform Memory Access (NUMA)

NUMA architectures assign each processor (or group of processors) its own local memory. Processors can still access all memory in the system, but accessing local memory is faster than accessing remote memory (belonging to another processor's node).

Characteristics of NUMA

• Each processor node has local memory with fast access (50-80 ns)
• Remote memory access goes through an interconnect and takes longer (100-300 ns)
• The ratio of remote to local latency (NUMA factor) is typically 2-4x
• Scales to large processor counts (64-256+ processors) without bus bottlenecks
• Requires NUMA-aware memory allocation for best performance

NUMA Architecture Diagram

Intel's multi-socket Xeon systems use NUMA with QPI (QuickPath Interconnect) or UPI (Ultra Path Interconnect) connecting sockets. AMD's EPYC processors are NUMA even within a single socket due to their chiplet design.

NUMA-Aware Programming

For best performance on NUMA systems, software should:

• Allocate memory on the same node where it will be accessed (first-touch policy)
• Keep threads and their data on the same node (thread pinning)
• Minimize cross-node communication
• Use local allocation policies in the operating system (e.g., Linux's numactl)

A poorly written program that scatters memory allocations randomly can run 2-4x slower on a NUMA system compared to a NUMA-aware version.

Synchronization Mechanisms

When multiple processors share memory, they need synchronization mechanisms to coordinate access and prevent data corruption. Without synchronization, concurrent writes to shared data produce unpredictable results.

Atomic Operations

Atomic operations are hardware-guaranteed indivisible operations. The processor ensures that no other processor can observe the operation in a half-completed state:

Test-and-Set (TAS): Atomically reads a memory location, returns the old value, and sets it to 1. Used to implement simple spin locks:

Compare-and-Swap (CAS): Atomically compares a memory location with an expected value and, if they match, replaces it with a new value. More powerful than TAS — enables lock-free algorithms:

Fetch-and-Add: Atomically adds a value to a memory location and returns the old value. Useful for counters and ticket locks.

Barriers

A barrier is a synchronization point where all threads must arrive before any can proceed. This is essential in parallel algorithms with phases — all threads must complete phase 1 before any begins phase 2. Hardware barriers use special counters; software barriers use atomic operations and spinning.

Memory Fences

Memory fences (barriers) enforce ordering of memory operations. Modern processors reorder loads and stores for performance, but this can cause one processor to observe another processor's writes in a different order than they were issued. Memory fences prevent this reordering:

• Store fence: Ensures all prior stores are visible before subsequent stores
• Load fence: Ensures all prior loads complete before subsequent loads
• Full fence: Prevents reordering of both loads and stores across the fence point

Memory Consistency Models

A memory consistency model defines the rules for when a write by one processor becomes visible to reads by other processors:

Sequential Consistency: The strongest model — all processors see all memory operations in the same total order, and each processor's operations appear in program order. Simple to reason about but restricts hardware optimization.

Total Store Order (TSO): Used by x86 processors. Stores may be delayed (buffered) but appear in program order. Loads can pass stores (reads may see local writes before they become globally visible). Most programs work correctly without modification.

Relaxed Models: ARM and RISC-V use relaxed models where both loads and stores can be reordered. Programmers must explicitly insert memory fences where ordering matters. More hardware optimization freedom but harder to program.

Interconnection Networks

Shared memory systems use different interconnect topologies depending on scale:

Key Takeaways

1. Shared memory systems allow all processors to access a common address space, simplifying parallel programming
2. UMA provides uniform access time but scales poorly beyond 8 processors due to bus contention
3. NUMA assigns local memory to each node, scaling to hundreds of processors at the cost of non-uniform latency
4. Atomic operations (test-and-set, compare-and-swap) provide hardware-guaranteed indivisible memory operations for building locks
5. Barriers synchronize threads at phase boundaries; memory fences enforce ordering of memory operations
6. Memory consistency models (sequential, TSO, relaxed) define the rules for inter-processor memory visibility
7. NUMA-aware programming that keeps data local to the accessing processor can improve performance by 2-4x