GPU Architecture

Graphics Processing Unit architecture, SIMT execution model, and parallel computing capabilities.

Introduction

A GPU (Graphics Processing Unit) is a massively parallel processor containing thousands of simple cores designed to perform the same operation on many data elements simultaneously. While a CPU excels at complex sequential tasks (running an OS, parsing JSON, evaluating conditionals), a GPU excels at simple repetitive tasks applied to massive datasets (rendering millions of pixels, training neural networks, processing video frames). Understanding GPU architecture reveals a fundamentally different approach to computing — one where parallelism, not single-thread speed, is the primary design goal.

CPU vs GPU: Fundamental Design Difference

CPU Design Philosophy (Latency-Optimized)

┌──────────────────────────────────────────────────┐

│ ████████████████ Large Cache (reduce latency) │

│ ████████████████ │

│ ████████ Control Logic (branch prediction, OoO) │

│ ████ ████ ████ ████ 4-8 Cores │

│ (Each core: very powerful, complex) │

└──────────────────────────────────────────────────┘

GPU Design Philosophy (Throughput-Optimized)

┌──────────────────────────────────────────────────┐

│ ██ Small Cache │

│ █ Minimal Control │

│ ████████████████████████████████████████████████ │

│ (Thousands of simple cores — raw compute power) │

└──────────────────────────────────────────────────┘

Metric	Modern CPU (i9-14900K)	Modern GPU (RTX 4090)
Cores	24 (8P + 16E)	16,384 CUDA cores
Clock	Up to 6 GHz	2.5 GHz
Cache	36 MB L3	72 MB L2
Memory BW	90 GB/s (DDR5)	1,008 GB/s (GDDR6X)
FP32 TFLOPS	~1.5	82.6
Power	253W	450W
Transistors	~3 billion	~76 billion

The GPU has 55× more raw floating-point throughput but is useless for a single sequential task.

GPU Architecture (NVIDIA Example)

Streaming Multiprocessor (SM)

The SM is the fundamental building block. An RTX 4090 has 128 SMs:

┌───────────────────────────────────────────────────┐ │ Streaming Multiprocessor (SM) │ ├───────────────────────────────────────────────────┤ │ │ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │ │ FP32 Core │ │ FP32 Core │ │ FP32 Core │... │ │ │ (×128) │ │ │ │ │ │ │ └────────────┘ └────────────┘ └────────────┘ │ │ │ │ ┌────────────┐ ┌────────────┐ │ │ │INT32 Cores │ │Tensor Cores│ (Matrix multiply) │ │ │ (×128) │ │ (×4) │ │ │ └────────────┘ └────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Warp Scheduler (×4) — issues instructions │ │ │ └────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Shared Memory / L1 Cache (128 KB) │ │ │ └────────────────────────────────────────────┘ │ │ │ │ Register File: 256 KB (65,536 × 32-bit regs) │ └───────────────────────────────────────────────────┘

SIMT Execution Model

GPU cores execute in groups of 32 threads called a warp (NVIDIA) or wavefront (AMD). All 32 threads in a warp execute the same instruction simultaneously:

Thread 0	ADD R1, R2, R3 (on data element 0)
Thread 1	ADD R1, R2, R3 (on data element 1)
Thread 2	ADD R1, R2, R3 (on data element 2)
Thread 31	ADD R1, R2, R3 (on data element 31)

This is SIMT (Single Instruction, Multiple Threads) — similar to SIMD but each thread has its own registers and can follow different paths (with performance penalties for divergence).

Branch Divergence Problem

Example

if (threadIdx < 16) A(); // Threads 0-15 execute this else B(); // Threads 16-31 execute this

Since all threads in a warp share one instruction, both paths execute serially:

Threads 0-15 execute A(), threads 16-31 are masked (idle)
Threads 16-31 execute B(), threads 0-15 are masked (idle)
Efficiency: 50% (half the threads idle at any time)

This is why GPU code should minimize branch divergence within warps.

Memory Hierarchy

Register File	256 KB per SM (fastest, private per thread)
Shared Memory	128 KB per SM (shared among threads in block)
L1 Cache	Combined with shared memory
L2 Cache	72 MB (shared across all SMs)
Global Memory	24 GB GDDR6X (1 TB/s bandwidth, 400+ cycle latency)

Key insight: GPU hides memory latency through massive parallelism rather than caches. While one warp waits for memory, another warp executes — thousands of warps keep the cores busy.

Programming Model (CUDA/OpenCL)

Example

// CUDA Kernel: Add two vectors __global__ void vectorAdd(float *A, float *B, float *C, int n) { int i = blockIdx.x * blockDim.x + threadIdx.x; if (i < n) C[i] = A[i] + B[i]; } // Launch 1 million threads! vectorAdd<<<4096, 256>>>(A, B, C, 1000000); // 4096 blocks × 256 threads = 1,048,576 parallel threads

Each thread computes ONE element. The GPU hardware schedules millions of threads across its SMs.

Applications

Application	Why GPU?
Gaming (3D rendering)	Each pixel computed independently (millions of pixels per frame)
Deep Learning	Neural networks are matrix multiplications (perfectly parallel)
Cryptocurrency mining	Hash computation is embarrassingly parallel
Scientific simulation	Physics simulations with millions of particles
Video encoding	Each macroblock can be processed independently

Key Takeaways

GPUs trade single-thread performance for massive parallelism — thousands of simple cores vs few complex cores
SIMT execution runs 32 threads in lockstep on the same instruction — efficient only when threads do the same thing
Branch divergence is the GPU performance killer — different code paths within a warp serialize execution
Memory latency is hidden by switching between thousands of warps rather than using large caches
GPU memory bandwidth (1 TB/s) dwarfs CPU bandwidth (90 GB/s) — essential for feeding thousands of cores
GPUs excel at data-parallel problems where the same operation applies to many data elements independently

Introduction

CPU vs GPU: Fundamental Design Difference

CPU Design Philosophy (Latency-Optimized)

┌──────────────────────────────────────────────────┐

│ ████████████████ Large Cache (reduce latency) │

│ ████████████████ │

│ ████████ Control Logic (branch prediction, OoO) │

│ ████ ████ ████ ████ 4-8 Cores │

│ (Each core: very powerful, complex) │

└──────────────────────────────────────────────────┘

GPU Design Philosophy (Throughput-Optimized)

┌──────────────────────────────────────────────────┐

│ ██ Small Cache │

│ █ Minimal Control │

│ ████████████████████████████████████████████████ │

│ (Thousands of simple cores — raw compute power) │

└──────────────────────────────────────────────────┘

Metric	Modern CPU (i9-14900K)	Modern GPU (RTX 4090)
Cores	24 (8P + 16E)	16,384 CUDA cores
Clock	Up to 6 GHz	2.5 GHz
Cache	36 MB L3	72 MB L2
Memory BW	90 GB/s (DDR5)	1,008 GB/s (GDDR6X)
FP32 TFLOPS	~1.5	82.6
Power	253W	450W
Transistors	~3 billion	~76 billion

The GPU has 55× more raw floating-point throughput but is useless for a single sequential task.

GPU Architecture (NVIDIA Example)

Streaming Multiprocessor (SM)

The SM is the fundamental building block. An RTX 4090 has 128 SMs:

SIMT Execution Model

GPU cores execute in groups of 32 threads called a warp (NVIDIA) or wavefront (AMD). All 32 threads in a warp execute the same instruction simultaneously:

Thread 0	ADD R1, R2, R3 (on data element 0)
Thread 1	ADD R1, R2, R3 (on data element 1)
Thread 2	ADD R1, R2, R3 (on data element 2)
Thread 31	ADD R1, R2, R3 (on data element 31)

This is SIMT (Single Instruction, Multiple Threads) — similar to SIMD but each thread has its own registers and can follow different paths (with performance penalties for divergence).

Branch Divergence Problem

Example

if (threadIdx < 16) A(); // Threads 0-15 execute this else B(); // Threads 16-31 execute this

Since all threads in a warp share one instruction, both paths execute serially:

Threads 0-15 execute A(), threads 16-31 are masked (idle)
Threads 16-31 execute B(), threads 0-15 are masked (idle)
Efficiency: 50% (half the threads idle at any time)

This is why GPU code should minimize branch divergence within warps.

Memory Hierarchy

Register File	256 KB per SM (fastest, private per thread)
Shared Memory	128 KB per SM (shared among threads in block)
L1 Cache	Combined with shared memory
L2 Cache	72 MB (shared across all SMs)
Global Memory	24 GB GDDR6X (1 TB/s bandwidth, 400+ cycle latency)

Key insight: GPU hides memory latency through massive parallelism rather than caches. While one warp waits for memory, another warp executes — thousands of warps keep the cores busy.

Programming Model (CUDA/OpenCL)

Example

Each thread computes ONE element. The GPU hardware schedules millions of threads across its SMs.

Applications

Application	Why GPU?
Gaming (3D rendering)	Each pixel computed independently (millions of pixels per frame)
Deep Learning	Neural networks are matrix multiplications (perfectly parallel)
Cryptocurrency mining	Hash computation is embarrassingly parallel
Scientific simulation	Physics simulations with millions of particles
Video encoding	Each macroblock can be processed independently

Key Takeaways

GPUs trade single-thread performance for massive parallelism — thousands of simple cores vs few complex cores
SIMT execution runs 32 threads in lockstep on the same instruction — efficient only when threads do the same thing
Branch divergence is the GPU performance killer — different code paths within a warp serialize execution
Memory latency is hidden by switching between thousands of warps rather than using large caches
GPU memory bandwidth (1 TB/s) dwarfs CPU bandwidth (90 GB/s) — essential for feeding thousands of cores
GPUs excel at data-parallel problems where the same operation applies to many data elements independently

GPU Architecture

Introduction

CPU vs GPU: Fundamental Design Difference

CPU Design Philosophy (Latency-Optimized)

GPU Design Philosophy (Throughput-Optimized)

GPU Architecture (NVIDIA Example)

Streaming Multiprocessor (SM)

SIMT Execution Model

Branch Divergence Problem

Memory Hierarchy

Programming Model (CUDA/OpenCL)

Applications

Key Takeaways

Continue learning this concept

GPU Architecture

Introduction

CPU vs GPU: Fundamental Design Difference

CPU Design Philosophy (Latency-Optimized)

GPU Design Philosophy (Throughput-Optimized)

GPU Architecture (NVIDIA Example)

Streaming Multiprocessor (SM)

SIMT Execution Model

Branch Divergence Problem

Memory Hierarchy

Programming Model (CUDA/OpenCL)

Applications

Key Takeaways

Continue learning this concept