Cloud Computing Hardware

Introduction

When you use Google Search, watch Netflix, or store files in Dropbox, your request travels to a massive data center containing thousands of servers. Cloud computing hardware is designed differently from your laptop — it optimizes for throughput (serving millions of users simultaneously), reliability (99.999% uptime), and cost-per-computation rather than single-user responsiveness. Understanding cloud hardware architecture shows you where computer organization principles scale to their extreme — and why companies like Google, Amazon, and Microsoft design their own custom processors.

Data Center Architecture

Physical Layout

Server Hardware

A typical cloud server node contains:

• CPU: 2× AMD EPYC or Intel Xeon (64-96 cores each)
• RAM: 256 GB - 2 TB DDR5 (8-12 memory channels)
• Storage: 4-8× NVMe SSDs (4-16 TB each)
• Network: 2× 100 Gbps or 400 Gbps Ethernet
• Accelerators: Optional GPU, FPGA, or custom ASIC

Network Architecture

Total bisection bandwidth of a modern data center can exceed 1 Petabit/second.

Custom Silicon for Cloud

Why Build Custom Chips?

At cloud scale, even small efficiency improvements save millions of dollars:

AWS Graviton (ARM-based Server CPU)

Amazon designed their own ARM server processor:

• Graviton3: 64 cores, ARM Neoverse V1, DDR5
• Performance: Matches x86 Xeon on most workloads
• Power efficiency: 60% better energy efficiency than comparable x86
• Cost: 40% better price-performance for AWS customers

This is ARM architecture applied at data center scale — proving that ARM's efficiency advantage scales beyond mobile.

Google TPU (Tensor Processing Unit)

Custom ASIC designed specifically for machine learning:

• TPU v4: 275 TFLOPS BF16 per chip
• Architecture: Systolic array of multiply-accumulate units
• Memory: High-bandwidth memory (HBM) for model weights
• Interconnect: Custom high-speed chip-to-chip links for distributed training

The TPU is 10-30× more power-efficient than GPUs for specific ML workloads because it does only one thing (matrix multiplication) and does it perfectly.

Reliability and Redundancy

Hardware Failure at Scale

With 100,000 servers, failures are constant:

• A server fails every few minutes
• A disk fails every few hours
• A network switch fails every few days

Architecture must tolerate failures without service interruption:

• ECC Memory: Corrects single-bit errors, detects double-bit
• RAID/Erasure Coding: Survives disk failures without data loss
• Redundant power: Dual power supplies, UPS, generators
• Hot-swap everything: Replace failed components without shutdown

Warehouse-Scale Computer Concepts

Google coined "warehouse-scale computer" — treating the entire data center as one computer:

• Software manages hardware failures transparently
• Data replicated across multiple servers/data centers
• Load balancing distributes work across healthy nodes
• Graceful degradation rather than hard failure

Power and Cooling

A large data center consumes 50-200 MW of power (equivalent to a small city):

Cooling innovations:

• Liquid cooling: Water pipes directly to CPU heatsinks
• Hot/cold aisle containment: Separate hot exhaust from cold intake air
• Free cooling: Use outside air when ambient temperature allows
• Immersion cooling: Submerge servers in non-conductive liquid

Heterogeneous Computing in Cloud

Modern cloud servers combine multiple processor types:

• CPU: General computation, OS, control plane
• GPU: AI inference/training, video transcoding
• FPGA: Custom acceleration (compression, encryption)
• SmartNIC/DPU: Network processing offloaded from CPU
• Custom ASIC: TPU for ML, video encoders, crypto mining

Key Takeaways

1. Cloud hardware optimizes for throughput and efficiency at scale, not single-user latency
2. Custom silicon (Graviton, TPU) provides 30-60% better efficiency than general-purpose CPUs for specific workloads
3. At data center scale, hardware failures are routine events — architecture must be fault-tolerant by design
4. Power and cooling consume 30-40% of total data center cost — energy efficiency directly impacts profitability
5. Heterogeneous computing (CPU + GPU + FPGA + custom ASIC) assigns each task to the most efficient processor type
6. The same architecture principles (caching, pipelining, parallelism) apply at cloud scale — just with thousands of machines instead of components within one chip