Shannon

Shannon's Theorem: The Ultimate Speed Limit of Communication

Shannon's Channel Capacity Theorem is the single most important result in all of communication theory. Published in 1948, it answers the question that every communication engineer asks: "What is the maximum rate at which information can be transmitted over a noisy channel with arbitrarily small error probability?" The answer is both profound and practical, guiding the design of every modern communication system.

The Revolutionary Statement

Claude Shannon proved that for a band-limited channel with additive white Gaussian noise:

C = B × log₂(1 + S/N) bits/second

Where:

• C = channel capacity (maximum achievable data rate with negligible errors)
• B = channel bandwidth in Hz
• S/N = signal-to-noise ratio (linear, not dB)
• S = average signal power (watts)
• N = average noise power (watts) = N₀ × B

This single equation encompasses two revolutionary claims:

1. Achievability: For any rate R < C, there exist codes that achieve error probability as close to zero as desired
2. Converse: For any rate R > C, reliable communication is impossible — error probability cannot be made arbitrarily small

Think of it this way: capacity is like the speed limit on a highway. Shannon proved that you can drive at any speed below the limit safely (with the right vehicle/code), but going above the limit guarantees an accident (errors) eventually.

Understanding the Formula

The bandwidth term (B): More bandwidth = more capacity, linearly. This makes intuitive sense — a wider pipe carries more water.

The logarithmic SNR term: Capacity grows only logarithmically with SNR. Doubling transmit power gives only one extra bit/s/Hz. This reveals that power is an inefficient way to increase rate — bandwidth is more valuable.

Numerical examples:

• B = 1 MHz, SNR = 30 dB (1000): C = 10⁶ × log₂(1001) ≈ 10 Mbps
• B = 1 MHz, SNR = 20 dB (100): C = 10⁶ × log₂(101) ≈ 6.66 Mbps
• B = 1 MHz, SNR = 10 dB (10): C = 10⁶ × log₂(11) ≈ 3.46 Mbps
• B = 1 MHz, SNR = 0 dB (1): C = 10⁶ × log₂(2) = 1 Mbps

Doubling SNR from 10 dB to 13 dB only adds about 1 Mbps, but doubling bandwidth from 1 MHz to 2 MHz doubles capacity at any SNR.

The Bandwidth-Power Trade-off

Shannon's theorem reveals a fundamental exchange between bandwidth and power:

For fixed capacity C: You can achieve the same rate using:

• Narrow bandwidth + high power (bandwidth-efficient but power-hungry)
• Wide bandwidth + low power (power-efficient but bandwidth-hungry)

Limiting cases:

Infinite bandwidth (B → ∞): C approaches C_∞ = (S/N₀) × log₂(e) = 1.44 × S/N₀

Even with unlimited bandwidth, capacity is bounded by available power.

Minimum Eb/N₀: As spectral efficiency (C/B) approaches zero: Eb/N₀(min) = ln(2) = 0.693 = -1.59 dB

This is the Shannon limit — no digital system can operate reliably below Eb/N₀ = -1.59 dB regardless of bandwidth, coding, or modulation used.

Spectral Efficiency Bound

Shannon's theorem also bounds the spectral efficiency:

C/B = log₂(1 + S/N) bits/s/Hz

For real-world systems:

Modern systems operate within 1-3 dB of Shannon capacity — remarkably close to the theoretical limit.

Why 75 Years to Approach Capacity?

Shannon proved capacity exists in 1948, but practical codes approaching it took decades:

• 1950s-60s: Block codes (Hamming, BCH) — far from capacity
• 1970s: Convolutional + Viterbi — within 3-5 dB
• 1993: Turbo codes — within 0.5 dB (breakthrough!)
• 1990s-2000s: LDPC codes — within 0.1 dB
• 2009: Polar codes — provably capacity-achieving

The key was Shannon's proof is existential (random codes achieve capacity) but non-constructive (no algorithm given). Finding practical codes with feasible encoding/decoding was the grand challenge.

Implications for System Design

1. Power-limited systems (satellite, deep space):

• Operate at low spectral efficiency (C/B small)
• Use low-rate codes (rate 1/6 to 1/2)
• Use bandwidth-expanding modulation (BPSK, spread spectrum)
• Goal: minimize Eb/N₀ requirement

2. Bandwidth-limited systems (terrestrial wireless, cable):

• Operate at high spectral efficiency (C/B large)
• Use high-order modulation (64-QAM, 256-QAM)
• Use high-rate codes (rate 3/4 to 5/6)
• Goal: maximize bits per Hertz

The Channel Coding Theorem (Noisy Channel Coding Theorem)

The more general form of Shannon's theorem applies to any discrete memoryless channel:

C = max_{p(x)} I(X;Y) bits/channel use

Where the maximum is over all possible input distributions p(x), and I(X;Y) is mutual information between input X and output Y.

For the binary symmetric channel (BSC) with crossover probability p: C_BSC = 1 - H(p) = 1 + p×log₂(p) + (1-p)×log₂(1-p)

For the binary erasure channel (BEC) with erasure probability ε: C_BEC = 1 - ε

Practical Significance

Shannon's theorem tells engineers:

• The speed limit exists — stop trying to exceed it
• Approaching the limit requires long codes and complex decoding — there is no shortcut
• The gap between current performance and capacity shows how much room for improvement remains
• System design is fundamentally about allocating bandwidth and power resources optimally

Key Takeaways

1. Shannon capacity C = B×log₂(1+SNR) is the maximum achievable error-free data rate through an AWGN channel — an absolute physical law.
2. Capacity grows linearly with bandwidth but only logarithmically with power — bandwidth is the more valuable resource.
3. The Shannon limit of Eb/N₀ = -1.59 dB is the absolute minimum energy per bit for reliable communication at any rate.
4. Modern codes (LDPC, Turbo, Polar) operate within 0.1-0.5 dB of capacity — approaching theoretical perfection.
5. System design must choose between power efficiency (spread spectrum, low-rate codes) and bandwidth efficiency (high-order QAM, high-rate codes).
6. Shannon's theorem guides all modern communication design by establishing what is achievable and what is impossible.