Information Content

Information Content: Quantifying Surprise

Information content — also called self-information or surprisal — is the fundamental building block of Shannon's information theory. It answers a deceptively simple question: how much "information" does learning the outcome of an event provide? The answer, as Shannon elegantly showed, depends entirely on how surprising the outcome is.

The Intuition: Surprise Equals Information

Think of it this way: if a weather forecaster in the Sahara Desert announces "It will be sunny today," that statement carries essentially zero information — you already knew it would be sunny. But if they announce "It will snow today," that carries enormous information precisely because it is so unexpected.

Shannon formalized this intuition with three reasonable requirements for an information measure:

1. Rare events carry more information than common events — Learning something surprising should give more information than learning something expected
2. Certain events carry zero information — If something was guaranteed to happen, learning it happened tells you nothing new
3. Information from independent events adds — If you learn two unrelated facts, the total information should be the sum of individual information values

The only mathematical function satisfying all three requirements is the logarithm:

I(x) = -log₂(p(x)) = log₂(1/p(x)) bits

Where p(x) is the probability of event x occurring.

Why Logarithm? Why Base 2?

The logarithm is the unique function that converts multiplication into addition. Since independent event probabilities multiply (P(A and B) = P(A) × P(B) for independent events), their information should add:

I(A and B) = -log₂(P(A) × P(B)) = -log₂(P(A)) - log₂(P(B)) = I(A) + I(B)

Using base 2 means information is measured in bits (binary digits):

• 1 bit = information gained from one fair coin flip
• Base e gives "nats" (natural units)
• Base 10 gives "hartleys" or "bans"

The choice of base is merely a unit conversion — like choosing meters vs. feet.

Calculating Information Content

Examples with a standard deck of 52 cards:

• Drawing any specific card: I = -log₂(1/52) = log₂(52) = 5.70 bits
• Drawing a heart: I = -log₂(13/52) = -log₂(1/4) = 2 bits
• Drawing a face card: I = -log₂(12/52) = 2.12 bits
• Drawing the ace of spades: I = -log₂(1/52) = 5.70 bits

Examples with dice:

• Rolling a specific number on fair die: I = log₂(6) = 2.585 bits
• Rolling an even number: I = log₂(2) = 1 bit
• Rolling a 6 on a loaded die (p=0.5): I = log₂(2) = 1 bit

Properties of Self-Information

1. I(x) ≥ 0 — Information is never negative (you cannot "unlearn")
2. I(x) = 0 when p(x) = 1 — Certain events provide zero information
3. I(x) → ∞ as p(x) → 0 — Extremely rare events carry enormous information
4. Monotonically decreasing in p — Higher probability = less information
5. Additive for independent events — I(x,y) = I(x) + I(y) when x,y independent

From Information Content to Entropy

Entropy is simply the expected (average) value of self-information across all possible outcomes:

H(X) = E[I(X)] = Σ p(xᵢ) × I(xᵢ) = -Σ p(xᵢ) × log₂(p(xᵢ))

Think of entropy as "how surprised you expect to be on average." A source that always produces the same symbol has zero entropy (no surprise ever). A source with many equally-likely symbols has maximum entropy (always surprising).

Information in Communication Systems

In digital communication, information content directly relates to how many bits are needed to represent an event:

• A 256-gray-level pixel: If all levels equally likely, I = log₂(256) = 8 bits (hence why raw images use 8 bits per pixel)
• A compressed pixel: If level 128 has probability 0.5, that pixel only carries 1 bit of information
• English text: Individual letters carry about 4.7 bits if uniform, but only ~1.0-1.5 bits accounting for language statistics

Kraft Inequality and Coding Limits

The Kraft inequality connects information content to achievable code lengths. For any prefix-free code:

Σ 2^(-lᵢ) ≤ 1 (where lᵢ is the codeword length for symbol i)

The optimal codeword length for symbol xᵢ is:

l(xᵢ) = ⌈-log₂(p(xᵢ))⌉ = ⌈I(xᵢ)⌉ bits

This means the information content directly tells you the ideal code length — symbols carrying 3.2 bits of information should ideally have 3.2-bit codes (achievable with arithmetic coding) or 4-bit codes (Huffman rounding).

Mutual Information: Shared Information

When two events are not independent, learning one tells you something about the other. The mutual information between observing x and y is:

I(x;y) = log₂(p(x|y)/p(x)) = I(x) - I(x|y)

This measures how much the information content of x is reduced by knowing y. If x and y are independent, I(x;y) = 0. If knowing y completely determines x, then I(x;y) = I(x).

Real-World Applications

Data compression: Information content determines the minimum bits needed per symbol. ZIP, JPEG, and MP3 all exploit the fact that predictable data carries less information.

Communication system design: Channel capacity equals the maximum average mutual information — directly built on information content concepts.

Cryptography: Maximum security requires maximum entropy (unpredictability) in keys. A key where some bits are predictable (low information content) weakens encryption.

Machine learning: Cross-entropy loss measures how well a model predicts data, directly using information content to penalize wrong predictions.

Key Takeaways

1. Information content I(x) = -log₂(p(x)) measures the surprise or "information gain" from learning that event x occurred.
2. Rare events carry more information; certain events carry zero — formalizing the intuition that surprise equals information.
3. The logarithm is the unique function ensuring information from independent events adds together.
4. One bit is the information gained from one fair coin flip — the fundamental unit of digital information.
5. Information content determines optimal codeword lengths: I(x) bits of information ideally requires I(x) bits to encode.
6. Entropy (average information content) sets the fundamental limit on data compression and channel capacity.