A Mathematical Theory of Communication

Round 1 — Core Claim & Mental Model

The problem it solves

Before 1948 engineers tuned radios and telephone lines by intuition; there was no theory of how much could be sent over a noisy medium, or how compact a message could be made. Shannon reframed communication as a probabilistic problem: the receiver already has a model of what the source might say, and information is whatever reduces that uncertainty.

Mental model

Information ≈ surprise. A message you could have predicted carries nothing; a surprising one carries a lot. Entropy is your average surprise per symbol — the irreducible number of yes/no questions needed to pin down the next symbol. A fair coin: 1 bit. A coin that lands heads 99% of the time: ~0.08 bits, because you're rarely surprised.

The two theorems, in one breath

Source coding: you cannot compress below the entropy \(H\) without losing information — but you can get arbitrarily close. Channel coding: below capacity \(C\), error can be driven to zero; above \(C\), it cannot. Crucially, the two are separable: compress optimally, then protect optimally, and you lose nothing.

Round 2 — Mathematical Model

Entropy

\[ H(X) = -\sum_{i} p_i \log_2 p_i \quad\text{[bits/symbol]} \]

Shannon derived this form axiomatically: any "uncertainty" measure that is continuous in the \(p_i\), monotonic for uniform distributions, and additive over independent choices must be \(-K\sum p_i\log p_i\). The bit falls out by choosing \(\log_2\).

Mutual information & capacity

\[ I(X;Y)=H(X)-H(X\mid Y),\qquad C=\max_{p(x)} I(X;Y) \]

Capacity is the best mutual information achievable over all input distributions — the most the output tells you about the input, optimized over how you use the channel.

The Shannon–Hartley law (key special case)

\[ C = B\,\log_2\!\left(1+\frac{S}{N}\right)\quad\text{[bits/s]} \]

For a bandwidth-\(B\) Gaussian channel with signal/noise power ratio \(S/N\). This single equation governs every link budget in the Networks domain — from 5G NR to fiber.

Derivation sketch (channel coding, AEP)

By the Asymptotic Equipartition Property, for large block length \(n\) the source emits one of \(\approx 2^{nH}\) "typical" sequences, each almost equiprobable. Random codebooks of \(2^{nR}\) words with \(Rexistence proof — it says good codes exist, not how to build them.

Complexity / cost

The catch hidden in "\(n\to\infty\)": approaching capacity needs long blocks ⇒ long codes ⇒ latency and decoding cost. This rate–latency tension is exactly the fronthaul/edge constraint that dominates modern RAN design.

Invariants & limiting cases

\(0\le H(X)\le \log_2|\mathcal X|\), maximized by the uniform distribution (max ignorance), zero for a deterministic source. As \(S/N\to\infty\), \(C\to\infty\) (but only logarithmically — doubling power adds ~1 bit, while doubling bandwidth nearly doubles \(C\): bandwidth is the cheaper lever). As \(S/N\to 0\), \(C\to 0\): no signal, no information.

How It Works & Visual Diagrams

Round 3 — Limitations & Community Response

Existence, not construction. Shannon proved capacity-achieving codes exist via random coding; he gave no practical decoder. Closing the gap took 45+ years — Turbo codes (1993) and rediscovered LDPC (Gallager 1962 → 1990s) finally operate within a fraction of a dB of the limit; Polar codes (Arıkan 2009) are the first provably capacity-achieving construction, now in 5G control channels.

Asymptotic blind spot. The clean theorems assume \(n\to\infty\). Real systems use short blocks under latency budgets; finite-blocklength information theory (Polyanskiy–Poor–Verdú 2010) rewrites the limit with a penalty term and is the relevant theory for ultra-reliable low-latency (URLLC) 5G.

Semantics excluded by design. Shannon explicitly bracketed meaning — "the semantic aspects are irrelevant to the engineering problem." Weaver's framing and today's "semantic communication" (transmit task-relevant meaning, not raw bits) is the live frontier reopening exactly what Shannon set aside.

Round 4 — AI × Networks Connection

This is arguably the strongest single bridge node in the KB — it sits in both directions.

Shannon → AI. Cross-entropy, the loss training almost every classifier and LLM, is literally the expected code length under a wrong model: \(H(p,q)=-\sum p\log q\). Minimizing it = building an optimal source code for the data. KL divergence, perplexity, the information bottleneck, and VAEs are all his calculus reused.

AI → Shannon. Deep learning now attacks the constructive gap he left open: neural decoders for LDPC/Polar, autoencoder-designed end-to-end physical layers, and learned compression that beats hand-built codecs. The 1948 bound is the referee these models are measured against.

Networks core. \(C=B\log_2(1+S/N)\) is the budget every scheduler, beamformer, and slice negotiates. ML for RAN (link adaptation, CQI prediction) is, at bottom, estimating how close to \(C\) it is safe to operate.

Verify — Credibility Check

Sanity-check the headline law: Wi-Fi at \(B=20\) MHz, \(S/N=30\) dB \((=1000)\) ⇒ \(C=20\times10^6\log_2(1001)\approx 199\) Mbit/s — the right order of magnitude for the PHY ceiling, consistent with real 802.11 rates after overhead. The theorems are mathematical and have been formally verified and universally adopted for 78 years; no empirical result has ever beaten \(C\) (every claim of "exceeding Shannon" has reduced to a redefinition of the channel). The single experiment that would break it — reliable transmission strictly above capacity on a well-characterized channel — has never been observed.