Uniform over four symbols — Shannon entropy
corelog₂n for a uniform distribution
Answer
2.0000 bits
Why this example is worth doing
A uniform distribution over n symbols has entropy log₂n exactly, which is two bits for four symbols. The page pairs it with the matching Huffman example, where the same distribution produced two-bit codes and no compression, and notes that the two results agreeing is not a coincidence — a uniform distribution is the case where the entropy bound is achieved exactly with whole bits.
Try your own input in the Shannon entropy. Bits per symbol for any distribution, with the surprisal of each symbol shown.
How the answer is reached
Shannon entropy of 4 symbols
4 symbols, 4 total occurrences. Divide by the total to get probabilities, then take −Σ p·log₂ p.
| Symbol | p | Surprisal −log₂ p (bits) | Contribution p·(−log₂ p) |
|---|---|---|---|
| a | 0.250000 | 2.000000 | 0.500000 |
| b | 0.250000 | 2.000000 | 0.500000 |
| c | 0.250000 | 2.000000 | 0.500000 |
| d | 0.250000 | 2.000000 | 0.500000 |
H2.000000 bits/symbol— the probability-weighted mean of the surprisal column
Total information8.000000 bits— 4 symbols × H
Maximum possible H2.000000 bits/symbol— log₂(4), reached only by the uniform distribution
H / log₂(m)1.000000— how close to uniform this source is
A symbol of probability 0 contributes exactly 0: the convention 0·log₂0 = 0 is a definition (the limit as p → 0), not an approximation. Evaluating it instead returns NaN, which is the usual bug on this page.
Source: C. E. Shannon, Bell System Technical Journal 27:379–423 (1948), Theorem 2