Maximum entropy over 8 symbols — Shannon entropy
exam standardthe upper bound for a given alphabet
Answer
3.0000 bits
Why this example is worth doing
For a fixed alphabet size, entropy is maximised when every symbol is equally likely, giving log₂n. Any imbalance reduces it. The page states this as the bound it is — entropy over n symbols cannot exceed log₂n — which is what makes entropy a measure of how far a distribution is from uniform, and connects it to the redundancy that compression exploits.
Try your own input in the Shannon entropy. Bits per symbol for any distribution, with the surprisal of each symbol shown.
How the answer is reached
Shannon entropy of 8 symbols
8 symbols, 8 total occurrences. Divide by the total to get probabilities, then take −Σ p·log₂ p.
| Symbol | p | Surprisal −log₂ p (bits) | Contribution p·(−log₂ p) |
|---|---|---|---|
| a | 0.125000 | 3.000000 | 0.375000 |
| b | 0.125000 | 3.000000 | 0.375000 |
| c | 0.125000 | 3.000000 | 0.375000 |
| d | 0.125000 | 3.000000 | 0.375000 |
| e | 0.125000 | 3.000000 | 0.375000 |
| f | 0.125000 | 3.000000 | 0.375000 |
| g | 0.125000 | 3.000000 | 0.375000 |
| h | 0.125000 | 3.000000 | 0.375000 |
H3.000000 bits/symbol— the probability-weighted mean of the surprisal column
Total information24.000000 bits— 8 symbols × H
Maximum possible H3.000000 bits/symbol— log₂(8), reached only by the uniform distribution
H / log₂(m)1.000000— how close to uniform this source is
A symbol of probability 0 contributes exactly 0: the convention 0·log₂0 = 0 is a definition (the limit as p → 0), not an approximation. Evaluating it instead returns NaN, which is the usual bug on this page.
Source: C. E. Shannon, Bell System Technical Journal 27:379–423 (1948), Theorem 2