Text "mississippi" — Shannon entropy
exam standardempirical entropy of a string
Answer
1.8231 bits
Why this example is worth doing
The same string as the Huffman example, so the two tools can be compared directly on one input. The page is careful to call this the empirical zeroth-order entropy: it treats each character as independent, which is a poor model for English, where q is followed by u almost always. Real text has much lower entropy once context is taken into account, and the page says so rather than overclaiming.
Try your own input in the Shannon entropy. Bits per symbol for any distribution, with the surprisal of each symbol shown.
How the answer is reached
Shannon entropy of 4 symbols
4 symbols, 11 total occurrences. Divide by the total to get probabilities, then take −Σ p·log₂ p.
| Symbol | p | Surprisal −log₂ p (bits) | Contribution p·(−log₂ p) |
|---|---|---|---|
| i | 0.363636 | 1.459432 | 0.530702 |
| m | 0.090909 | 3.459432 | 0.314494 |
| p | 0.181818 | 2.459432 | 0.447169 |
| s | 0.363636 | 1.459432 | 0.530702 |
H1.823068 bits/symbol— the probability-weighted mean of the surprisal column
Total information20.053748 bits— 11 symbols × H
Maximum possible H2.000000 bits/symbol— log₂(4), reached only by the uniform distribution
H / log₂(m)0.911534— how close to uniform this source is
A symbol of probability 0 contributes exactly 0: the convention 0·log₂0 = 0 is a definition (the limit as p → 0), not an approximation. Evaluating it instead returns NaN, which is the usual bug on this page.
Source: C. E. Shannon, Bell System Technical Journal 27:379–423 (1948), Theorem 2