Surprisal of each symbol in "mississippi" — Shannon entropy
edge casethe term-by-term breakdown
Answer
i=1.4594 m=3.4594 p=2.4594 s=1.4594
Why this example is worth doing
The per-symbol surprisals, −log₂p, before they are weighted and summed. The rare m carries the most information at over three and a half bits while the common i and s carry less than one and a half. Displaying the breakdown is what turns entropy from a formula into an explanation, and it makes the connection to Huffman immediate: the surprisal is the ideal code length, and Huffman is the best whole-bit approximation to it.
Try your own input in the Shannon entropy. Bits per symbol for any distribution, with the surprisal of each symbol shown.
How the answer is reached
Shannon entropy of 4 symbols
4 symbols, 11 total occurrences. Divide by the total to get probabilities, then take −Σ p·log₂ p.
| Symbol | p | Surprisal −log₂ p (bits) | Contribution p·(−log₂ p) |
|---|---|---|---|
| i | 0.363636 | 1.459432 | 0.530702 |
| m | 0.090909 | 3.459432 | 0.314494 |
| p | 0.181818 | 2.459432 | 0.447169 |
| s | 0.363636 | 1.459432 | 0.530702 |
H1.823068 bits/symbol— the probability-weighted mean of the surprisal column
Total information20.053748 bits— 11 symbols × H
Maximum possible H2.000000 bits/symbol— log₂(4), reached only by the uniform distribution
H / log₂(m)0.911534— how close to uniform this source is
A symbol of probability 0 contributes exactly 0: the convention 0·log₂0 = 0 is a definition (the limit as p → 0), not an approximation. Evaluating it instead returns NaN, which is the usual bug on this page.
Source: C. E. Shannon, Bell System Technical Journal 27:379–423 (1948), Theorem 2