Shannon Entropy Calculator

Entropy is the average number of bits you need to identify one symbol drawn from a known distribution. It is a property of the distribution, not of any particular message.

Two consequences follow immediately, and both catch people out. Shuffling a message does not change its entropy, because the counts do not change. And a message can be highly compressible while its first-order entropy is high, because entropy as computed here knows nothing about the order the symbols arrive in — see the last section.

The information carried by one occurrence of a symbol is its surprisal, I(x) = log₂(1/p) = −log₂ p. An event of probability ½ carries exactly one bit; an event of probability ¼ carries two; an event that is certain carries none, because you learn nothing by observing what you already knew.

Surprisal is also the ideal code length for that symbol: a symbol you expect one time in eight deserves three bits and no more. The table above prints it per symbol, next to the whole-bit length a real prefix code has to round it to.

Entropy is the probability-weighted mean of surprisal: H = Σ pᵢ·I(xᵢ) = −Σ pᵢ log₂ pᵢ. Every step of that sum for the current input is above — normalise, take a logarithm per symbol, weight, add — with the numbers substituted rather than a bare formula printed. For MISSISSIPPI it lands at H = 1.8230679823 bits per symbol, and 11 symbols carry 20.0537 bits in total.

The form is not a modelling choice. Shannon’s Theorem 2 shows it is forced: “The only H satisfying the three above assumptions is of the form H = −K Σ pᵢ log pᵢ” — where the assumptions are that H is continuous in the pᵢ, that it increases with m for a uniform source, and that it is unchanged when a choice is broken into successive choices. The constant K “merely amounts to a choice of a unit of measure”; K = 1 with base-2 logarithms gives bits.

Never assume the input sums to 1. Counts are divided by their total before any log is taken. The normalisation is printed above the table rather than performed silently, because a tool that accepts both counts and probabilities and never says which it assumed is unusable for checking an answer.

Changing the base of the logarithm changes the unit and nothing else, so the toggle above converts rather than recomputes: H in another base is H in bits multiplied by a constant. The unit name is printed with every figure, because “H = 1.26” means different things in a statistics paper and a coding-theory paper.

Entropy is never negative and never exceeds log₂ m for an alphabet of m symbols. H = 0 happens exactly when the source is deterministic — one symbol has probability 1 — and H = log₂ m happens exactly when the distribution is uniform. Everything else sits strictly between, and the ratio H / H max is how far along that range a distribution is.

For MISSISSIPPI, m = 4, so H max = 2.0000000000 bits per symbol and H is 91.1534% of it. The distribution is close to uniform in the sense the ratio measures, and the two symbols that make it uneven — the single M against four Is — are exactly the ones a code can exploit.

A message of N symbols from this distribution carries N × H bits of information — 20.0537 bits here. That figure is a floor for an ideal code, not a file size: it assumes the decoder already knows the distribution, and it does not include the cost of transmitting the codebook, the file header or anything else.

It is also not an integer, and cannot be rounded down. A real encoding of these 11 symbols needs at least ⌈N × H⌉ bits, and a prefix code needs more than that — 21 bits here, as the next section explains.

A symbol of probability 0 contributes exactly 0 to the sum. That is a definition taken from the limit — p log₂ p → 0 as p → 0 — and not an evaluation: computing it directly gives 0 × (−∞) = NaN, so the zero case is handled before the logarithm is taken.

The consequence is worth stating, because it is the check that the convention is the right one: a distribution containing zero-weight symbols has exactly the same entropy as the same distribution with those symbols deleted. A symbol that never occurs tells you nothing when it does not occur.

The entropy figure means something because a code is measured against it. L is the average codeword length of the Huffman code for the same distribution, computed here by the same implementation that page uses, so the two can never disagree about the input.

For MISSISSIPPI: H = 1.8230679823, L = 1.9090909091, so H ≤ L holds with a redundancy of L − H = 0.0860229268 bits per symbol and an efficiency of H / L = 95.4940%. The upper bound holds too, with room to spare.

The bound is Shannon’s, not folklore. §9 of the 1948 paper: “The expansion is carried out to mₛ places, where mₛ is the integer satisfying: log₂(1/pₛ) ≤ mₛ < 1 + log₂(1/pₛ)”. That inequality, applied symbol by symbol and averaged, is where H ≤ L < H + 1 comes from. It is a statement about Shannon–Fano coding in the original paper; it holds for Huffman codes because a Huffman code is never worse.

Equality, L = H, happens exactly when every probability is a power of ½. Then each ideal length log₂(1/p) is already a whole number and there is nothing to round. The dyadic distribution A 8, B 4, C 2, D 2 gives H = L = 1.75 exactly; a uniform alphabet of 4 gives 2 and of 8 gives 3. Every other distribution pays something for the rounding.

The upper bound can be very loose. With weights 96, 1, 1, 1, 1 the entropy is 0.3222921891 bits per symbol while the Huffman code averages 1.08 — 235% above the floor. No prefix code can spend less than one whole bit on a symbol, and this source wants about 0.06 bits on its dominant symbol. Arithmetic coding is what closes that gap.

There is exactly one case where the strict upper bound fails, and it is a convention rather than a defect. For a single-symbol alphabet H = 0, and this toolkit emits a one-bit codeword because a zero-length codeword cannot be transmitted — so L = 1 and L < H + 1asks for 1 < 1, which is false. Under the alternative zero-length convention L = H = 0 and the bound holds trivially. The page reports the failure rather than rounding it away; a test that asserted the bound universally would be asserting something false.

Shannon’s H and Boltzmann’s S = k log W have the same shape, and the connection between them is real and well studied. This calculator answers the information-theoretic question only: how many bits per symbol a source produces. Nothing on this page is a statement about heat, disorder or any physical system, and the numbers here should not be carried into a physics problem.

This is first-order entropy: symbols are assumed independent, and only their counts are used. Real sources have structure, and conditional entropy H(X|Y) — how surprised you are by the next symbol given the previous one — is lower, often far lower.

English is the standard illustration. The first-order entropy of English letter frequencies is about 4.18 bits per letter against a maximum of log₂ 26 = 4.7004 bits, so letter frequency alone saves about half a bit. But letters are not independent: q is followed by u, this everywhere, and Shannon’s own estimate for English text with context taken into account is closer to one bit per letter. A first-order entropy of 4.18 bits per letter does not mean English needs 4.18 bits per letter; it means an encoder that ignores everything except letter frequency needs that much.

The same caveat applies to every number on this page, and it is why a compressor that models context beats one that only counts symbols. It is not a limitation of the formula — it is what the formula was asked.

• Logarithms are base 2 unless the unit toggle says otherwise, and the unit is named with every figure.
• p is a probability in [0, 1]; counts are normalised by their total before any logarithm is taken, and the normalisation is shown.
• 0 · log 0 is defined as 0, by the limit.
• I(x) = −log₂ p(x) is the surprisal, or self-information, of one symbol, and H is its probability-weighted mean.
• L is the average codeword length of the Huffman code over the same distribution, computed by the same code as the Huffman page, with a one-symbol alphabet given the one-bit code 0.
• Entropies are displayed to ten decimal places and computed in a form that is exact for dyadic distributions.
• Whitespace is counted as a symbol unless the switch above is turned off, because that choice changes every number on the page.

• C. E. Shannon, “A Mathematical Theory of Communication” (opens in a new tab), Bell System Technical Journal27:379–423 (July 1948) and 623–656 (October 1948) — Theorem 2 for the form of H, quoted above, and §9 for the log₂(1/pₛ) ≤ mₛ < 1 + log₂(1/pₛ) bound the source-coding section rests on.
• D. A. Huffman, “A Method for the Construction of Minimum-Redundancy Codes” (opens in a new tab), Proceedings of the IRE 40(9):1098–1101, September 1952 — the code L is measured from, worked in full on the Huffman coding calculator.
• R. E. Lewand, Cryptological Mathematics, Mathematical Association of America, 2000 — the tabulated English letter frequencies behind the 4.18 bits per letter figure. They sum to 99.999% rather than 100%, and this page normalises by the actual total rather than pretending otherwise.