For a discrete random variable X with probability mass p, the Shannon entropy is H(X) = −Σ p(x) log p(x). Measures average uncertainty / information content in bits (log₂) or nats (ln). Non-negative; zero iff X is deterministic;…
Shannon entropy
Related concepts
- Random variable
- Shannon entropy (1948)
- Kraft-McMillan (1949)
- Hamming codes (1950)
- Kolmogorov complexity (1965)
- von Neumann entropy S(ρ) = −Tr(ρ log ρ) on qubit channels: Shannon-quantum bridge
- Fitts' law: MT = a + b · log₂(2A/W); ID = log₂(2A/W); Fitts' 1954 information-theoretic aiming
- Friston free-energy principle: F[q]=KL[q||p(z|x)]−log p(x); variational bound on surprise
- Wolpert positional information (French flag)