Principle of maximum entropy
Also known as: entropy maximization · entropy maximisation
- 1Selects the distribution that maximizes Shannon entropy given specific constraints.
- 2Produces the least committal distribution compatible with known information.
- 3Used extensively in Bayesian statistics for objective inference.
- 4Avoids introducing unsupported structure or assumptions into a model.
The principle of maximum entropy states that, among all probability distributions consistent with a given set of constraints, the distribution that maximizes Shannon entropy should be selected.
Connections1
Principle for statistical inference
This principle states that the probability distribution which best represents the current state of knowledge is the one with the largest entropy.
The principle of maximum entropy is a foundational concept in Bayesian statistics used for selecting a probability distribution. It dictates that among all distributions that satisfy a given set of constraints, one should choose the distribution that maximizes Shannon entropy.
By selecting the distribution with the highest entropy, this principle ensures that the model is the least committal possible. It introduces no additional structure or assumptions beyond what is logically required by the available information, making it a standard approach for objective statistical inference.