Kullback–Leibler divergence
Also known as: information divergence · information gain · relative entropy · KL divergence · KLIC
- 1Acts as a statistical distance measure between two probability distributions.
- 2Quantifies the difference between an approximating distribution Q and a true distribution P.
- 3Widely utilized in the field of mathematical statistics.
- 4Provides a formal way to measure the information discrepancy between two models.
In mathematical statistics, the Kullback–Leibler (KL) divergence, denoted , is a type of statistical distance: a measure of how much an approximating probability distribution Q is different from a true probability distribution P.
Connections1
Measure of difference between probability distributions
This metric quantifies how one probability distribution differs from a second, reference distribution, building directly upon the concept of Shannon entropy.
The Kullback–Leibler divergence, often denoted as KL divergence, is a fundamental measure used in mathematical statistics. It functions as a type of statistical distance that quantifies how much one probability distribution differs from a reference or true distribution.
Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as T o {\displaystyle T_{o}} times the Kullback–Leibler divergenceWikimedia Commons: File:ArgonKLdivergence.pngIllustration of the relative entropy for two normal distributions. The typical asymmetry is clearly visible.Wikimedia Commons: File:KL-Gauss-Example.pngTwo distributions to illustrate relative entropyWikimedia Commons: File:Kullback–Leibler_distributions_example_1.svg Specifically, it measures the discrepancy between an approximating distribution Q and a true distribution P. By calculating this divergence, researchers can assess the information loss or the degree of similarity when one distribution is used to represent another in statistical modeling.