KL 散度与分布比较KL Divergence & Comparing Distributions
怎样衡量用预测分布代替真实分布带来的额外代价？How can we measure the extra cost of using a predicted distribution in place of the true one?
前置知识：Prerequisites: B19：信息熵与交叉熵。B19: entropy and cross-entropy.
复习前置：Review prerequisites: B19 · 信息熵与交叉熵Entropy & Cross-Entropy
学完后你能：After this lesson you can: 在统一单位下计算 KL，解释方向与支持范围。Calculate KL in a consistent unit and explain direction and support.
动手观察Explore
调整二类真实分布和预测分布，对照熵、交叉熵、KL 与各类贡献。Adjust binary true and predicted distributions and compare entropy, cross-entropy, KL and class contributions.
从观察到解释From observation to explanation
从交叉熵出发Start from cross-entropy
交叉熵 H(p,q) 衡量按真实分布 p 出现的结果，用预测分布 q 编码时的平均代价。减去 p 自身熵 H(p)，剩下的就是 KL 散度。Cross-entropy H(p,q) measures average cost under true distribution p using prediction q. Subtract the entropy H(p) of p itself and the remainder is KL divergence.
整体非负，单项可负The total is nonnegative; individual terms may be negative
当 p=q 时 KL 为 0；对合法归一化分布，整体 KL 非负。但单个类别项 pᵢ ln(pᵢ/qᵢ) 可以为负，不应逐项截断成零。KL is zero when p=q and is nonnegative overall for valid normalized distributions. An individual term pᵢ ln(pᵢ/qᵢ) can be negative and must not be clipped to zero.
注意方向与支持范围Direction and support matter
一般 DKL(p∥q) 不等于 DKL(q∥p)。若 pᵢ>0 而 qᵢ=0，理论代价为无穷大；若 pᵢ=0，按极限约定该项为 0。In general DKL(p∥q) differs from DKL(q∥p). If pᵢ>0 while qᵢ=0, theoretical cost is infinite; if pᵢ=0, that term is defined as 0 by a limit convention.
公式与符号Formula and symbols
pᵢ 和 qᵢ 分别是真实分布与预测分布在类别 i 的概率，D 表示 KL 散度；本章统一使用自然对数，单位 nat。pᵢ and qᵢ are probabilities of class i in the true and predicted distributions; D denotes KL divergence. We use natural logs and nats throughout.
留意这个区别Watch for this
信息量必须使用同一对数底数才能相减；数值 epsilon 截断得到的有限值只是近似。Information values must use the same log base before subtraction; a finite result from epsilon clipping is only a numerical approximation.
继续探索Explore further
课后练习
检验你的理解