信息熵与交叉熵Entropy & Cross-Entropy
如何同时描述结果的不确定性与预测概率分配的代价？How can we describe both uncertainty in outcomes and the cost of a predicted distribution?
前置知识：Prerequisites: B02：对数；B15：离散分布；B18：似然与损失。B02: logarithms; B15: discrete distributions; B18: likelihood and loss.
复习前置：Review prerequisites: B02 · 指数与对数：增长和尺度Exponentials & Logarithms: Growth and Scale · B15 · 随机变量与离散分布Random Variables & Discrete Distributions · B18 · 从似然到损失函数From Likelihood to Loss Functions
学完后你能：After this lesson you can: 用统一对数底数计算熵与交叉熵，并识别无限代价。Compute entropy and cross-entropy with a consistent log base and identify infinite cost.
动手观察Explore
调整真实分布 p 与预测分布 q，对照熵和交叉熵；将预测概率设为零，观察无限代价。Adjust the true distribution p and prediction q, comparing entropy with cross-entropy; set a predicted probability to zero to see infinite cost.
从观察到解释From observation to explanation
结果有多不确定How uncertain are outcomes?
一个结果概率为 p 时，其信息量是 −ln p。分布的熵是各结果信息量的概率加权平均。确定的二分类分布熵为 0，公平的二分类分布熵为 ln 2。An outcome with probability p has information −ln p. Entropy is the probability-weighted average of outcome information. A certain binary distribution has entropy 0; a fair binary distribution has entropy ln 2.
预测分配的代价Cost of a prediction
交叉熵用真实概率 p 加权预测的 −ln q。若真实分布是 (1,0)，预测第一类概率为 q，代价简化为 −ln q；q 越小，代价越大。Cross-entropy weights the prediction’s −ln q by true probabilities p. If the true distribution is (1,0) and the predicted probability of the first class is q, cost is −ln q; smaller q gives higher cost.
统一单位Use one unit
真实分布与预测分布相同时，交叉熵等于熵。这里统一使用自然对数和 nat；若换成以 2 为底，单位为 bit，数值会改变。When true and predicted distributions match, cross-entropy equals entropy. We use natural logarithms and nats; base 2 gives bits and different numerical values.
公式与符号Formula and symbols
pᵢ 是真实类别 i 的概率，qᵢ 是预测概率；两组各自非负且和为 1。H(p) 是熵，H(p,q) 是交叉熵；本章使用自然对数，单位为 nat。pᵢ is the true probability of class i and qᵢ its predicted probability; each distribution is nonnegative and sums to 1. H(p) is entropy and H(p,q) cross-entropy. This lesson uses natural logarithms, measured in nats.
留意这个区别Watch for this
约定 0 ln 0 为 0；如果真实概率大于 0 而预测概率为 0，理论交叉熵为无穷大。实际代码中的截断是数值近似。Use the convention 0 ln 0 = 0. If a true probability is positive and its predicted probability is zero, theoretical cross-entropy is infinite. Clipping in code is a numerical approximation.
继续探索Explore further
继续学习算法Continue to an algorithm
课后练习
检验你的理解