折扣回报与贝尔曼递推Discounted Returns & Bellman Recursion

回报给未来奖励打折Discount future rewards

折扣回报把下一步奖励、再下一步奖励等分别乘 1、γ、γ² 后相加。若每步奖励恒为 1 且 0≤γ<1，这个无穷和为 1/(1−γ)。Discounted return weights the next reward, the reward after that and so on by 1, γ, γ². If every reward is 1 and 0≤γ<1, the infinite sum is 1/(1−γ).

把后续回报折叠Fold the future into the next value

当前价值等于期望的下一步奖励加折扣后的下一状态价值。本实验状态 0 奖励为 1，下一状态以各 1/2 的概率为 0 或 1；状态 1 奖励为 0 并留在状态 1。Current value equals expected next reward plus discounted next-state value. In this lab state 0 yields reward 1 and transitions to state 0 or 1 with probability 1/2 each; state 1 yields reward 0 and remains in state 1.

一次概率加权备份One probability-weighted backup

因此更新为 V′(0)=1+γ(0.5V(0)+0.5V(1))，V′(1)=γV(1)。这里评估的是一个固定策略；最优算子还会在动作间取最大值。Thus V′(0)=1+γ(0.5V(0)+0.5V(1)) and V′(1)=γV(1). This evaluates a fixed policy; an optimality operator additionally maximizes over actions.

公式与符号Formula and symbols

V(s) 是状态 s 的期望折扣回报，Rₜ₊₁ 是下一步奖励，Sₜ₊₁ 是下一状态，γ∈[0,1) 是折扣因子，E 表示条件期望。V(s) is expected discounted return from state s; Rₜ₊₁ is the next reward, Sₜ₊₁ the next state, γ∈[0,1) the discount factor, and E the conditional expectation.

继续探索Explore further

• 继续学习：MDP 与贝尔曼方程Continue with MDPs and Bellman equations

继续学习算法Continue to an algorithm

强化学习中的贝尔曼方程Bellman equations in reinforcement learning