不动点、收缩与收敛证明Fixed Points, Contraction & Convergence
为什么折扣贝尔曼备份会控制两个价值函数之间的最大误差？Why does a discounted Bellman backup control the largest difference between two value functions?
前置知识：Prerequisites: R02：折扣回报与贝尔曼递推。R02: discounted returns and Bellman recursion.
复习前置：Review prerequisites: R02 · 折扣回报与贝尔曼递推Discounted Returns & Bellman Recursion
学完后你能：After this lesson you can: 定义最大误差，执行真实贝尔曼备份并解释收缩上界。Define maximum error, perform actual Bellman backups and explain the contraction bound.
动手观察Explore
对两组初始价值分别执行真实的最优贝尔曼备份，比较新最大差与折扣上界。Apply actual optimal Bellman backups to two value vectors and compare the new maximum difference with the discounted bound.
从观察到解释From observation to explanation
先定义最大误差Define the maximum error first
比较两组价值 U、V 时，最大误差 δ=maxₛ|U(s)−V(s)| 同时控制每个状态的差。概率加权平均不会把一个被 δ 控制的差放大。For two value vectors U and V, maximum error δ=maxₛ|U(s)−V(s)| bounds the difference at every state. A probability-weighted average cannot enlarge a difference bounded by δ.
真实贝尔曼备份An actual Bellman backup
本实验从奖励与下一状态直接算最优备份：T(V)₀=max(1+γV₁, γV₀)，T(V)₁=max(γV₀, 2+γV₁)。没有预先给定最优价值。This lab computes the optimal backup directly from rewards and next states: T(V)₀=max(1+γV₁, γV₀), T(V)₁=max(γV₀, 2+γV₁). No optimal value is supplied in advance.
最大差至多缩小The maximum difference shrinks by at most the factor
max 运算的两个结果之差不超过各候选差的最大值；结合概率权重和折扣，就得到新最大差不超过 γδ。有限状态、动作、有界奖励且 γ<1 时，精确值迭代收敛到唯一不动点。The difference of two maxima is at most the largest difference between their candidates. Combining that with probability weights and discount gives a new maximum difference at most γδ. With finite states and actions, bounded rewards and γ<1, exact value iteration converges to a unique fixed point.
公式与符号Formula and symbols
T 是贝尔曼最优算子，U、V 是两组价值；∥U−V∥∞ 表示所有状态绝对差的最大值，γ 是折扣因子。T is the Bellman optimality operator, U and V are value vectors, ∥U−V∥∞ is the maximum absolute difference over states, and γ is the discount factor.
留意这个区别Watch for this
不等式是“至多”缩小，并非每一步恰好乘 γ；采样更新或函数近似需要额外条件。The inequality says “at most” the factor, not exactly γ at every step. Sampled updates or function approximation require additional conditions.
继续探索Explore further
课后练习
检验你的理解