Hamilton-Jacobi-Bellman Equation

Let us assume we are trying to minimize the total cost $$J(\mathbf{x}(t_0),\mathbf{u}(\tau ),t_0)=\Phi (\mathbf{x}(t_f))+{\int }_{t_0}^{t_f}l(\mathbf{x}(\tau ),\mathbf{u}(\tau ),\tau )d\tau .$$ of a continuous-time system's trajectory $\mathbf{x}(\tau )$, $\tau \in ⟨t_0,t_f⟩$ with dynamics in the form $$\dot{\mathbf{x}}(t)=\mathbf{f}(\mathbf{x}(t),\mathbf{u}(t),t),$$ starting from the state $\mathbf{x}(t_0)={\tilde{\mathbf{x}}}_0$.

The concept of the total cost can be generalized for any $t\in ⟨t_0,t_f⟩$ to a cost-to-go $$J(\mathbf{x}(t),\mathbf{u}(\tau ),t)=\Phi (\mathbf{x}(t_f))+{\int }_t^{t_f}l(\mathbf{x}(\tau ),\mathbf{u}(\tau ),\tau )d\tau ,$$ for which we may define a value function $$V(\mathbf{x}(t),t)=\underset{\mathbf{u}(\tau )}{\min }J(\mathbf{x}(t),\mathbf{u}(\tau ),t),$$ and optimal control policy $${\mathbf{u}}^{\ast }(\tau )=\underset{\mathbf{u}(\tau )}{\mathrm{argmin}}J(\mathbf{x}(t),\mathbf{u}(\tau ),t),\tau \in ⟨t,t_f)$$ the application of which results in the system following an optimal trajectory ${\mathbf{x}}^{\ast }(\tau )$, $\tau \in (t,t_f⟩$.

For the previously defined value function the Hamilton-Jacobi-Bellman (HJB) equation can be derived¹ as $$-\frac{\partial V}{\partial t}(\mathbf{x}(t),t)=\underset{\mathbf{u}(t)}{\min }\left(l(\mathbf{x}(t),\mathbf{u}(t),t)+{\left(\frac{\partial V}{\partial \mathbf{x}}\right)}^{\top }(\mathbf{x}(t),t)\mathbf{f}(\mathbf{x}(t),\mathbf{u}(t),t)\right).$$