Hamilton-Jacobi-Bellman Equation

Let us assume we are trying to minimize the total cost $$J(x(t_0),u(\tau ),t_0)=\Phi (x(t_f))+{\int }_{t_0}^{t_f}l(x(\tau ),u(\tau ),\tau )d\tau .$$ of a continuous-time system’s trajectory $x(\tau )$, $\tau \in [t_0,t_f]$ with dynamics in the form $$\dot{x}(t)=f(x(t),u(t),t),$$ starting from the state $x(t_0)={\tilde{x}}_0$.

The concept of the total cost can be generalized for any $t\in [t_0,t_f]$ to a cost-to-go $$J(x(t),u(\tau ),t)=\Phi (x(t_f))+{\int }_t^{t_f}l(x(\tau ),u(\tau ),\tau )d\tau ,$$ for which we may define a value function $$\begin{array}{rl}V(x(t),t)=\underset{u(\tau )}{\min }& J(x(t),u(\tau ),t),\tau \in [t,t_f),\\ \text{s.t.}& \dot{x}(\tau )=f(x(\tau ),u(\tau ),\tau ),\forall \tau \in [t,t_f]\end{array}$$ and optimal control policy $$\begin{array}{rl}{u}^{\ast }(\tau )=\underset{u(\tau )}{\mathrm{argmin}}& J(x(t),u(\tau ),t),\tau \in [t,t_f),\\ \text{s.t.}& \dot{x}(\tau )=f(x(\tau ),u(\tau ),\tau ),\forall \tau \in [t,t_f]\end{array}$$ the application of which results in the system following an optimal trajectory $x^{\ast }(\tau )$, $\tau \in (t,t_f]$.

For the previously defined value function the Hamilton-Jacobi-Bellman (HJB) equation can be derived¹ as $$-\frac{\partial V}{\partial t}(x(t),t)=\underset{u(t)}{\min }\left(l(x(t),u(t),t)+{\left(\frac{\partial V}{\partial x}\right)}^{\top }(x(t),t)f(x(t),u(t),t)\right).$$

1. An overview of the derivation is presented by Steven Brunton in one of his videos Brunton2022-HJB also available here. As a note, there is a small mistake, acknowledged by the presenter in the comments, at 9:11 of the video where “$0$” should be replaced with “$t$”. ↩