Continuous-Time Linear-Quadratic Regulator

Infinite horizon

For a linear time-invariant continuous-time system $$\dot{x}(t)=\underset{f(x(t),u(t),t)}{\underbrace{Ax(t)+Bu(t)}}$$ and a quadratic total cost $$J(x(t_0),u(\tau ),t_0)={\int }_{t_0}^{\infty }\underset{l(x(\tau ),u(\tau ),\tau )}{\underbrace{x(\tau {)}^{\top }Qx(\tau )+u(\tau {)}^{\top }Ru(\tau )}}d\tau ,$$ where $Q⪰0$ and $R\succ 0$, of its trajectory $x(\tau )$, $\tau \in (t_0,t_f⟩$ the optimal controller can be derived based on the assumption that the value function takes the form $$V(x(t),t)=x(t{)}^{\top }Sx(t),S\succ 0.$$ When substituted into the Hamilton-Jacobi-Bellman Equation along with the system's dynamics we attain $$0=\underset{u}{\min }\left\{x^{\top }Qx+u^{\top }Ru+2x^{\top }S\left(Ax+Bu\right)\right\},\text{\hspace{1em}(1)}$$ where $x=x(t)$ and $u=u(t)$. To find the minimum, we may take the gradient of its argument (which is by design quadratic and convex) with respect to $u(t)$, set it to zero and find the solution (optimal control input) $$u^{\ast }(t)=-R^{-1}{B}^{\top }Sx(t).$$

The input can then be substituted back into (1). As the equation must hold for all $x(t)$, through basic manipulations we then attain the continuous-time algebraic Riccati equation (CARE) $$0=Q-S{B}^{\top }{R}^{-1}BS+SA+A^{\top }S,S\succ 0.$$

Finite horizon

For a linear time-invariant continuous-time system $$\dot{x}(t)=\underset{f(x(t),u(t),t)}{\underbrace{Ax(t)+Bu(t)}}$$ and a quadratic total cost $$J(x(t_0),u(\tau ),t_0)=\underset{\Phi (x_N)}{\underbrace{x(t_f{)}^{\top }{Q}_{f}x(t_f)}}+{\int }_{t_0}^{t_f}\underset{l(x(\tau ),u(\tau ),\tau )}{\underbrace{x(\tau {)}^{\top }Qx(\tau )+u(\tau {)}^{\top }Ru(\tau )}}d\tau ,$$ where $Q_f⪰0$, $Q⪰0$, and $R\succ 0$, of its trajectory $x(\tau )$, $\tau \in (t_0,t_f⟩$ the optimal controller can be derived based on the assumption that the value function takes the form $$V(x(t),t)=x(t{)}^{\top }S(t)x(t),S(t)\succ 0.$$ When substituted into the Hamilton-Jacobi-Bellman Equation along with the system's dynamics we attain $$-x^{\top }\dot{S}x=\underset{u}{\min }\left\{x^{\top }Qx+u^{\top }Ru+2x^{\top }S\left(Ax+Bu\right)\right\},\text{\hspace{1em}(1)}$$ where $x=x(t)$ and $u=u(t)$. To find the minimum, we may take the gradient of its argument (which is by design quadratic and convex) with respect to $u(t)$, set it to zero and find the solution (optimal control input) $$u^{\ast }(t)=-R^{-1}{B}^{\top }S(t)x(t).$$

The input can then be substituted back into (1). As the equation must hold for all $x(t)$, through basic manipulations we then attain the continuous-time differential Riccati equation (CDRE) $$-\dot{S}(t)=Q-S(t)B^{\top }{R}^{-1}BS(t)+S(t)A+A^{\top }S(t)$$ for a finite horizon $t_f\in \mathbb{R}$. Supplemented with the boundary conditon $S(t_f)=Q_f$, the CDRE forms a initial value problem (IVP) which can be solved using numerical integration. Numerical errors in the integration process may lead to the loss of positive-semi-definiteness. To overcome this, instead of integrating $S(t)$ directly, we may use its factorized form $S(t)=P(t)P^{\top }(t)$ a.k.a. the "square-root form" where $$-\dot{P}(t)=\frac{1}{2}Q{P}^{-\top }(t)-\frac{1}{2}S(t)B{R}^{-1}{B}^{\top }P(t)+A^{\top }P(t).$$ As $P(t)$ must be invertible $Q_f$ must be (at least numerically) positive definite. Consequenlty we may use the cholesky factorization in order to form the boundary condition $$P(t_f)=L,Q_f=L{L}^{\top }.$$