Discrete-Time Linear-Quadratic Regulator

Infinite horizon

For a linear time-invariant discrete-time system $$x_{k+1}=\underset{f(x_k,u_k,k)}{\underbrace{A{x}_k+B{u}_k}}$$ and a quadratic total cost $$J(x_0,\{u_i{\}}_{i=0}^{\infty },0)=\sum _{i=0}^{\infty }\underset{l(x_i,u_i,i)}{\underbrace{x_i^{\top }Q{x}_i+u_i^{\top }R{u}_i}},Q⪰0,R\succ 0$$ of its trajectory $\{x_i{\}}_{i=0}^{\infty }$ the optimal controller can be derived based on the assumption that the value function takes the form $$V(x_k,k)=x_k^{\top }S{x}_k,S\succ 0.$$

When substituted into the Bellman Equation along with the system's dynamics we attain $$x_k^{\top }S{x}_k=\underset{u_k}{\min }\left\{x_k^{\top }Q{x}_k+u_k^{\top }R{u}_k+(A{x}_k+B{u}_k{)}^{\top }S(A{x}_k+B{u}_k)\right\}.\text{\hspace{1em}(1)}$$ To find the minimum, we may take the gradient of its argument (which is by design quadratic and convex) with respect to $u_k$, set it to zero and find the solution (optimal control input) $$u_k^{\ast }=-{\left(R+B^{\top }SB\right)}^{-1}{B}^{\top }SA{x}_k.$$

The input can then be substitued back into (1). As the equation must hold for all $x_k$, through basic manipulations we then attain the discrete-time algebraic Riccati equation (DARE) $$S=Q+A^{\top }SA-A^{\top }SB{\left(B^{\top }SB+R\right)}^{-1}{B}^{\top }SA,S\succ 0.$$

Finite horizon

For a linear time-invariant discrete-time system $$x_{k+1}=\underset{f(x_k,u_k,k)}{\underbrace{A{x}_k+B{u}_k}}$$ and a quadratic total cost $$J(x_0,\{u_i{\}}_{i=0}^N,0)=\underset{\Phi (x_N)}{\underbrace{x_N^{\top }{Q}_N{x}_N}}+\sum _{i=0}^{N-1}\underset{l(x_i,u_i,i)}{\underbrace{x_i^{\top }Q{x}_i+u_i^{\top }R{u}_i}},Q_N⪰0,Q⪰0,R\succ 0$$ of its trajectory $\{x_i{\}}_{i=0}^N$ the optimal controller can be derived based on the assumption that the value function takes the form $$V(x_k,k)=x_k^{\top }{S}_k{x}_k,S_k\succ 0.$$ When substituted into the Bellman Equation along with the system's dynamics we attain $$x_k^{\top }{S}_k{x}_k=\underset{u_k}{\min }\left\{x_k^{\top }Q{x}_k+u_k^{\top }R{u}_k+(A{x}_k+B{u}_k{)}^{\top }{S}_{k+1}(A{x}_k+B{u}_k)\right\}.\text{\hspace{1em}(1)}$$ To find the minimum, we may take the gradient of its argument (which is by design quadratic and convex) with respect to $u_k$, set it to zero and find the solution (optimal control input) $$u_k^{\ast }=-{\left(R+B^{\top }{S}_{k+1}B\right)}^{-1}{B}^{\top }{S}_{k+1}A{x}_k.$$

The input can then be substitued back into (1). As the equation must hold for all $x_k$, through basic manipulations we then attain the discrete-time dynamic Riccati equation (DDRE) $$S_k=Q+A^{\top }{S}_{k+1}A-A^{\top }{S}_{k+1}B{\left(B^{\top }{S}_{k+1}B+R\right)}^{-1}{B}^{\top }{S}_{k+1}A,S_N=Q_N$$ for a finite horizon $N\in \mathbb{N}$.