Control Theory in Stochastic Systems

Control theory in stochastic systems deals with the analysis and design of controllers that can handle systems influenced by random disturbances. This field is crucial for managing systems where uncertainty plays a significant role, such as financial markets, communication networks, and robotic systems.

Basic Concepts

Stochastic Processes

A stochastic process is a collection of random variables indexed by time or space. For example, $ {X_t : t \geq 0} $ is a stochastic process if $ X_t $ is a random variable for each time $ t $.

State-Space Representation

A stochastic control system can be represented in state-space form as:

$$ \begin{align}
dx_t &= f(x_t, u_t) dt + g(x_t, u_t) dW_t, \
y_t &= h(x_t) + v_t,
\end{align} $$

where:

$ x_t $ is the state vector,
$ u_t $ is the control input,
$ y_t $ is the output,
$ f $ and $ g $ are known functions,
$ W_t $ is a Wiener process (or Brownian motion),
$ v_t $ is measurement noise.

Stochastic Differential Equations (SDEs)

The evolution of the state in a stochastic control system is governed by a stochastic differential equation (SDE):

$$ dx_t = f(x_t, u_t) dt + g(x_t, u_t) dW_t $$

Objective Function

The objective in stochastic control is to find a control policy $ u_t $ that minimizes a cost function. A common cost function is the expected value of an integral cost over a time horizon $ [0, T] $:

$$ J(u) = \mathbb{E} \left[ \int_0^T L(x_t, u_t) \, dt + \Phi(x_T)
\right] $$

where:

$ L(x_t, u_t) $ is the running cost,
$ \Phi(x_T) $ is the terminal cost.

Hamilton-Jacobi-Bellman (HJB) Equation

The optimal control policy can be found by solving the Hamilton-Jacobi-Bellman (HJB) equation, a partial differential equation given by:

$$ \begin{align} \frac{\partial V(x,t)}{\partial t} + \min_u \left\{ L(x,u) + \frac{\partial V(x,t)}{\partial x} f(x,u) \right. \\ \left. + \frac{1}{2} \text{Tr} \left( \frac{\partial^2 V(x,t)}{\partial x^2} g(x,u) g(x,u)^T \right) \right\} & = 0 \end{align} $$

with the boundary condition:

$$ V(x,T) = \Phi(x) $$

where:

$ V(x,t) $ is the value function, representing the minimum cost starting from state $ x $ at time $ t $,
$ ext{Tr} $ denotes the trace of a matrix.

Linear-Quadratic-Gaussian (LQG) Control

A special case of stochastic control is the Linear-Quadratic-Gaussian (LQG) control problem, which involves linear dynamics, quadratic cost, and Gaussian noise. The system is described by:

$$ \begin{align}
dx_t &= (Ax_t + Bu_t) dt + G dW_t, \
y_t &= Cx_t + v_t,
\end{align} $$

and the cost function is:

$$ J(u) = \mathbb{E} \left[ \int_0^T (x_t^T Q x_t + u_t^R u_t) \, dt + x_T^T Q_f x_T \right] $$

where $ A $, $ B $, $ C $, $ G $, $ Q $, $ R $, and $ Q_f $ are matrices of appropriate dimensions.

Solution to LQG Control

The optimal control law in LQG control is given by:

$$ u_t = -K x_t $$

where $ K $ is the solution to the Riccati equation:

$$ \dot{P} = -A^T P – P A + P B R^{-1} B^T P – Q, \quad P(T) = Q_f $$

Kalman Filter

In the presence of measurement noise, the state $ x_t $ cannot be directly observed. The Kalman filter provides an optimal estimate $ \hat{x}_t $ of the state. The Kalman filter equations are:

Prediction Step

$$ \hat{x}{t|t-1} &= A \hat{x}{t-1} + B u_{t-1}, \
P_{t|t-1} &= A P_{t-1} A^T + G G^T, $$

Update Step

$$ K_t = P_{t|t-1} C^T (C P_{t|t-1} C^T + R)^{-1}, \
\hat{x}t = $$

$$ \hat{x}{t|t-1} + K_t (y_t – C \hat{x}{t|t-1}), \ P_t = (I – K_t C) P{t|t-1}, $$

where:

$ \hat{x}_{t|t-1} $ is the predicted state,
$ P_{t|t-1} $ is the predicted error covariance,
$ K_t $ is the Kalman gain,
$ \hat{x}_t $ is the updated state estimate,
$ P_t $ is the updated error covariance.

Conclusion

Control theory in stochastic systems provides a framework for designing controllers that can handle uncertainty and randomness. The techniques discussed, such as the HJB equation, LQG control, and the Kalman filter, are fundamental tools in this field and have widespread applications in engineering, finance, and beyond.