ECE 455 Control of Stochastic Systems
 
This is a graduate-level course on stochastic systems, covering a wide range of topics, including Markovian and linear stochastic models; stochastic stability; dynamic programming; parameter and state estimation; and reinforcement learning.

Prerequisites: ECE 415 is required for background in control, and ECE 434 provides necessary background on stochastic processes (or consent of instructor).

Text: The following are useful, but not required:

  • Meyn & Tweedie, Markov chains and stochastic stability, Springer-Verlag, 1992
  • P.R. Kumar & P. Varaiya, Stochastic Systems, Prentice-Hall, 1986.
  • E.A. Feinberg & A. Shwartz, (eds.) Handbook of Markov Decision Processes: Methods and Applications, Kluwer Academic Publishers, 2001.
  • J. Tsitsiklis & D. Bertsimas, Neuro-dynamic Programming, Athena Scientific, 1996

Lecture notes are available at TIS on Green Street

Outline

I. Markov Models

  • Markov & state space models
  • Linear models
  • Storage models
  • Reflected Brownian motion

II. Stochastic Stability

  • Recurrence and transience
  • Ergodicity
  • Geometric ergodicity

III. Performance

  • Lyapunov equation for linear models
  • Poisson's equation
  • Simulation
  • Queueing models

IV. Controlled Models

  • Optimality equations
  • Dynamic programming
  • Partial observations & information states

V. Linear Models

  • Linear systems & spectral densities
  • Optimal control: LQ & LQG
  • Kalman filter

VI. Queueing Models

  • Markov models
  • Approximating Poisson's equation
  • Structure of policies
  • Heavy traffic approximations

VII. Stochastic Approximation

  • SA approaches
  • The ODE method
  • Adaptive control

VIII. Reinforcement Learning

  • Q-learning
  • Actor-critic algorithms
  • Function approximation