15-853 Theory of Markov Processes: Selected Topics with Applications in ML and GenAI
12 units, Carnegie Mellon University
Markov processes are a fundamental mathematical concept with broad applications, including emerging fields such as reinforcement learning and diffusion models. This course is structured into two parts. Part I covers the core theory of Markov processes, including discrete-time and continuous-time Markov chains, as well as Markov processes with continuous state space such as diffusion processes. Part II builds on the core theory and covers selected topics in the theoretical foundation of reinforcement learning and diffusion models in generative AI. Key topics: - Discrete-state Markov processes: ergodicity, Lyapunov drift analysis - Continuous-state Markov processes: Brownian motion, Markov semigroups, Itô integral, stochastic differential equations, Langevin diffusions - Mixing time analysis: coupling method, Poincaré inequality, log Sobolev inequalities - Reinforcement learning: Bellman equation, value iteration, policy iteration, Q-learning, TD-learning, policy gradient - Diffusion models: sampling algorithms, DDPM, DDIM, flow matching, convergence analysis, discrete diffusion models
Prerequisites: none listed
Offered: Spring 2026
Courseweave needs JavaScript to show sections, workloads and planning tools.