10-704 Information Processing and Learning
12 units, Carnegie Mellon University
Whats the connection between how many bits we can send over a channel and how accurately we can classify documents or fit a curve to data Is there any link between decision trees, prefix codes and wavelet transforms What about maxentropy and maximum likelihood, or universal coding and online learning This interdisciplinary course will explore these and other questions that link the fields of information theory, signal processing, and machine learning, all of which aim to understand the information contained in data. The goal is to highlight the common concepts and establish concrete links between these fields that enable efficient information processing and learning. The course will do a short but introductory review of basic information theory, including entropy and fundamental limits of data compression, data processing and Fanos inequalities, channel capacity, and ratedistortion theory. Then we will dig into the connections to learning including estimation of information theoretic quantities such entropy, mutual information, and divergence and their applications in learning, information theoretic lower bounds for machine learning problems, duality of max entropy and maximum likelihood, connections between clustering and ratedistortion theory, universal coding and online learning, and more. Prerequisites Fundamentals of Probability, Statistics, Linear Algebra and Real analysis.
Prerequisites: none listed
Courseweave needs JavaScript to show sections, workloads and planning tools.