11-661 Language and Statistics
12 units, Carnegie Mellon University
Language technologies search, text mining, information retrieval, speech recognition, machine translation, question answering, biological sequence analysis... are at the forefront of this centurys information revolution. In addition to their use of machine learning, these technologies rely centrally on classic statistical estimation techniques. Yet most CS and engineering undergraduate programs do not prepare students in this area beyond an introductory probstats course. This course is designed to plug this hole. The goal of Language and Statistics is to ground the datadriven techniques used in language technologies in sound statistical methodology. We start by formulating various language technology problems in both an information theoretic framework the sourcechannel paradigm and a Bayesian framework the Bayes classifier. We then discuss the statistical properties of words, sentences, documents and whole languages, and the computational formalisms used to represent language. These discussions naturally lead to specific concepts in statistical estimation. Topics include Zipfs distribution and typetoken curves point estimators, Maximum Likelihood estimation, bias and variance, sparseness, smoothing and clustering interpolation, shrinkage, and backoff entropy, cross entropy and mutual information decision tree models applied to language latent variable models and the EM algorithm hidden Markov models exponential models and maximum entropy semantic modeling and dimensionality reduction probabilistic contextfree grammars and syntactic language models. The course is designed for LTI SCS graduate students, but others are welcome. CS UG upperclassmen whove taken it have done well, though they found it challenging. The 11661 version does not require the course project. Prerequisites Strong quantitative aptitude. Comfort with basic UGlevel probability. Some programming skill.
Prerequisites: none listed
Courseweave needs JavaScript to show sections, workloads and planning tools.