36-492 Topic Detection and Document Clustering
6 units, Carnegie Mellon University
Imagine if someone read all your email. Everything you sent, everything you received. What would they find Do you have repeating topics How do the topics change over time The Enron Corporation was an energy, commodities, and services company in Houston, Texas that went spectacularly bankrupt in 2001 after it was revealed that it was engaging in systematic, planned accounting fraud. At its peak, it employed over 20,000 people with revenues over 100 billion. Its downfall was related to deregulation of Californias energy commodity trading and a series of rolling power blackouts over months. For example, Enron traders encouraged the removal of power during the energy crisis by suggesting plant shutdowns. The resulting increase in the price for power made them a fortune. After Enrons collapse, journalists used the Freedom of Information Act to release the emails sentreceived by the employees of Enron. Subsequently, the emails were analyzed to see who knew what and when. Every news article, email, letter, blog, tweet, etc can be thought of as an observation. We characterize these documents by their length, what words they use and how often, and possibly extra information like the time, the recipient, etc. Topic detection and document clustering methods are statistical and machine learning tools that extract and identify related documents, possibly over time. These methods need to be flexible enough to handle both very small and very large clusters of documents, topics that change in importance, and topics that appear and disappear. This class will emphasize application of methods and realworld data analysis. Class time will be split into lecture and lab. Bring your laptop. Occasional homeworks and final project, but mostly well focus on the downfall of Enron as our overarching case study.
Prerequisites: 36-401
Courseweave needs JavaScript to show sections, workloads and planning tools.