36-614 Data Engineering and Distributed Environments
6 units, Carnegie Mellon University
Statisticians and data scientists in industry are increasingly expected to work with data from complex sources: not just spreadsheets and CSV files, but relational databases, distributed databases, and streams, often integrating data from dozens of different systems. This course introduces the basic principles of data engineering, beginning with relational databases and continuing to distributed databases in the cloud. Students will learn SQL, practice using Python to query databases, and be introduced to Apache Spark as a tool for distributed data access. This course is only for students in the Master of Science in Applied Data Science program. Students in this course should have prior experience programming in Python (such as through 36-650), using Git, and working from the command-line shell. By the end of this course, students will be able to: - Design SQL tables to store complex, frequently updated data. - Write SQL queries to group, aggregate, and summarize data stored in multiple interconnected tables. - Use Python to execute SQL queries and dynamically act upon stored data. - Load data into SQL databases and update existing data. - Develop Python data pipelines that ingest data from multiple sources, load it into a database, and produce automated reports summarizing that data. - Write PySpark code to manipulate data in Apache Spark. - Execute code on a cloud platform such as Azure.
Prerequisites: none listed
Offered: Fall 2026, Fall 2025
Courseweave needs JavaScript to show sections, workloads and planning tools.