10-643 Socio-technical Evaluations of Generative AI

12 units, Carnegie Mellon University

This course aims to introduce students to the growing number of evaluation metrics, measures, and methods proposed for assessing the capabilities and safety risks of Generative AI systems when they are deployed in society in ways that impact human lives and wellbeing. The course will also provide a methodological framework for designing and evaluating existing evaluation metrics, methods, and approaches. We formalize the necessary characteristics of evaluation benchmarks and automated methods to ensure that they capture the benefits and risks of GenAI systems in deployment. We focus on four key criteria: Validity, Reliability, Feasibility, and Usability. Through course projects, students will be prompted to design and/evaluate methods for evaluating the risks and capabilities of GenAI applications.

Prerequisites: 15-122 (min C) and (21-127 (min C) or 15-151 (min C)) and (36-217 (min C) or 15-259 (min C) or 15-359 (min C) or 36-225 (min C)) and (10-701 (min C) or 10-715 (min C) or 10-423 (min C) or 10-623 (min C) or 10-723 (min C) or 10-301 (min C) or 10-601 (min C) or 10-315 (min C))

Offered: Fall 2025

Courseweave needs JavaScript to show sections, workloads and planning tools.