11-851 Talking to Robots

6–12 units, Carnegie Mellon University

Household robots need to move beyond simple programmed tasks like those a roomba, and become full-fledged digital assistants. A robotic agent that exists (physically) in the world, gains access to rich and personalized knowledge of its environment. How much do things weigh? What's fragile? Where you store the extra chocolates that you don't want anyone to find because they are just for you. Building an agent that can accomplish tasks requires the integration of a diverse set of technologies and engineering. Language models, SLAM, semantic mapping, task planning, understanding affordances, and end effector control. This course will cover both foundational works in grounding language to action and analyze (or reimplement) state-of-the-art Large Language Model based task planners.

Prerequisites: none listed

Offered: Spring 2027, Fall 2025

Courseweave needs JavaScript to show sections, workloads and planning tools.