Perception-Manipulation Pipeline

End-to-end perception-manipulation integration: scene understanding, task-level state estimation, keypoint detection, and complete pick-and-place pipelines.

Learning Objectives

1. From Perception to Action

1.1 The Perception-Action Loop

1.2 System Architecture Overview

2. Scene Understanding

2.1 Instance Segmentation

2.2 6DoF Object Pose Estimation

2.3 Scene Graph Representation

3. Task-Level State Estimation

3.1 Keypoint Detection

3.2 Dense Correspondences

3.3 Scene Flow

4. Complete Pick-and-Place Pipeline

4.1 Detect → Segment → Pose → Grasp → Place

4.2 Failure Detection & Recovery

5. Foundation Models for Manipulation

5.1 SAM for Manipulation

5.2 LLM-Based Task Planning

5.3 Vision-Language-Action Models

6. ROS 2 Integration: Full Pipeline

7. Debugging & Performance Tuning

Exercises

References