Manipulation in Simulation

Simulating robotic manipulation: contact-rich simulation, grasp simulation, manipulation benchmark environments, and sim-to-real for manipulation tasks.

Learning Objectives

1. Why Simulate Manipulation?

1.1 Contact-Rich Challenges

1.2 Data Generation at Scale

2. Physics Engines for Manipulation

2.1 MuJoCo for Manipulation

2.2 Isaac Sim for Manipulation

2.3 Drake for Manipulation

3. Grasp Simulation

3.1 Parallel-Jaw Gripper Simulation

3.2 Dexterous Hand Simulation

3.3 Grasp Quality Evaluation in Sim

4. Manipulation Benchmark Environments

4.1 RLBench

4.2 CALVIN

4.3 ManiSkill

5. Sim-to-Real for Manipulation

5.1 Domain Randomization for Grasping

5.2 System Identification for Contact

5.3 Fine-Tuning on Real Robot

6. Training Policies in Simulation

6.1 RL for Manipulation Tasks

6.2 Imitation Learning in Sim

7. ROS 2 + Simulation Integration

Exercises

References