[CS391R] Overview of Robot perception

1. Robot perception: seeing and understanding the physical world by multimodal robot sensors
   1. Robot vision vs. Computer vision
      1. Robot vision is embodied, active and environmentally situated.
      2. Embodied: Robots have physical bodies and experience the world directly. Their actions are part of a dynamic with the world and have immediate feedback on their own sensation.
      3. Active: Robots are active perceivers. It knows why it wishes to sense, and chooses what to perceive, and determines how, when and where to achieve that perception.
      4. Situated: Robots are situated in the world. They do not deal with abstract descriptions, but with the here and now of the world directly influencing the behavior of the system.
2. Modalities: neural network architectures designed for different sensory modalities 
   1. Pixels (from RGB camera)
   2. Point cloud (from structure sensors)
   3. Time series (from F/T sensors)
   4. Tactile data (from the GelSights sensors)
3. Representations: representation learning algorithms without strong supervision.
   • A fundamental problem in robot perception is to learn the proper representations of the unstructured world.
   •  밑 그림 참조
   • Learn representations of the world with limited supervision: self-supervised learning. Supervision comes from the unlabeled data themselves
   • Learn representations that fuse multiple sensory modalities together: Multimodal sensor fusion 
4. Tasks: state estimation tasks for robot navigation and manipulation
   • State estimation methods: Bayes filtering
   • What if models are hard to specify? Learning -> Embodied view of perception.
5. Frontiers: embodied visual learning & synthetic data for visualAI

 

6. S094, MIT