Multimodal Physics and Sensor Simulation for Synthesizing Training Data for Robot Manipulation

The goal of Sim4Dexterity is to develop powerful simulation tools for the economical and time-efficient virtual learning of AI robot manipulation skills in handling and assembly.

For this purpose, the capabilities of the industrially recognized simulator CoppeliaSim as well as the open source simulator Mujoco will be significantly improved. On the one hand, the Mujoco physics engine is extended by the realistic simulation of planar contact forces and tactile sensors. On the other hand, a realistic simulation of projector-based 3D sensors based on ray tracing is created. Furthermore, tools for the generation of dynamic robot simulation scenes are developed, which can be automatically varied in a multifactorial manner by augmentation and randomization techniques, e.g. with respect to the environmental conditions, the work equipment or the objects to be handled. A generative method to be developed for the synthesis of arbitrary realistic objects or components from real data supports this.

The functional verification of the synthesis methods is performed by the development of virtual validation techniques as well as by the evaluation of different AI algorithms trained with synthetic data. These will be validated in three use cases (bin picking, shelf picking, assembly) after purely virtual commissioning and made available to the scientific community and industry in challenges and benchmarks.

Project Duration: 1^st October 2021 – 30^th September 2023