Multimodal Physics and Sensor Simulation for Synthesizing Training Data for Robot Manipulation
Multimodal Physics and Sensor Simulation for Synthesizing Training Data for Robot Manipulation

© Universität Bielefeld

ICRA 2023 Virtual Manipulation Challenge: Assembly

Assembly Track Tasks

The assembly track of the challenge uses an extended MuJoCo4 environment with ROS integration. The assembly environments provide a ros_control interface to send position, velocity, or effort commands to the robot. Furthermore, joint position and velocity as well as force/torque sensor readings of the end-effector are accessible via ROS topics. For visual processing, a fixed RGB-D camera facing the workspace and a hand-mounted RGB camera will be available. The track is split into a screwing and a plugging task and both will be offered with multiple levels of difficulty.

© Universität Bielefeld
© Universität Bielefeld

Screwing

For this task, a screw, a nut, and a fixture are placed on the workspace table – their object meshes will be available. The goal is to thread the screw into the nut. This requires solving the following sub tasks: pick up the nut, place the nut into the fixture, pick up the screw, align screw into nut, and tighten the screw. Each sub task should be completed within 5 minutes, otherwise the whole run will be terminated prematurely.

Plugging

For this task, a peg-like plug and its counterpart, a frame with a hole, are placed randomly on the workspace table. The object meshes will be available to participants again. Additionally, the Cartesian poses of the plug and the frame will be observable from the environment. However, the frame’s actual pose will be perturbed along the table’s surface by a random sample from a given uniform distribution, depending on the difficulty level. The goal is to pick the plug and insert it into the frame against a clipping force profile, which requires solving the following sub tasks: pick up the plug, localize the frame, align plug into frame, and click-insert the plug. Each sub task must be completed within 30 seconds, otherwise the whole run will be terminated prematurely during evaluation.

Evaluation

The Screwing task will be evaluated for a fixed number of iterations on each difficulty level. Initial poses of objects will be randomized across iterations, but identical for all participants. The total score is aggregated as the weighted sum of all successfully completed sub tasks over all iterations as follows:

 

score = Σd∈DΣs∈S ws ∙ P(d,s)

 (1)

where D is the set of difficulty levels, S is the set of sub tasks, ws is the weight of sub task s as indicated above, and P(d,s) is the success rate of sub task s on difficulty level d.

The Plugging task evaluation to the contributed solutions will be for a fixed number of iterations on each difficulty level. The total score is, analogously to the screwing tasks, the weighted sum of all completed sub tasks over all iterations. In the case of ties, we break them by dividing the scores by the sum of the weighted time the solutions took to achieve them, i.e.,

 

 tiebreak score = score / (Σd∈DΣs∈S ws ∙ T(d,s)) (2)

where T(d,s) denotes the mean required time to solve sub task s on difficulty level d.

Organizers

David Philip Leins, Florian Patzelt, Robert Haschke, Balázs András Bálint

The Virtual Manipulation Challenge is organized by the research project Sim4Dexterity. The Sim4Dexterity project has received funding from the German Federal Ministry of Education and Research (BMBF) through grant 01IS21061 (Sim4Dexterity).