Benchmark Usage Instructions

For the RoboMIND 2.0 dataset, we provide an open-source, Isaac Sim-based simulation environment, complete with training data and standardized benchmark code. This empowers researchers to utilize these out-of-the-box resources for fast model evaluation and algorithm verification in a simulated setting.

• The currently open-sourced hdf5 data was collected in the isaacsim4.5 environment, so for better results, please use the isaacsim-4.5 branch code

Installation

1. Install Isaac Sim 5.1

   • Official Documentation: https://docs.isaacsim.omniverse.nvidia.com/5.1.0/installation/download.html
   • Linux Version Download: https://download.isaacsim.omniverse.nvidia.com/isaac-sim-standalone-5.1.0-linux-x86_64.zip
   • Installation steps:

   # Create installation directory
   mkdir ~/isaacsim_5.1
   
   # Unzip package (assuming downloaded to ~/Downloads)
   cd ~/Downloads
   unzip "isaac-sim-standalone-5.1.0-linux-x86_64.zip" -d ~/isaacsim_5.1
   
   # Run post-installation setup
   cd ~/isaacsim_5.1
   ./post_install.sh
   
   # Launch Isaac Sim
   ./isaac-sim.selector.sh

2. Install required packages

   ~/isaacsim_5.1/python.sh -m pip install -r RoboMIND-Sim/requirements.txt

Configuration

Edit RoboMIND-Sim/common/isaac_config.toml file:

[isaac_sim]
# Your Isaac Sim Python path, for example
python_path = "~/isaacsim_5.1/python.sh"

How to Start

Start Inference

Run RoboMIND-Sim/tools/policy_infer.py first.

python3 RoboMIND-Sim/tools/policy_infer.py

• The script only provides interfaces for interacting with the simulation, and model inference needs to be implemented by the user themselves !

Start Sim Benchmark

Then run Isaac Sim

Batch

# TienKung task
~/isaacsim_5.1/python.sh RoboMIND-Sim/benchmark.py --task TienKung_task_01 --loop 3 --timeout 300

Single

# TienKung task
~/isaacsim_5.1/python.sh RoboMIND-Sim/tasks/run_task.py  --task TienKung_task_01 --usd ABSOLUTE_PATH_TO_USD_FILE

Arguments Description

• --loop is the number of tests, with a default value of 1

• --timeout is the maximum duration for a single test, with a default of 300 seconds

• --task is the name of the evaluation task

  --task	Robot	Task description	Hdf5 links (collected in isaacsim4.5)	Example scenario	Our act test results
  TienKung_task_01	TienKung2.0	Put the paper cup into the basket with right arm	121-right_arm_put_paper_cup_into_box		62.2
  TienKung_task_02	TienKung2.0	Rotate the pan handle, from 6 o'clock to 9 o'clock direction	122-rotate_pot_handle		50.8
  TienKung_task_03	TienKung2.0	Organize the desktop, use left and right robotic arms to put the industrial switch objects on the desktop into the basket	103-put_switch_on_desktop_into_basket		19.8
  TienKung_task_04	TienKung2.0	Right arm pulls out the shelf, left arm puts the button on the shelf, right arm pushes the shelf back	115-places_switch_on_shelf		9.6

Note

• The currently open-sourced sim hdf5 data was collected in IsaacSim 4.5, and the sim data collected in IsaacSim 5.1 will be open-sourced in the near future

• Simulation run logs and evaluation results paths: RoboMIND-Sim/logs

• Each task provides 50 scenarios, and each time the simulation is initiated, one will be randomly selected