Question on `extrinsic` in SEM Challenge-Cup-2025 real-robot dataset

[Chloe-Yan-3465]

Hello HorizonRobotics team,

I have a question regarding the real-robot RoboTwin dataset, specifically the meaning of:

• observation/left_camera/cam2world_gl
• observation/left_camera/extrinsic_cv

Previously, I thought cam2world_gl represents the transform between the camera and the base_link (0,0,0) point defined in the URDF. Is that correct?

Could you clarify the correct understanding of cam2world_gl and extrinsic_cv?

Specifically:

• What World Coordinate System or Base Frame are they based on?
• Is it the URDF base_link (0,0,0) or left_link6 (where the right_camera is fixed)?

My tf tree (generated from urdf file):

Example (from left_camera in place_empty_cup task):

=== cam2world_gl ===
[[[ 9.9573857e-01, -3.0173045e-03, -9.2171371e-02, -1.7736199e-01],
  [ 3.6063038e-02,  9.3261611e-01,  3.5906354e-01,  1.0416870e-01],
  [ 8.4877104e-02, -3.6085740e-01,  9.2875069e-01,  9.3343174e-01],
  [ 0.0000000e+00,  0.0000000e+00,  0.0000000e+00,  1.0000000e+00]], ... ]

=== extrinsic_cv ===
[[[ 9.9573863e-01,  3.6063075e-02,  8.4877074e-02,  9.3622595e-02],
  [ 3.0173063e-03, -9.3261617e-01,  3.6085749e-01, -2.3915121e-01],
  [ 9.2171311e-02, -3.5906351e-01, -9.2875075e-01,  9.2067635e-01]], ... ]

Thank you for your clarification!

[Muyun99]

Hello @Chloe-Yan-3465

As shown in the URDF, our world coordinate system is defined at left_base.

The calibration results are for left_camera to left_link6, right_camera to right_link6, and middle_camera to left_link6. You can refer to these values for the three sets of calibration results:

{
    "left_camera_2_left_ee": {
        "position": [
            -0.07354390146283836,
            0.007804615886680326,
            0.038433103882865485
        ],
        "orientation": [
            0.12827667735969864,
            -0.14997843923484816,
            0.7053150782153692,
            -0.6808687114652002
        ]
    },
    "right_camera_2_right_ee": {
        "position": [
            -0.07249626512541643,
            0.006136740643933003,
            0.038600128562769884
        ],
        "orientation": [
            0.1256876224765708,
            -0.1305963178064823,
            0.6996428946145881,
            -0.6911201366961431
        ]
    },
    "middle_camera_2_left_ee": {
        "position": [
            -0.004952661140681651,
            -0.25816692880684156,
            0.5124376131835251
        ],
        "orientation": [
            0.6511673965083838,
            -0.6455620264926346,
            0.30154504254448844,
            -0.2613451338595361
        ]
    }
}

The results in the LMDB dataset store the cam2world transformation for each frame. For the wrist-mounted cameras, you need to combine it with the end-effector (ee) transformation of each frame to get the final cam2world transformation.

[Muyun99]

Additionally, the specific meaning of a transform can often be ambiguous due to unclear frame definitions in the transformation matrices.

We have provided an Arrow dataset that you can try. It offers clearer messages and includes the parent_frame_id and child_frame_id for the camera transforms. Below is some example code that can export an MCAP file, which provides a very clear definition of the coordinate systems.

def build_dataset(config):
    from robo_orchard_lab.dataset.robot.dataset import (
        ConcatRODataset,
        ROMultiRowDataset,
    )
    from robo_orchard_lab.dataset.robotwin.transforms import (
        ArrowDataParse,
        EpisodeSamplerConfig,
    )

    dataset_list = []
    data_parser = ArrowDataParse(
        cam_names=config["cam_names"],
        load_image=True,
        load_depth=True,
        load_extrinsic=True,
        depth_scale=1000,
    )
    joint_sampler = EpisodeSamplerConfig(target_columns=["joints", "actions"])

    for path in config["data_path"]:
        dataset = ROMultiRowDataset(
            dataset_path=path, row_sampler=joint_sampler
        )
        dataset.set_transform(data_parser)
        dataset_list.append(dataset)

    dataset = ConcatRODataset(dataset_list)
    return dataset


config = dict(
    data_path=[
        "arrow_dataset_empty_cup_place_2025_08_19",
    ],
    cam_names=["left", "middle", "right"],
)
dataset = build_dataset(config)

# Show all key
frame_index = 0
print(len(dataset))
print(dataset[frame_index].keys())

# Show Camera Extrinsic
for camera in config["cam_names"]:
    print(f"{camera} extrinsic:\n {dataset[0][camera].pose}")

# Show important key
for key in [
    "joint_state",
    "master_joint_state",
    "imgs",
    "depths",
    "intrinsic",
    "T_world2cam",
]:
    print(f"{key}, shape is {dataset[frame_index][key].shape}")
print(f"Instuction: {dataset[frame_index]['text']}")
print(f"Dataset index: {dataset[frame_index]['dataset_index']}")


def export_mcap(dataset, episode_index, target_path):
    """Export the specified episode to an MCAP file."""
    from robo_orchard_lab.dataset.experimental.mcap.batch_encoder.camera import (  # noqa: E501
        McapBatchFromBatchCameraDataEncodedConfig,
    )
    from robo_orchard_lab.dataset.experimental.mcap.batch_encoder.joint_state import (  # noqa: E501
        McapBatchFromBatchJointStateConfig,
    )
    from robo_orchard_lab.dataset.experimental.mcap.writer import (
        Dataset2Mcap,
        McapBatchEncoderConfig,
    )

    dataset2mcap_cfg: dict[str, McapBatchEncoderConfig] = {
        "joints": McapBatchFromBatchJointStateConfig(
            target_topic="/observation/robot_state/joints"
        ),
    }
    dataset2mcap_cfg["actions"] = McapBatchFromBatchJointStateConfig(
        target_topic="/action/robot_state/joints"
    )

    for camera_name in config["cam_names"]:
        dataset2mcap_cfg[camera_name] = (
            McapBatchFromBatchCameraDataEncodedConfig(
                calib_topic=f"/observation/cameras/{camera_name}/calib",
                image_topic=f"/observation/cameras/{camera_name}/image",
                tf_topic=f"/observation/cameras/{camera_name}/tf",
            )
        )
        dataset2mcap_cfg[f"{camera_name}_depth"] = (
            McapBatchFromBatchCameraDataEncodedConfig(
                image_topic=f"/observation/cameras/{camera_name}/depth",
            )
        )

    to_mcap = Dataset2Mcap(dataset=dataset)
    to_mcap.save_episode(
        target_path=target_path,
        episode_index=episode_index,
        encoder_cfg=dataset2mcap_cfg,
    )
    print(f"Export episode {episode_index} to {target_path}")


# Export mcap file and use foxglove to viz
dataset_index = dataset[frame_index]["dataset_index"]
episode_index = dataset[frame_index]["episode"].index
export_mcap(
    dataset=dataset.datasets[dataset_index],
    episode_index=episode_index,
    target_path=f"./viz_dataidx_{dataset_index}_episodeidx_{episode_index}.mcap",
)

[AlanHuangT]

@Muyun99 为什么会有“ cam2world_gl” ？

[Muyun99]

@Muyun99 为什么会有“ cam2world_gl” ？

事实上真机数据的打包以及 transform 是没有这个 key 的，应该是用户自己处理出来的，不清楚咋处理的。我们提供的 lmdb dataset 以及 arrow dataset 的 key 都在这里

[Chloe-Yan-3465]

明白了，谢谢。

cam2world_gl是我们follow RoboTwin仿真器的数据集，既然sem无需这个Key那么太好了，少了一个问题。

但是我们仍有疑问：

1. 为什么middle相机选择标定到left_joint6,而不是一个static tf，比如left_base_link,是实验发现这样模型效果更好么？

2. 我们解析了一个分享在huggingface的 LMDB数据集，其中一个episode，left_camera extrinsic是变动的数值，如果它是left_camera到left_link6的外参，那么这个值应当是不变的，不是么？

（附 截图对应的txt，我们导出了其中一个episode的extrinsic数据）
extrinsic.txt

3. 确认一下，left_camera到left_link6的标定，是想标定在法兰盘位置对吧？

感谢您分享的Arrow数据集，不过我们暂时会先专注于使用LMDB，因此更关注这部分的数据。感谢您的答复

[Muyun99]

@Chloe-Yan-3465 oh sorry，我发现我表述错误了，中间相机标的是 middle_camera_2_left_base，标定流程产出的是一个不变量。所以对于你第一个问题：

• 手上相机标定结果是相机到 EE
• 全局相机是相机到某一个静止的坐标系（我们这边选择的是左臂的 base link）

2. LMDB dataset 解析出来是 left_camera到left_link6的外参吗？

这个涉及到标定流程和实际使用的区别。标定流程得到的是一个不变量，在实际使用的时候，我们需要统一相机外参到 world 坐标系，

• 对于全局相机，和标定结果一致，是一个定值
• 对于手上相机，需要通过 EE Pose（ee2world）以及 标定结果（cam2ee）得到（cam2world）

这个流程在打包过程中我们已经转过了，所以解析出来的三个相机的外参都是 cam2world

3. left_camera到left_link6的标定是在法兰盘的位置吗

事实上，我们是想标 left_camera 到 left_ee，对于 piper 臂来说，left_ee 的位置和 left_link6 是重合的（某些臂不一样，EE 和 TCP 的概念是有区分的）。left_link6 的位置可以从 urdf 的可视化上能够看到是在法兰盘的位置而不是夹爪中心

此外，我们会尽快放出真机数据的打包脚本供大家参考，会更加清晰一点，预计在这周内

[AlanHuangT]

@Muyun99 下一个版本我们把TF信息也放到arrow里面吧， 如果来得及。我近期更新一版本库。

[Chloe-Yan-3465]

明白了，非常感谢！
P.S. 以及感谢昨天发的标定工具