Skip to content

Parallel conversion writes every episode to chunk-000 #1

Description

@griswaldbrooks

_process_episode_worker writes every episode to data/chunk-000/file-000.parquet, so with
--workers > 1 all but one episode's data is overwritten and finalize() reports length=0
for them.

In forge/convert/converter.py, the worker calls writer.write_episode(...), which flushes and
clears the current chunk before the worker sets writer._current_chunk_index = episode_index.
The later _flush_chunk() is therefore a no-op and the chunk index never reaches the file name.

This predates the image-mode change and affects video mode equally; image mode inherits it
because camera frames now travel inside the data parquet.

Repro: convert a multi-episode source with --workers 2 and inspect data/.

Deferred from the LeRobot v3 image-mode PR; targeted at a patch release.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions