Data onboarding: remove hardcoded paths, S3 dataset fetch, nuPlan docs by eugenevinitsky · Pull Request #525 · Emerge-Lab/PufferDrive

eugenevinitsky · 2026-07-10T19:44:24Z

Switch to the data being sourced from s3. Remove old evaluators that are unused.

The default config enabled 13 evaluators pointing at a maintainer-local /scratch/ev2237 nuPlan tree, giving external users 13 FileNotFoundError tracebacks per eval pass. validation_replay and behaviors_full_dir now ship disabled with a placeholder map_dir; the 11 per-category behaviors_* sections (irreproducible categories_v021 split) are removed in favor of a documented pattern users instantiate against their own labelled bins. Cluster yaml configs drop the enabled=0 overrides whose flags would no longer be registered by the ini-driven parser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add docs/nuplan_data.md covering the mini-split download, the external py123d converter (repo URL and invocation left as maintainer TODOs, with a reduced-parallelism RAM note), the expected .bin layout, an example replay training command, and how to enable the shipped-disabled nuPlan evaluators. Link it from the README Data section alongside the WOMD download scripts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Datasets live in the lab S3 buckets and are declared in data_utils/datasets.yaml; fetch_data.py syncs them by name into $PUFFERDRIVE_DATA_ROOT (default <repo>/data, gitignored). docs/data_storage.md records the bucket layout, access process, and the upstream license constraints on redistributing each dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Replaces the maintainer placeholders with the real two-stage flow: py123d downloads/parses nuPlan into arrow, 123Drive converts arrow to PufferDrive bins. Includes the RAM guidance (--workers, and the nuplan preset's 20 s log chunking). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Manifest entries marked public: true sync with unsigned requests (aws s3 sync --no-sign-request), so public-read buckets need no AWS account. Verified against Motional's public nuplan bucket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The dev prefix holds nuplan_train (~475 GB, ~170k bins) and nuplan_val (~48 GB, ~17k bins); one entry covering both made the smallest possible fetch half a terabyte. Entries now carry a required size field, printed in --list and before every sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fetching nuplan_mini_train/val from the lab buckets is the default path; the py123d + 123Drive pipeline stays as the do-it-yourself route. Depends on the fetch script from the s3-data-fetch PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

nuPlan bins now live at s3://pufferdrive-bins/nuplan/0.3.2/{train,val} with ~10 GB samples at 0.3.2-mini/{train,val}. The mini entries are the documented default fetch; the personal-bucket dev entries are gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

data/nuplan_mini_val is where data_utils/fetch_data.py lands the default eval set, so enabling the eval is just enabled = true after a fetch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…data-fetch # Conflicts: # README.md

The pufferdrive-bins policy now grants anonymous GetObject + ListBucket scoped to the nuplan/ prefix, with the CC BY-NC-SA notice uploaded at nuplan/LICENSE.txt. The four nuplan entries fetch with unsigned requests, so no AWS account is needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…cratch-paths' into ev/s3-data-fetch

A nonexistent map_dir previously surfaced as a bare os.listdir FileNotFoundError. When its basename matches a dataset registered in data_utils/datasets.yaml, the error now names the exact fetch_data.py command; otherwise it states plainly that the path does not exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A bare fetch_data.py run downloads the manifest entries marked default: true (the ~10 GB minis); multiple names are accepted. The map_dir tests use absolute tmp_path paths so they no longer depend on the CWD or on whether the real dataset was fetched. Stale docstring examples, dead line-number citations, and redundant doc/ini prose removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PyPI carries AWS CLI v1, which handles the unsigned s3 sync identically to a system v2. With this the fresh-user flow is: install, run data_utils/fetch_data.py, train. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The default <repo>/data is where the drive.ini defaults expect data, so an env var that silently moves fetches away from it invited a config mismatch. Custom destinations are now explicit and per-invocation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

riccardosavorgnan · 2026-07-13T15:54:32Z

+pip install "nuplan-devkit @ git+https://github.com/motional/nuplan-devkit/@nuplan-devkit-v1.2"
+
+# Mini set (~11 GB) — enough for replay training and the nuPlan evals:
+py123d-download dataset=nuplan \


Should we move the instruction that you must set export NUPLAN_DATA_ROOT before this point?
this command will otherwise fail.

riccardosavorgnan · 2026-07-13T16:28:42Z

+    'dataset.downloader.splits=[nuplan-mini_train, nuplan-mini_val, nuplan-mini_test]'
+
+# Parse the downloaded logs + maps into py123d's arrow format:
+py123d-conversion datasets=["nuplan-mini"]


This returns an error, I think the correct syntax to override is:
py123d-conversion dataset=nuplan-mini

Error log:

(pufferdata_2) (base) ricky@rickynyuserver:~/pufferdata_2$ py123d-conversion datasets=["nuplan-mini"] /home/ricky/miniconda3/lib/python3.13/site-packages/requests/__init__.py:86: RequestsDependencyWarning: Unable to find acceptable character detection dependency (chardet or charset_normalizer). warnings.warn( Could not override 'datasets'. To append to your config use +datasets=[nuplan-mini] Key 'datasets' is not in struct full_key: datasets object_type=dict Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

riccardosavorgnan · 2026-07-13T16:58:05Z

+```bash
+git clone https://github.com/vcharraut/123Drive && cd 123Drive
+uv sync
+uv run convert --preset nuplan --py123d_path /path/to/py123d/data --output ./nuplan_bins


this does not find the logs correctly.

After processing the maps according to the previous instructions, py123d outputs 2 folders ./None/maps, ./None/logs in the directory where you run the command. When I then run the command pointing to that folder I get the following error:

(pufferdata_2) (base) ricky@rickynyuserver:~/pufferdata_2/123Drive$ uv run convert --preset nuplan --py123d_path ~/pufferdata_2/None/ --output ./nuplan_bins
warning: VIRTUAL_ENV=/home/ricky/pufferdata_2/.venv does not match the project environment path .venv and will be ignored; use --active to target the active environment instead
INFO bin_factory: 123Drive: /home/ricky/pufferdata_2/None/ -> ./nuplan_bins
INFO bin_factory: Filters - datasets: ['nuplan'], split_types: None, split_names: None, log_names: None, duration_s: 20, map_only: False
INFO bin_factory: No scenarios to process.

From claude: the error appears because the nuplan-mini dataset has a different prefix for the scenes so the command scans the folder and doesn't find any. The fix is to pass the dataset name (nuplan-mini) as a flag in the command.

uv run convert --preset nuplan --datasets nuplan-mini --py123d_path ~/pufferdata_2/None/ --output ./nuplan_bins

riccardosavorgnan · 2026-07-13T17:23:35Z

+```bash
+git clone https://github.com/vcharraut/123Drive && cd 123Drive
+uv sync
+uv run convert --preset nuplan --py123d_path /path/to/py123d/data --output ./nuplan_bins


This command also requires to set
export PY123D_DATA_ROOT=/path/to/outputofpy123d/
otherwise the conversion fails.
The reason is that internally the call to the maps data is performed via this path and not via the flag --py123d_path

Eugene Vinitsky and others added 13 commits July 10, 2026 11:05

Remove AGENTIC_PR file

c4a3aeb

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Remove AGENTIC_PR file

bf9f0ec

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Point README Data section at the S3 fetch script

029ab2c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Point disabled nuPlan eval placeholders at the default fetch path

57fc612

data/nuplan_mini_val is where data_utils/fetch_data.py lands the default eval set, so enabling the eval is just enabled = true after a fetch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/ev/nuplan-data-docs' into ev/s3-…

beb22b9

…data-fetch # Conflicts: # README.md

eugenevinitsky changed the title ~~Add S3 dataset registry and fetch script~~ Add S3 dataset registry, fetch script, and nuPlan data docs Jul 11, 2026

eugenevinitsky mentioned this pull request Jul 11, 2026

Document nuPlan download, conversion, and replay usage #519

Closed

Eugene Vinitsky and others added 2 commits July 11, 2026 12:14

Merge remote-tracking branch 'origin/fix/drive-ini-remove-hardcoded-s…

5dc2629

…cratch-paths' into ev/s3-data-fetch

eugenevinitsky changed the title ~~Add S3 dataset registry, fetch script, and nuPlan data docs~~ Data onboarding: remove hardcoded paths, S3 dataset fetch, nuPlan docs Jul 11, 2026

eugenevinitsky mentioned this pull request Jul 11, 2026

Remove hardcoded /scratch nuPlan paths from shipped drive.ini #518

Closed

Eugene Vinitsky and others added 8 commits July 11, 2026 12:25

Apply ruff format to fetch_data.py

d661b69

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Trim data docs

141f69c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

validation_replay renders via obs_html; drop behaviors block comment

462b506

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Drop map_dir error-message test

113d77c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

riccardosavorgnan reviewed Jul 13, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Data onboarding: remove hardcoded paths, S3 dataset fetch, nuPlan docs#525

Data onboarding: remove hardcoded paths, S3 dataset fetch, nuPlan docs#525
eugenevinitsky wants to merge 23 commits into
3.0from
ev/s3-data-fetch

eugenevinitsky commented Jul 10, 2026 •

edited

Loading

Uh oh!

riccardosavorgnan Jul 13, 2026

Uh oh!

riccardosavorgnan Jul 13, 2026 •

edited

Loading

Uh oh!

riccardosavorgnan Jul 13, 2026

Uh oh!

riccardosavorgnan Jul 13, 2026 •

edited

Loading

Uh oh!

riccardosavorgnan Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

eugenevinitsky commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

riccardosavorgnan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

riccardosavorgnan Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

riccardosavorgnan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

riccardosavorgnan Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

riccardosavorgnan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eugenevinitsky commented Jul 10, 2026 •

edited

Loading

riccardosavorgnan Jul 13, 2026 •

edited

Loading

riccardosavorgnan Jul 13, 2026 •

edited

Loading