Download pre-generated tutorial data instead of generating it#1057
Open
bendichter wants to merge 4 commits intomainfrom
Open
Download pre-generated tutorial data instead of generating it#1057bendichter wants to merge 4 commits intomainfrom
bendichter wants to merge 4 commits intomainfrom
Conversation
bendichter
force-pushed
the
cache-tutorial-data
branch
2 times, most recently
from
43f9cdf to
dbb7ebe
Compare
SpikeGLX recording data is generated locally (fast, just binary writes). Phy sorting data is downloaded from a GitHub release asset (17MB), avoiding ~2 min of PCA fitting per CI run. - Split generate_test_data into _generate_spikeglx_data (fast) + Phy (slow) - Add download_test_data: generates SpikeGLX + downloads pre-built Phy - App tries download first, falls back to full generation if offline - CI caches Phy data; E2E restores it before tests
bendichter
force-pushed
the
cache-tutorial-data
branch
from
3b7e94b to
ceaa846
Compare
Instead of downloading everything or generating everything: - SpikeGLX recording data is generated locally (fast, ~10s) - Phy sorting data is downloaded from GitHub release (17MB, avoids ~2min PCA) - Falls back to full generation if download fails (offline support)
bendichter
force-pushed
the
cache-tutorial-data
branch
from
70d5e03 to
f370f92
Compare
for more information, see https://pre-commit.ci
Collaborator
|
Tutorial generation now takes 13 seconds on my Mac M1. Do we still want to cache the tutorial data and use that in tests? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tutorial test data generation runs every E2E CI run, spending ~2-3 minutes fitting PCA across 50 units × 385 channels. This is unnecessary since the data is deterministic (seeded).
Solution
Pre-generated tutorial data is now hosted as GitHub release assets and downloaded instead of generated:
Changes
download_test_data()anddownload_test_dataset()functions +/data/downloadand/data/download/datasetAPI endpointsExampleDataCacheworkflow caches tutorial data from GitHub release; E2E workflow restores it before testsBenefits