Skip to content

[ML] Refactor the RDataLoader: new cluster-aligned reading pattern, improved shuffling strategy and new Python API #21888

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
siliataider wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[ML] Refactor the RDataLoader: new cluster-aligned reading pattern, improved shuffling strategy and new Python API #21888

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bc5cb85
[ML] Add RecalculateBatchCounts helper
siliataider Apr 10, 2026
b2dd64d
[ML] Add RClusterLoader
siliataider Apr 10, 2026
26e9f70
[ML] Rename RBatchGenerator and use RClusterLoader
siliataider Apr 10, 2026
2b89d6a
[Python][ML] Implement the new Python interface of the RDataLoader
siliataider Apr 10, 2026
8dfbb5d
[Python][ML] Adapt tests to the new RDataLoader interface
siliataider Apr 10, 2026
f170883
[Python][ML] Adapt tutorials to the new RDataLoader interface
siliataider Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
976 changes: 313 additions & 663 deletions bindings/pyroot/pythonizations/python/ROOT/_pythonization/_ml_dataloader.py
View file
Open in desktop

Large diffs are not rendered by default.

2,282 changes: 1,732 additions & 550 deletions bindings/pyroot/pythonizations/test/ml_dataloader.py
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 2 additions & 4 deletions tree/ml/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,15 @@ endif()

ROOT_STANDARD_LIBRARY_PACKAGE(ROOTMLDataLoader
HEADERS
ROOT/ML/RBatchGenerator.hxx
ROOT/ML/RDataLoaderEngine.hxx
Comment thread
siliataider marked this conversation as resolved.
ROOT/ML/RBatchLoader.hxx
ROOT/ML/RChunkLoader.hxx
ROOT/ML/RChunkConstructor.hxx
ROOT/ML/RClusterLoader.hxx
ROOT/ML/RFlat2DMatrix.hxx
ROOT/ML/RFlat2DMatrixOperators.hxx
ROOT/ML/RDatasetLoader.hxx
ROOT/ML/RSampler.hxx
SOURCES
src/RBatchLoader.cxx
src/RChunkConstructor.cxx
src/RFlat2DMatrixOperators.cxx
src/RSampler.cxx
DEPENDENCIES
Expand Down
2 changes: 1 addition & 1 deletion tree/ml/LinkDef.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
// corresponding dictionary source file, and apparently only the second one is necessary for cppyy.
#pragma link C++ namespace ROOT::Experimental::ML;
#pragma link C++ namespace ROOT::Experimental::Internal::ML;
#pragma link C++ class ROOT::Experimental::Internal::ML::RBatchGenerator<int>;
#pragma link C++ class ROOT::Experimental::Internal::ML::RDataLoader < int>;
#endif
#endif
1 change: 1 addition & 0 deletions tree/ml/inc/ROOT/ML/RBatchLoader.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ public:
std::unique_ptr<RFlat2DMatrix> CreateBatch(RFlat2DMatrix &chunkTensor, std::size_t idxs);
RFlat2DMatrix GetBatch();
void CreateBatches(RFlat2DMatrix &chunkTensor, bool isLastBatch);
void RecalculateBatchCounts(std::size_t numEntries);

bool isProducerDone() { return fProducerDone; }
std::size_t GetNumBatches() { return fNumBatches; }
Expand Down
113 changes: 0 additions & 113 deletions tree/ml/inc/ROOT/ML/RChunkConstructor.hxx
View file
Open in desktop

This file was deleted.

Loading
Loading