Add ERA ComBat BBKNN method. - #84
Conversation
|
I have a couple questions about the submission and testing:
Thanks! |
Yes, it needs to be added there to be included in the workflow. It sounds like you ran the workflow already but I'm not sure that it would include the new method without doing this.
When we do the full benchmark run on the cloud any failed metrics are ignored (or more accurately given a score of zero). We don't usually do the full runs locally so there might be some differences in the settings that causes it to not produce an output. Generally we wouldn't want to disable a metric just for a specific dataset/method. |
|
Thank you for the responses. I noticed that the Regarding my other question, this was ultimately just a vanilla NextFlow question, I've now figured out how to pipe |
|
Okay this PR is ready for review. @mumichae please note that src/methods/bbknn_ts/script.py lines 18-226 does not require detailed review comments, this is an entirely LLM-generated function that we do not want to modify. |
|
Friendly ping -- do you need anything else prior to re-reviewing this PR @mumichae? Thank you for your help. |
|
Hi @cmclean, apologies for my late reply, I've been swamped with events the last couple weeks. I had a look at the prompt and the code in more detail, and found that it suggest a rather unconventional workflow, since you're running 2 integration benchmarks back-to-back. But since the point of openproblems is to benchmark novel approaches, so it's definitely interesting to include this approach. I added some clarification on the relationship between preprocessing and integration here, but ultimately decided to keep the LLM approach as its own end-to-end workflow, where preprocessing steps aren't tunable |
mumichae
left a comment
mumichae
left a comment
There was a problem hiding this comment.
Review on the output type of the method to be consistent with the evaluation workflow (i.e. the correct integrated representation gets evaluated)
|
As I understand it the above comments suggest to split the two features out to be evaluated independently. But that seems at odds with both the description of the v2.0.0 version of the leaderboard ("The batch integrated output can be a feature matrix, a low dimensional embedding and/or a neighbourhood graph") and the implementations provided by some existing methods that provide multiple outputs (e.g. I wrote questions in more detail at https://docs.google.com/document/d/1i2PqZjEVU8O544VbPQ9Bxh6zJai52LxK8D0Q0YXr4hY/edit?resourcekey=0-4x66cW9213nDd13ts0JhhQ&tab=t.0 |
|
Hi @cmclean! The "and/or" in the leaderboard description does apply here. No need to split anything in this case; a single component can and should emit both an embedding and a graph, it's part of the same method. I'll look into why I approved the scanorama split pr. Since your code sets Proposed changes: Config:
Script:
What do you think about a name like |
1. Renamed to `era_combat_bbknn`. 2. Fixed the human-written parts of script.py to save a lightweight output and use the VIASH-injected config values. 3. Updated `method_types` to reflect it is both an embedding and graph. 4. Updated paper reference.
|
Thank you for the additional review @rcannood! I've submitted a new commit that addresses your proposed changes and all of @mumichae comments that are consistent with this being an embedding+graph method. I made sure I can execute the updated |
There was a problem hiding this comment.
Thanks for the quick response! Almost lgtm, a few last remarks about the config.
I made sure I can execute the updated script.py locally but have not run it through the full pipeline; as long as only obsm['X_emb'], obsp, and uns are needed then it should be fine.
If the CI already passes, that's good enough. Any issues when the method is ran as part of the wf will get flagged in the issues later on ^^
With these last changes I'm happy to merge the PR.
| label: ERA ComBat BBKNN | ||
| summary: "A combination of ComBat and BBKNN discovered and implemented by Gemini." | ||
| description: | | ||
| "The ERA ComBat BBKNN solution (named "BBKNN (TS)" in the associated publication) applies standard scRNA-seq preprocessing steps, including total count normalization, log-transformation, and scaling of gene expression data. Batch effect correction is performed using scanpy.pp.combat directly on the gene expression matrix (before dimensionality reduction). Dimensionality reduction is then applied using PCA on the ComBat-corrected data, and this PCA embedding (adata.obsm['X_pca']) is designated as the integrated embedding (adata.obsm['X_emb']). A custom batch-aware nearest neighbors graph is constructed based on this integrated embedding; for each cell, neighbors are independently identified within its own batch and other batches, up to n_neighbors_per_batch. These candidate neighbors are merged, keeping the minimum distance for duplicate entries, and the top total_k_neighbors are selected for each cell. Finally, a symmetric sparse distance matrix and a binary connectivities matrix are generated to represent the integrated neighborhood graph. This code was entirely written by the AI system described in the associated publication." |
There was a problem hiding this comment.
If you're using |, the extra quotes are redundant
| "The ERA ComBat BBKNN solution (named "BBKNN (TS)" in the associated publication) applies standard scRNA-seq preprocessing steps, including total count normalization, log-transformation, and scaling of gene expression data. Batch effect correction is performed using scanpy.pp.combat directly on the gene expression matrix (before dimensionality reduction). Dimensionality reduction is then applied using PCA on the ComBat-corrected data, and this PCA embedding (adata.obsm['X_pca']) is designated as the integrated embedding (adata.obsm['X_emb']). A custom batch-aware nearest neighbors graph is constructed based on this integrated embedding; for each cell, neighbors are independently identified within its own batch and other batches, up to n_neighbors_per_batch. These candidate neighbors are merged, keeping the minimum distance for duplicate entries, and the top total_k_neighbors are selected for each cell. Finally, a symmetric sparse distance matrix and a binary connectivities matrix are generated to represent the integrated neighborhood graph. This code was entirely written by the AI system described in the associated publication." | |
| The ERA ComBat BBKNN solution (named "BBKNN (TS)" in the associated publication) applies standard scRNA-seq preprocessing steps, including total count normalization, log-transformation, and scaling of gene expression data. Batch effect correction is performed using scanpy.pp.combat directly on the gene expression matrix (before dimensionality reduction). Dimensionality reduction is then applied using PCA on the ComBat-corrected data, and this PCA embedding (adata.obsm['X_pca']) is designated as the integrated embedding (adata.obsm['X_emb']). A custom batch-aware nearest neighbors graph is constructed based on this integrated embedding; for each cell, neighbors are independently identified within its own batch and other batches, up to n_neighbors_per_batch. These candidate neighbors are merged, keeping the minimum distance for duplicate entries, and the top total_k_neighbors are selected for each cell. Finally, a symmetric sparse distance matrix and a binary connectivities matrix are generated to represent the integrated neighborhood graph. This code was entirely written by the AI system described in the associated publication. |
| bibtex: | | ||
| @article{GoogleScienceAI2026, | ||
| title={An {AI} system to help scientists write expert-level empirical software}, | ||
| author={Ayg{\"u}n, Eser and Belyaeva, Anastasiya and Comanici, Gheorghe and Coram, Marc and Cui, Hao and Garrison, Jake and Johnston, Renee and Kast, Anton and McLean, Cory Y and Norgaard, Peter and others}, | ||
| journal={Nature}, | ||
| volume={654}, | ||
| pages={909-–916}, | ||
| year={2026}, | ||
| } |
There was a problem hiding this comment.
If the publication has a doi, it's easier to just use the doi
| bibtex: | | |
| @article{GoogleScienceAI2026, | |
| title={An {AI} system to help scientists write expert-level empirical software}, | |
| author={Ayg{\"u}n, Eser and Belyaeva, Anastasiya and Comanici, Gheorghe and Coram, Marc and Cui, Hao and Garrison, Jake and Johnston, Renee and Kast, Anton and McLean, Cory Y and Norgaard, Peter and others}, | |
| journal={Nature}, | |
| volume={654}, | |
| pages={909-–916}, | |
| year={2026}, | |
| } | |
| # Aygün, E., Belyaeva, A., Comanici, G. et al. | |
| # An AI system to help scientists write expert-level empirical software. | |
| # Nature 654, 909-916 (2026). https://doi.org/10.1038/s41586-026-10658-6 | |
| doi: 10.1038/s41586-026-10658-6 |
|
|
||
| engines: | ||
| - type: docker | ||
| image: openproblems/base_python:1.0.0 |
There was a problem hiding this comment.
| image: openproblems/base_python:1.0.0 | |
| image: openproblems/base_python:1 |
Use the :1 tag to get updates from time to time
4 participants
Describe your changes
This PR adds the top-performing "BBKNN (TS)" method from our recent preprint [1]. I have verified that tests pass, and running the
run_full_local.shscript on my own machine successfully recapitulates the numbers published in the preprint.[1] Aygun et al, An AI system to help scientists write expert-level empirical software, arXiv:2509.06503 (2025), https://arxiv.org/abs/2509.06503 .
Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!