Initial framework of the niche refactor

[shashkat]

Description

Niche refactor improving the current implementation giving the user facing functions a more explicit API, more flexibility through embedder and clusterer classes and possibility of using combinations of them using calculate_niche_custom. Postprocessor classes allow more organized and flexible postprocessing actions like masking and min_niche_size filtering of niches.
The core idea is similar to the spatial_neighbors refactor in #1147 and builds on top of the relatively smaller fix in #1229.

The below graph demonstrates the code flow planned (read children of a node from left to right). This allows usage of different combinations of embeddings and clusterings to obtain niches, and also exposing easy to use and explicit user facing functions for the different flavors. Brown nodes are user-facing functions.

How has this been tested?

Keeping the API of function calculate_niche same, for i) backward compatibility and ii) being able to validate results being same before and after refactor for some test cases that I wrote for the purpose of this refactor, for the different flavors. Though later on, I think calculate_niche should be deprecated as its not clear how arguments of it interact with different values of flavor argument.

Closes

Closes #1192, where this was initially discussed

[codecov]

Codecov Report

❌ Patch coverage is 78.83959% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.59%. Comparing base (b71dae1) to head (a3df888).

Files with missing lines	Patch %	Lines
src/squidpy/gr/_niche.py	78.47%	43 Missing and 19 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1245      +/-   ##
==========================================
+ Coverage   77.11%   78.59%   +1.47%     
==========================================
  Files          63       63              
  Lines        9418     9459      +41     
  Branches     1585     1585              
==========================================
+ Hits         7263     7434     +171     
+ Misses       1554     1425     -129     
+ Partials      601      600       -1     

Files with missing lines	Coverage Δ
src/squidpy/_docs.py	95.12% <100.00%> (+0.31%)	⬆️
src/squidpy/gr/_niche.py	76.33% <78.47%> (+38.80%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shashkat]

This is ready for review @selmanozleyen @grst

Some important points:

• I think one of the most important next steps after this PR is that the naming of the columns added in the different flavors is changed. Currently, column names added look like: adata.obs['nhood_niche_res=1.0']. However, sdata doesn't support such column names with equal-to sign More robust name validation spatialdata#703. Hence, if one used any of the current niche functions with sdata and inplace=True, it will give error. Currently, I kept naming strategy same for backward compatibility, but we should change it after this. It should be relatively straightforward after the refactor.

• I believe the test cases I have added in test_niche.py are quite useful. They helped me validate that the output was the same before and after refactor for an example object (dummy_adata2) for different flavors. However, I realized that with scanpy<1.12.1 in the test env, the output was different than with scanpy>=1.12.1. This is because of a detail in scanpy mentioned here: fix: copy connectivities before passing to umap-learn scanpy#4031, and fixed in 1.12.1 onwards. This means that if someone will run hatch-test on squidpy and they have scanpy<1.12.1 in their test env, some tests will fail. I dont know what is the best way to handle this, hence would like to know your suggestions.

[selmanozleyen]

I will clean up the docstrings to have a dropdown per flavour so we get rid of this mess and will add the AutoK function in a separate PR. But still wanted to ask for a review from you @Intron7

[selmanozleyen]

@shashkat , thanks a lot for the PR I hope you don't mind my commits directly without asking but wdyt?

also you mentioned spatial_neighbours didn't do something as it should like reusing the spatial_neighbours_from_builder or something? I'd like to refactor that

[shashkat]

Hi @selmanozleyen , your welcome! Yeah no problem at all. I went through your changes and they look good to me. Just one point:
This is probably my bad, but in the functions calculate_niche_neighborhood, calculate_niche_utag, calculate_niche_spatialleiden, the order of args in docstrings (resolutions argument) doesn't seem to match the function signature.

Thanks a lot for reviewing the code!

[shashkat]

About the spatial_neighbors_from_builder, this is what we discussed: