performance: Refactor BasePlot & subclasses to copy less

[dorzhey]

• Closes #3718
• [Tests][plotting] this is a pure refactor of plotting internals, no new behavior, and all existing plotting tests pass locally (the one failing heatmap test also fails on main in my environment).

• Release notes not necessary because: there are no user-facing API changes, only internal performance and memory optimizations.

[dorzhey]

Ah, I failed some of the style checks. I apologize, pre-commit does not work on my environment for some reason, gives error with UTF encoding and tests fail on main

[flying-sheep]

no worries, I fixed it for you

[flying-sheep]

OK, I got rid of the “converting index to string” warnings that obscured the actual test failures, so you should be able to see which tests failed now: https://github.com/scverse/scanpy/actions/runs/16826467121/job/47664011892?pr=3757#step:6:7989

[gkneighb]

Thanks for tackling this @dorzhey — the move toward sc.get.aggregate is the right direction and matches what @ilan-gold suggested in #3718. A few things I noticed reading the diff:

Correctness: dotplot's mean_only_expressed path looks broken

In _dotplot.py, the new mean_only_expressed and expression_cutoff > 0 branch is:

mask = expression_cutoff < self._view.X
df_sum = self._agg_df("sum", mask=mask)
expr_counts = dot_size_df.values * group_sizes[:, None]
dot_color_df = df_sum.div(expr_counts).fillna(0)

But _agg_df(..., mask=mask) does view.X = mask.astype(view.X.dtype) and then aggregates — so:

• df_sum is the sum of the mask (= count of expressed cells per group/gene), not of X
• expr_counts = dot_size_df * group_size = fraction-expressed × group-size = count of expressed cells per group/gene

i.e. dot_color_df = count_expressed / count_expressed = 1.0 wherever any cell expresses (and 0 otherwise). The actual expression values from self._view.X are never read in this branch.

The semantic on main is "mean expression value, restricted to cells where expression > cutoff":

# main
self.obs_tidy.mask(~obs_bool).groupby(level=0, observed=True).mean().fillna(0)

A sc.get.aggregate-friendly equivalent would be to aggregate X * mask (sum of expressed values) and divide by the count of expressed cells:

mask = self._view.X > expression_cutoff
df_sum = self._agg_df("sum", x_override=self._view.X * mask)  # see #2 below
df_count = self._agg_df("sum", x_override=mask.astype(self._view.X.dtype))
dot_color_df = df_sum.div(df_count).fillna(0)

Worth adding a unit test for this case — comparing the new code to the main implementation on a small fixture would have caught it.

Also: the mean_only_expressed and expression_cutoff == 0 case is silently treated as "mean over all cells" rather than "mean over nonzero cells", which is the most common usage. That's a separate behavior regression in the same area.

API: the mask parameter does double duty

_agg_df's docstring says mask is "shape (n_groups, n_vars)" but every call site passes a (n_obs, n_vars) matrix, because the parameter actually overwrites view.X, not filters it. Renaming to x_override (or splitting into a separate method) would make this honest at the API surface.

Minor

• view = self._view.copy() in _agg_df runs on every aggregation. For sparse X that copy could dominate the savings the refactor is going for. Worth skipping the copy when mask is None.
• self._view = AnnData(X=obs_tidy.values, ...) densifies and copies (.values materializes a contiguous ndarray). Per the comment, _prepare_dataframe is still needed, so the upstream copy isn't avoided — most of the wins are downstream of construction. Worth noting in the PR body so reviewers don't expect init-time savings.
• The PR currently shows as CONFLICTING with main; needs a rebase once the above is sorted.

The dotplot/matrixplot/violin layering looks much cleaner with _scale_df and _agg_df factored out — that's a nice cleanup independent of the perf wins.