perf: reduce memory usage in build_model (sparse coefficients + per-effect share constraints) (#595)

* fix: memory issues due to dense large coeficients

1. flixopt/features.py — Added sparse_multiply_sum() function that takes a sparse dict of (group_id, sum_id) -> coefficient instead of a dense DataArray. This avoids ever
  allocating the massive dense array.
  2. flixopt/elements.py — Replaced _coefficients (dense DataArray) and _flow_sign (dense DataArray) with a single _signed_coefficients cached property that returns
  dict[tuple[str, str], float | xr.DataArray] containing only non-zero signed coefficients. Updated create_linear_constraints to use sparse_multiply_sum instead of
  sparse_weighted_sum.

  The dense allocation at line 2385 (np.zeros(n_conv, max_eq, n_flows, *time) ~14.5 GB) is completely eliminated. Memory usage is now proportional to the number of non-zero
  entries (typically 2-3 flows per converter) rather than the full cartesian product.

* fix(effects): avoid massive memory allocation in share variable creation

Replace linopy.align(join='outer') with per-contributor accumulation
and linopy.merge(dim='contributor'). The old approach reindexed ALL
dimensions via xr.where(), allocating ~12.7 GB of dense arrays.
Now contributions are split by contributor at registration time and
accumulated via linopy addition (cheap for same-shape expressions),
then merged along the disjoint contributor dimension.

* Switch to per contributor constraints to solve memmory issues

* fix(effects): avoid massive memory allocation in share variable creation

Replace linopy.align(join='outer') with per-contributor accumulation
and individual constraints. The old approach reindexed ALL dimensions
via xr.where(), allocating ~12.7 GB of dense arrays.

Now contributions are split by contributor at registration time and
accumulated via linopy addition (cheap for same-shape expressions).
Each contributor gets its own constraint, avoiding any cross-contributor
alignment. Reduces effects expression memory from 1.2 GB to 5 MB.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Switch to per contributor constraints to solve memmory issues

* perf: improve bus balance to be more memmory efficient

* Switch to per effect shares

* Firs succesfull drop to 10 GB

* Make more readable

* Go back to one variable for all shares

* ⏺ Instead of adding zero-constraints for uncovered combos, we should just set lower=0, upper=0 on those entries (fix the bounds), or better yet — use a mask on the per-effect
  constraints and set the variable bounds to 0 for uncovered combos. The simplest fix: create the variable with lower=0, upper=0 by default, then only the covered entries need
  constraints.

* Only create variables needed

* _create_share_var went from 1,674ms → 116ms — a 14x speedup! The reindex + + approach is much faster than per-contributor sel + merge

* Revert

* Revert

* 1. effects.py: add_temporal_contribution and add_periodic_contribution now raise ValueError if a DataArray has no effect dimension and no effect= argument is provided.
  2. statistics_accessor.py: Early return with empty xr.Dataset() when no contributors are detected, preventing xr.concat from failing on an empty list.

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

Author: FBumann

flixopt/components.py

@@ -917,25 +917,59 @@ def add_effect_contributions(self, effects_model) -> None:
         if inv.effects_per_size is not None:
             factors = inv.effects_per_size
             size = self.size.sel({dim: factors.coords[dim].values})
-            effects_model.add_periodic_contribution(size * factors, contributor_dim=dim)
+            for eid in factors.coords['effect'].values:
+                f_single = factors.sel(effect=eid, drop=True)
+                if (f_single == 0).all():
+                    continue
+                effects_model.add_periodic_contribution(size * f_single, contributor_dim=dim, effect=str(eid))
 
             # Investment/retirement effects
             invested = self.invested
             if invested is not None:
-                if (f := inv.effects_of_investment) is not None:
-                    effects_model.add_periodic_contribution(
-                        invested.sel({dim: f.coords[dim].values}) * f, contributor_dim=dim
-                    )
-                if (f := inv.effects_of_retirement) is not None:
-                    effects_model.add_periodic_contribution(
-                        invested.sel({dim: f.coords[dim].values}) * (-f), contributor_dim=dim
-                    )
+                if (ff := inv.effects_of_investment) is not None:
+                    for eid in ff.coords['effect'].values:
+                        f_single = ff.sel(effect=eid, drop=True)
+                        if (f_single == 0).all():
+                            continue
+                        effects_model.add_periodic_contribution(
+                            invested.sel({dim: f_single.coords[dim].values}) * f_single,
+                            contributor_dim=dim,
+                            effect=str(eid),
+                        )
+                if (ff := inv.effects_of_retirement) is not None:
+                    for eid in ff.coords['effect'].values:
+                        f_single = ff.sel(effect=eid, drop=True)
+                        if (f_single == 0).all():
+                            continue
+                        effects_model.add_periodic_contribution(
+                            invested.sel({dim: f_single.coords[dim].values}) * (-f_single),
+                            contributor_dim=dim,
+                            effect=str(eid),
+                        )
 
         # === Constants: mandatory fixed + retirement ===
         if inv.effects_of_investment_mandatory is not None:
-            effects_model.add_periodic_contribution(inv.effects_of_investment_mandatory, contributor_dim=dim)
+            mandatory = inv.effects_of_investment_mandatory
+            if 'effect' in mandatory.dims:
+                for eid in mandatory.coords['effect'].values:
+                    effects_model.add_periodic_contribution(
+                        mandatory.sel(effect=eid, drop=True),
+                        contributor_dim=dim,
+                        effect=str(eid),
+                    )
+            else:
+                effects_model.add_periodic_contribution(mandatory, contributor_dim=dim)
         if inv.effects_of_retirement_constant is not None:
-            effects_model.add_periodic_contribution(inv.effects_of_retirement_constant, contributor_dim=dim)
+            ret_const = inv.effects_of_retirement_constant
+            if 'effect' in ret_const.dims:
+                for eid in ret_const.coords['effect'].values:
+                    effects_model.add_periodic_contribution(
+                        ret_const.sel(effect=eid, drop=True),
+                        contributor_dim=dim,
+                        effect=str(eid),
+                    )
+            else:
+                effects_model.add_periodic_contribution(ret_const, contributor_dim=dim)
 
     # --- Investment Cached Properties ---
 

flixopt/effects.py

@@ -345,9 +345,9 @@ def __init__(self, model: FlowSystemModel, data):
         self.share_periodic: linopy.Variable | None = None
 
         # Registered contributions from type models (FlowsModel, StoragesModel, etc.)
-        # Each entry: a defining_expr with 'contributor' dim
-        self._temporal_share_defs: list[linopy.LinearExpression] = []
-        self._periodic_share_defs: list[linopy.LinearExpression] = []
+        # Per-effect, per-contributor accumulation: effect_id -> {contributor_id -> expr (no effect dim)}
+        self._temporal_shares: dict[str, dict[str, linopy.LinearExpression]] = {}
+        self._periodic_shares: dict[str, dict[str, linopy.LinearExpression]] = {}
         # Constant (xr.DataArray) contributions with 'contributor' + 'effect' dims
         self._temporal_constant_defs: list[xr.DataArray] = []
         self._periodic_constant_defs: list[xr.DataArray] = []
@@ -361,35 +361,76 @@ def effect_index(self):
         """Public access to the effect index for type models."""
         return self.data.effect_index
 
-    def add_temporal_contribution(self, defining_expr, contributor_dim: str = 'contributor') -> None:
+    def add_temporal_contribution(
+        self,
+        defining_expr,
+        contributor_dim: str = 'contributor',
+        effect: str | None = None,
+    ) -> None:
         """Register contributors for the share|temporal variable.
 
         Args:
-            defining_expr: Expression with a contributor dimension.
-                Accepts linopy LinearExpression/Variable or plain xr.DataArray (constants).
+            defining_expr: Expression with a contributor dimension (no effect dim if effect is given).
             contributor_dim: Name of the element dimension to rename to 'contributor'.
+            effect: If provided, the expression is for this specific effect (no effect dim needed).
         """
         if contributor_dim != 'contributor':
             defining_expr = defining_expr.rename({contributor_dim: 'contributor'})
         if isinstance(defining_expr, xr.DataArray):
+            if effect is not None:
+                defining_expr = defining_expr.expand_dims(effect=[effect])
+            elif 'effect' not in defining_expr.dims:
+                raise ValueError(
+                    "DataArray contribution must have an 'effect' dimension or an explicit effect= argument."
+                )
             self._temporal_constant_defs.append(defining_expr)
         else:
-            self._temporal_share_defs.append(defining_expr)
+            self._accumulate_shares(self._temporal_shares, self._as_expression(defining_expr), effect)
 
-    def add_periodic_contribution(self, defining_expr, contributor_dim: str = 'contributor') -> None:
+    def add_periodic_contribution(
+        self,
+        defining_expr,
+        contributor_dim: str = 'contributor',
+        effect: str | None = None,
+    ) -> None:
         """Register contributors for the share|periodic variable.
 
         Args:
-            defining_expr: Expression with a contributor dimension.
-                Accepts linopy LinearExpression/Variable or plain xr.DataArray (constants).
+            defining_expr: Expression with a contributor dimension (no effect dim if effect is given).
             contributor_dim: Name of the element dimension to rename to 'contributor'.
+            effect: If provided, the expression is for this specific effect (no effect dim needed).
         """
         if contributor_dim != 'contributor':
             defining_expr = defining_expr.rename({contributor_dim: 'contributor'})
         if isinstance(defining_expr, xr.DataArray):
+            if effect is not None:
+                defining_expr = defining_expr.expand_dims(effect=[effect])
+            elif 'effect' not in defining_expr.dims:
+                raise ValueError(
+                    "DataArray contribution must have an 'effect' dimension or an explicit effect= argument."
+                )
             self._periodic_constant_defs.append(defining_expr)
         else:
-            self._periodic_share_defs.append(defining_expr)
+            self._accumulate_shares(self._periodic_shares, self._as_expression(defining_expr), effect)
+
+    @staticmethod
+    def _accumulate_shares(
+        accum: dict[str, list],
+        expr: linopy.LinearExpression,
+        effect: str | None = None,
+    ) -> None:
+        """Append expression to per-effect list."""
+        # accum structure: {effect_id: [(expr, contributor_ids), ...]}
+        if effect is not None:
+            # Expression has no effect dim — tagged with specific effect
+            accum.setdefault(effect, []).append(expr)
+        elif 'effect' in expr.dims:
+            # Expression has effect dim — split per effect (DataArray sel is cheap)
+            for eid in expr.data.coords['effect'].values:
+                eid_str = str(eid)
+                accum.setdefault(eid_str, []).append(expr.sel(effect=eid, drop=True))
+        else:
+            raise ValueError('Expression must have effect dim or effect parameter must be given')
 
     def create_variables(self) -> None:
         """Create batched effect variables with 'effect' dimension."""
@@ -542,19 +583,19 @@ def finalize_shares(self) -> None:
         if (sm := self.model._storages_model) is not None:
             sm.add_effect_contributions(self)
 
-        # === Create share|temporal variable ===
-        if self._temporal_share_defs:
-            self.share_temporal = self._create_share_var(self._temporal_share_defs, 'share|temporal', temporal=True)
+        # === Create share|temporal variable (one combined with contributor × effect dims) ===
+        if self._temporal_shares:
+            self.share_temporal = self._create_share_var(self._temporal_shares, 'share|temporal', temporal=True)
             self._eq_per_timestep.lhs -= self.share_temporal.sum('contributor')
 
         # === Apply temporal constants directly ===
         for const in self._temporal_constant_defs:
             self._eq_per_timestep.lhs -= const.sum('contributor').reindex({'effect': self.data.effect_index})
 
-        # === Create share|periodic variable ===
-        if self._periodic_share_defs:
-            self.share_periodic = self._create_share_var(self._periodic_share_defs, 'share|periodic', temporal=False)
-            self._eq_periodic.lhs -= self.share_periodic.sum('contributor').reindex({'effect': self.data.effect_index})
+        # === Create share|periodic variable (one combined with contributor × effect dims) ===
+        if self._periodic_shares:
+            self.share_periodic = self._create_share_var(self._periodic_shares, 'share|periodic', temporal=False)
+            self._eq_periodic.lhs -= self.share_periodic.sum('contributor')
 
         # === Apply periodic constants directly ===
         for const in self._periodic_constant_defs:
@@ -573,39 +614,67 @@ def _share_coords(self, element_dim: str, element_index, temporal: bool = True)
 
     def _create_share_var(
         self,
-        share_defs: list[linopy.LinearExpression],
+        accum: dict[str, list[linopy.LinearExpression]],
         name: str,
         temporal: bool,
     ) -> linopy.Variable:
-        """Create a share variable from registered contributor definitions.
+        """Create one share variable with (contributor, effect, ...) dims.
+
+        accum structure: {effect_id: [expr1, expr2, ...]} where each expr has
+        (contributor, ...other_dims) dims — no effect dim.
+
+        Constraints are added per-effect: var.sel(effect=eid) == merged_for_eid,
+        which avoids cross-effect alignment.
 
-        Aligns all contributor expressions (outer join on contributor dimension),
-        then sums them to produce a single expression with the full contributor dimension.
+        Returns:
+            linopy.Variable with dims (contributor, effect, time/period).
         """
         import pandas as pd
 
-        # Ensure all share defs have canonical effect order before alignment.
-        # linopy merge uses join="override" when shapes match, which aligns by
-        # position not label — mismatched effect order silently shuffles coefficients.
+        if not accum:
+            return None
+
+        # Collect all contributor IDs across all effects
+        all_contributor_ids: set[str] = set()
+        for expr_list in accum.values():
+            for expr in expr_list:
+                all_contributor_ids.update(str(c) for c in expr.data.coords['contributor'].values)
+
+        contributor_index = pd.Index(sorted(all_contributor_ids), name='contributor')
         effect_index = self.data.effect_index
-        normalized = []
-        for expr in share_defs:
-            if 'effect' in expr.dims:
-                expr_effects = list(expr.data.coords['effect'].values)
-                if expr_effects != list(effect_index):
-                    expr = linopy.LinearExpression(expr.data.reindex(effect=effect_index), expr.model)
-            normalized.append(expr)
-
-        aligned = linopy.align(*normalized, join='outer', fill_value=0)
-        combined_expr = sum(aligned[1:], start=aligned[0])
-
-        # Extract contributor IDs from the combined expression
-        all_ids = [str(cid) for cid in combined_expr.data.coords['contributor'].values]
-        contributor_index = pd.Index(all_ids, name='contributor')
         coords = self._share_coords('contributor', contributor_index, temporal=temporal)
-        var = self.model.add_variables(lower=-np.inf, upper=np.inf, coords=coords, name=name)
 
-        self.model.add_constraints(var == combined_expr, name=name)
+        # Build mask: only create variables for (effect, contributor) combos that have expressions
+        mask = xr.DataArray(
+            np.zeros((len(contributor_index), len(effect_index)), dtype=bool),
+            dims=['contributor', 'effect'],
+            coords={'contributor': contributor_index, 'effect': effect_index},
+        )
+        covered_map: dict[str, list[str]] = {}
+        for eid, expr_list in accum.items():
+            cids = set()
+            for expr in expr_list:
+                cids.update(str(c) for c in expr.data.coords['contributor'].values)
+            covered_map[eid] = sorted(cids)
+            mask.loc[dict(effect=eid, contributor=covered_map[eid])] = True
+
+        var = self.model.add_variables(lower=-np.inf, upper=np.inf, coords=coords, name=name, mask=mask)
+
+        # Add per-effect constraints (only for covered combos)
+        for eid, expr_list in accum.items():
+            contributors = covered_map[eid]
+            if len(expr_list) == 1:
+                merged = expr_list[0].reindex(contributor=contributors)
+            else:
+                # Reindex all to common contributor set, then sum via linopy.merge (_term addition)
+                aligned = [e.reindex(contributor=contributors) for e in expr_list]
+                merged = aligned[0]
+                for a in aligned[1:]:
+                    merged = merged + a
+            var_slice = var.sel(effect=eid, contributor=contributors)
+            self.model.add_constraints(var_slice == merged, name=f'{name}({eid})')
+
+        accum.clear()
         return var
 
     def get_periodic(self, effect_id: str) -> linopy.Variable: