Skip to content

Batch calculated-channel solve into temp tables, mirroring aggregations/events - #80

Merged
tombonfert merged 2 commits into
mainfrom
feature/calc_channels_batching
Aug 13, 2026
Merged

Batch calculated-channel solve into temp tables, mirroring aggregations/events#80
tombonfert merged 2 commits into
mainfrom
feature/calc_channels_batching

Conversation

@tombonfert

@tombonfert tombonfert commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

What

Calculated channels are now solved in batches, following the same pattern
already used for aggregations and events. The batching is driven from the Report
class: the channels are partitioned by query_engine.batch_size, each batch is
solved and persisted as a Unity Catalog temp table
(__impulse_temp_{run_id}_{batch_idx}), and the batches are unioned into the final
calculated_channel_fact.

Why

Previously all calculated channels were solved in a single
query.select(*channels).solve_calculated_channels(...) call, unlike aggregations
and events which batch by batch_size and persist intermediate results as temp
tables. Large channel sets therefore built one big lazy plan with no intermediate
materialization. This change gives calculated channels the same bounded,
batch-at-a-time execution and a consistent orchestration shape across all entity
types.

How

  • New solve_calculated_channels_batched (report_utils.py): the narrow,
    row-append counterpart to solve_expressions_batched. It reuses build_batches
    (selector-aware, batch_size = max unique selectors per batch), solves each batch
    via query.select(*batch).solve_calculated_channels(...), persists each as a temp
    Delta table (or a Spark temp view when sinkless), and combines them with
    unionByName. This is the one deliberate difference from the wide solver, whose
    batches are one-row-per-container and combined with a container_id join.
  • Batching moved to Report: new Report._solve_calculated_channels_batched
    (parallel to _solve_expressions_batched); determine_report collects the
    changed/unchanged channel expressions, batch-solves each into a narrow
    solved_df, and dispatches those.
  • dispatch_calculated_channels now takes solved_df (dropping
    query/solver), mirroring dispatch_aggregations.
  • CalculatedChannel.determine_calculated_channels no longer solves; it shapes
    the already-solved solved_df (filters by channel_id, projects to the fact
    schema), like determine_aggregations.
  • Temp tables share the existing __impulse_temp_* prefix, so the start-of-run
    cleanup and the unity_sink.cleanup_temp_tables flag cover them automatically. No
    new config; reuses query_engine.batch_size. The add_calculated_channel API is
    unchanged.

Tests

  • Unit: TestSolveCalculatedChannelsBatched (sinkless temp view, sink Delta table,
    unionByName not join, per-batch select, unique run_id) and the reworked
    TestDetermineCalculatedChannels (shaping: filter by channel_id + project).
  • Integration: test_batched_calculated_channels_union runs batch_size=1 with two
    distinct-selector channels and asserts both land in the union plus two temp tables
    are created; the RAW-mode test is updated to the new solve path.
  • Regression: the full report_utils unit suite and the batched-pipeline
    integration test confirm the shared batching/cleanup path still works for
    aggregations and events.

Docs

Updated the channel reference, the impulse-channels skill, and the batch_size
descriptions in the config doc/skill; regenerated the pydoc-markdown API reference.

Test Plan

  • Unit tests added/updated
  • Manual testing completed
  • Documentation updated (if applicable)

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • No new linter warnings introduced

…ow solved_df

- Move the narrow solve for calculated channels from each channel type into
  `Report._solve_calculated_channels_batched`, mirroring the existing batched
  expression solve path.
- Add `solve_calculated_channels_batched` in `report_utils` to solve channels in
  configurable batches, persist each batch as a `__impulse_temp_*` Delta table or
  temp view, and combine results with `unionByName`.
- Update `CalculatedChannel.determine_calculated_channels` and
  `dispatch_calculated_channels` to accept the pre-solved narrow DataFrame and
  only filter by `channel_id` and project to the fact schema.
- Update unit and integration tests to exercise the new batched path, including
  multi-batch union, sink vs. sinkless persistence, and RAW-mode shaping.
@tombonfert
tombonfert requested a review from a team as a code owner August 13, 2026 09:20
@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.89%. Comparing base (d0418d9) to head (3529a71).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
+ Coverage   88.83%   88.89%   +0.05%     
==========================================
  Files          61       61              
  Lines        5170     5194      +24     
  Branches      621      625       +4     
==========================================
+ Hits         4593     4617      +24     
  Misses        466      466              
  Partials      111      111              
Flag Coverage Δ
query_engine 84.57% <ø> (ø)
reporting 94.22% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...c/impulse_reporting/channels/calculated_channel.py 100.00% <100.00%> (ø)
src/impulse_reporting/core/report.py 92.06% <100.00%> (+0.15%) ⬆️
src/impulse_reporting/core/report_utils.py 97.41% <100.00%> (+0.21%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tombonfert
tombonfert merged commit f748c11 into main Aug 13, 2026
6 checks passed
@tombonfert
tombonfert deleted the feature/calc_channels_batching branch August 13, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant