perf: route member-filtered activity relations to member-sorted mirror

[epipav]

#4499 (review)

[cursor]

PR Summary

Medium Risk
Touches a high-traffic Tinybird pipe and adds a full ~920M-row nightly replace copy. Routing bugs or a failed/stale mirror would affect member-feed latency and freshness, not auth or data mutation.

Overview
Speeds up the dominant member-feed shape of activities_relations_filtered by reading a new member-sorted mirror instead of scanning activityRelations_enriched_deduplicated_bucket_union.

Adds unpartitioned activityRelations_by_member_ds (sort key memberId, timestamp) and a 03:35 UTC replace-mode copy from the bucket union. When G1_memberIds is set and no G2–G5 params are present, Node 1 uses the mirror; all other shapes keep the union. Both sources are the same rows, so routing should not change results.

Local CLI now pushes the new datasource. Freshness tracks the nightly bucket snapshots; a failed copy keeps yesterday’s data.

^{Reviewed by Cursor Bugbot for commit 24586c5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6a9b77e. Configure here.}

[Copilot]

Pull request overview

Routes member-filtered activity queries through a member-sorted Tinybird mirror to reduce scanned rows and improve feed latency.

Changes:

• Adds a member-sorted activity-relations datasource.
• Rebuilds the mirror nightly using an atomic replace copy.
• Conditionally routes eligible member queries to the mirror.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
services/libs/tinybird/pipes/activityRelations_by_member_copy.pipe	Schedules nightly mirror rebuilding.
services/libs/tinybird/pipes/activities_relations_filtered.pipe	Adds conditional mirror routing.
services/libs/tinybird/datasources/activityRelations_by_member_ds.datasource	Defines the member-sorted mirror.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

services/libs/tinybird/pipes/activities_relations_filtered.pipe:101

• The fast path is selected even when the mirror is not ready or fresh. A newly created datasource is empty until the first 03:35 copy run, and the copy pipe explicitly retains yesterday's snapshot after a failure, while this branch continues serving it instead of the current bucket union. That makes member-filtered responses empty during rollout or stale after copy failures. Populate the mirror before publishing this route and add a readiness/freshness-controlled fallback to the union.

            {% if route_member_mirror == 1 %}
                FROM activityRelations_by_member_ds AS ar
            {% else %}
                FROM activityRelations_enriched_deduplicated_bucket_union AS ar

scripts/cli:759

• The local migration creates the mirror datasource and immediately publishes an endpoint that reads it, but it never pushes or runs activityRelations_by_member_copy. Since scheduled COPY pipes do not execute merely because their target datasource was pushed, local member-filtered queries remain empty. Push the copy pipe and run an initial copy before publishing the endpoint.

    tb push datasources/activityRelations_by_member_ds.datasource

services/libs/tinybird/pipes/activities_relations_filtered.pipe:35

• This duplicates the G2–G5 group-presence detection already implemented below as has_g2 through has_g5. Every future group parameter must now be added in two places, and drift can silently route an unsupported query shape to the mirror. Compute the group-presence flags once before source selection and derive route_member_mirror from them.

This issue also appears on line 98 of the same file.

    {% set route_member_mirror = 0 %}
    {% if defined(G1_memberIds) %} {% set route_member_mirror = 1 %} {% end %}
    {% if defined(G2_memberIds) %} {% set route_member_mirror = 0 %} {% end %}
    {% if defined(G2_memberIds_exclude) %} {% set route_member_mirror = 0 %} {% end %}

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/cli:759

• This local migration creates the mirror datasource and then publishes a pipe that routes member-filtered queries to it, but it never pushes/runs activityRelations_by_member_copy or otherwise loads the datasource. Consequently the local member-feed path always returns no rows even when activity relations exist. Populate the mirror during local migration, or keep local routing on a datasource that the migration actually loads.

    tb push datasources/activityRelations_by_member_ds.datasource

services/libs/tinybird/pipes/activities_relations_filtered.pipe:100

• The fast path switches to the mirror without verifying that it has been populated or is current. A newly deployed scheduled Copy Pipe does not run until the next 03:35 cron, so member-filtered requests return an empty result in that interval; after a failed nightly copy they continue reading yesterday's mirror while other filter shapes read the newer bucket union. Populate the mirror before enabling this branch and gate or fall back when its snapshot is stale.

            {% if route_member_mirror == 1 %} FROM activityRelations_by_member_ds AS ar
            {% else %} FROM activityRelations_enriched_deduplicated_bucket_union AS ar
            {% end %}

[gaspergrom]

LGTM!