Skip to content

Add ops-based server filtering for history API#290

Open
blazeapps007 wants to merge 18 commits into
steemit:nextfrom
blazeapps007:next
Open

Add ops-based server filtering for history API#290
blazeapps007 wants to merge 18 commits into
steemit:nextfrom
blazeapps007:next

Conversation

@blazeapps007
Copy link
Copy Markdown
Contributor

Introduce server-side op-type filtering to /api/query/history (ops param) with batching, cursor advancement, nextFrom/exhausted pagination and Redis fallbacks. Update the client API (apiClient.getHistory) and hooks (use-batch-history, use-rewards-history, use-activity-history) to request server-filtered pages instead of doing client-side filtering; simplify hook logic to consume already-normalized SteemHistoryItem[] and track server-provided nextFrom/exhausted. Add WALLET/ACTIVITY/REWARD op constants (src/lib/steem/history-ops.ts), normalize/cache helpers, and MAX_BATCHES/BATCH_SIZE safeguards. Add devTools sanitizers to redact auth key fields in Redux DevTools and tests for the new behaviors; mark some local filtering helpers as deprecated. Also remove some UI icons from recent-activity and adjust related tests.

blazeapps007 and others added 15 commits April 25, 2026 14:36
Replace useSyncExternalStore with useState + useEffect in Header so the
remembered username is always null on the initial render (matching the
server), then updated after hydration. Apply the same deferred pattern
to useTheme, which was reading localStorage inside the useState
initializer — a value the server can never produce, causing React to
render different trees on server vs client.
Cross-tab sync — window.addEventListener('storage', ...) in the module init block. When another tab writes to THEME_KEY, _theme is updated and _notify() is called so all subscribers in this tab re-render.

Duplicated classList ops — both sites kept, with comments explaining why: changeTheme applies synchronously for FOUC prevention, useLayoutEffect is the safety net for mount and cross-tab updates that come through _notify().

Set mutation during iteration — all forEach calls now go through _notify(), which snapshots with Array.from(_listeners) before iterating.

cycleTheme stability — removed the theme dependency by reading _theme directly from the module store. cycleTheme is now stable (only depends on changeTheme, which is itself stable with []), so it won't invalidate memoized children if one is added later.
Replace the single shared store with a makeStore factory and create the store in Providers via a persistent useRef so each server request gets an isolated store (prevents auth state leaking) while the client keeps a single instance. Update store types to AppStore/RootState/AppDispatch. Clarify Header's useSyncExternalStore usage in a comment for SSR-safe avatar initialization. Make getCurrentTheme SSR-safe by returning the internal _theme instead of reading localStorage.
Lock in the SSR-isolation invariant introduced in this PR so a future
refactor cannot quietly revert makeStore back to a module-level singleton
without failing CI.
The repo's vitest config requires 80% line/statement coverage, but ui.ts
and wallet.ts had no tests, dragging the global stats to 79.54% lines /
77.30% statements and failing CI. Add reducer tests for every action in
both slices.

After this change:
- Lines:      79.54% -> 84.46%
- Statements: 77.30% -> 81.91%
Introduce AUTH_KEY_FIELDS and AuthKeyField type and update serializableCheck to ignore auth key paths and the auth/setCredentials action to avoid middleware warnings for private keys. Disable Redux DevTools in production entirely, and in development add stateSanitizer and actionSanitizer to replace live private-key fields with "[REDACTED]" in the auth slice and in setCredentials actions so keys are never shown in plain text to extensions or logs.
Introduce AUTH_KEY_FIELDS constant covering all five key fields in the
auth slice (ownerKey, activeKey, postingKey, memoKey, privateKey) and
apply two layers of protection:

Production:
  devTools: false — the Redux DevTools extension cannot inspect the
  store on a real user session under any circumstances.

Development:
  stateSanitizer replaces every non-null key field with '[REDACTED]'
  before the extension renders state, so a developer's own keys are
  never shown in plain text in the DevTools panel or time-travel log.

  actionSanitizer applies the same redaction to auth/setCredentials
  payloads so keys are not captured at the moment of login either.

Also fixes the pre-existing serializableCheck config which referenced
a non-existent action ('auth/setPrivateKey') and only listed one of
the five key paths — updated to 'auth/setCredentials' and all paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce server-side op-type filtering to /api/query/history (ops param) with batching, cursor advancement, nextFrom/exhausted pagination and Redis fallbacks. Update the client API (apiClient.getHistory) and hooks (use-batch-history, use-rewards-history, use-activity-history) to request server-filtered pages instead of doing client-side filtering; simplify hook logic to consume already-normalized SteemHistoryItem[] and track server-provided nextFrom/exhausted. Add WALLET/ACTIVITY/REWARD op constants (src/lib/steem/history-ops.ts), normalize/cache helpers, and MAX_BATCHES/BATCH_SIZE safeguards. Add devTools sanitizers to redact auth key fields in Redux DevTools and tests for the new behaviors; mark some local filtering helpers as deprecated. Also remove some UI icons from recent-activity and adjust related tests.
@blazeapps007
Copy link
Copy Markdown
Contributor Author

The WALLET_OP_TYPES whitelist in src/lib/steem/history-ops.ts is an explicit include list. Only operations in this set are returned; all other operation types are silently filtered out server-side.

Included operation types:

  • author_reward — post/comment reward payout
  • curation_reward — upvote reward
  • comment_benefactor_reward — beneficiary reward
  • transfer — send/receive STEEM/SBD
  • transfer_to_vesting — power up
  • transfer_to_savings — move to savings
  • transfer_from_savings — withdraw from savings
  • withdraw_vesting — start power down
  • fill_vesting_withdraw — periodic power down payout
  • claim_reward_balance — claim pending rewards
  • delegate_vesting_shares — delegate SP
  • return_vesting_delegation — delegation returned
  • interest — savings interest payment

Excluded operation types (never returned):

  • vote
  • comment
  • comment_options
  • account_witness_vote
  • account_witness_proxy
  • custom_json
  • limit_order_create
  • limit_order_cancel
  • fill_order
  • account_update
  • witness_update
  • set_withdraw_vesting_route
  • Any operation not related to balance movement or reward activity

The server scans raw Steem account history in batches and only accumulates entries where op[0] matches a whitelisted operation type. Non-whitelisted operations are discarded before the response reaches the client.

@ety001
Copy link
Copy Markdown
Member

ety001 commented May 22, 2026

Architectural Conflict with Existing Lazy-Load Design

The use-batch-history hook on next was designed around a client-controlled, incremental lazy-load loop:

Client loop (max 5 iterations)
  → 1 request to our API server (1 batch = 100 ops from api.steemit.com)
  → Client-side filter
  → Stop once 10 matches accumulated
  → loadMore() fetches next page

This PR changes it to:

Client loop (max 5 iterations)
  → 1 request to our API server
     → Server inner loop (max 20 iterations)
        → Each iteration pulls 100 ops from api.steemit.com
        → Server-side filter
     → Returns filtered results
  → Client accumulates

The Problem: Two Nested Loops

The server-side batching in fetchFiltered introduces an inner loop that the client cannot see or control. In the worst case a single initial page load triggers 5 × 20 = 100 calls to api.steemit.com, completely bypassing the traffic control that the lazy-load design was built to enforce.

Summary

Dimension Original lazy-load (next branch) PR 290 behavior
Steem API call control Client drives batch-by-batch Server inner loop (up to 20×)
Worst-case API calls on initial load 5 100 (5 × 20)
Filter location Client Server
Pagination granularity 100 ops per batch, fine-grained Server may scan up to 2,000 ops in one shot

Recommendation

If server-side filtering is desired, the server should handle only one batch per request (pull limit ops from Steem, filter, return matches + nextFrom). The client should retain control of the loop and pagination — preserving the incremental load pattern.

The limit parameter should constrain the raw fetch volume from api.steemit.com, not the filtered result count. This keeps the lazy-load contract intact and prevents request amplification.

@blazeapps007
Copy link
Copy Markdown
Contributor Author

Addressing these .

@ety001
Copy link
Copy Markdown
Member

ety001 commented May 22, 2026

Minor: requestedOps Not Deduplicated — Cache Key Pollution

In route.ts:35-37:

const requestedOps: string[] | null = opsParam
  ? opsParam.split(",").map((s) => s.trim()).filter(Boolean)
  : null;

And in route.ts:94:

const opsKey = [...requestedOps].sort().join("+");

There is no deduplication on requestedOps. A caller can send ops=transfer,transfer,transfer,... (repeated N times). Since every element passes the ALLOWED_OPS whitelist check, the request is accepted as valid, but:

  1. Cache key pollutiontransfer+transfer+transfer and transfer produce different Redis keys, wasting cache space. An attacker could generate arbitrarily many distinct keys.
  2. Unnecessary string concatenation — joining thousands of repeated strings before using the key.

Suggested fix — deduplicate right after the whitelist check:

if (requestedOps) {
  const invalid = requestedOps.find((o) => !ALLOWED_OPS.has(o));
  if (invalid) {
    return NextResponse.json({ error: `Unknown op type: ${invalid}` }, { status: 400 });
  }
  requestedOps = [...new Set(requestedOps)]; // deduplicate
}

After dedup, the array has at most 14 elements (the size of WALLET_OP_TYPES), which also eliminates the need for an explicit length cap.

@blazeapps007
Copy link
Copy Markdown
Contributor Author

I will Address all these Concerns and Commit

@ety001
Copy link
Copy Markdown
Member

ety001 commented May 22, 2026

Minor Suggestions

1. Duplicated oldestIndexIn logic in fetchFiltered

The manual loop to find the oldest index in a batch (fetchFiltered in route.ts) reimplements the same logic that existed as oldestIndexIn() in the pre-PR use-batch-history.ts:

// route.ts — inside fetchFiltered, written inline twice
let oldestInBatch: number | undefined;
for (const item of normalized) {
  if (typeof item.index === "number") {
    if (oldestInBatch === undefined || item.index < oldestInBatch)
      oldestInBatch = item.index;
  }
}

This appears twice in fetchFiltered — once during batch iteration and once on the final result. Consider extracting it into a shared utility function (e.g. in normalize-history.ts) so both server and client code can reuse it if needed.


2. Missing edge-case tests

Two scenarios are not covered by the current test suite:

  • Empty ops param (ops=): filter(Boolean) produces an empty array, which falls through to the non-filtered path. This works correctly but has no test coverage.
  • from + ops combined pagination: the existing tests cover from and ops independently, but not the case where a non-default from value is passed alongside ops (i.e. a loadMore request to the filtered endpoint).

Neither is blocking, but would strengthen regression coverage for future changes.


3. Content-Type header — no action needed

NextResponse.json() already sets Content-Type: application/json by default. No explicit header needed. Calling this out only to confirm it was checked.

Remove server-side multi-batch scanning and simplify filtered history path to make exactly one Steem RPC call per HTTP request. Key changes:

- Remove MAX_BATCHES and server-side limit/loop; fetchFiltered now issues one SteemService.getAccountHistory call and returns matching ops from that batch.
- Deduplicate requested ops list and adjust handler call signatures (limit removed from filtered path).
- Clamp fetchLimit to BATCH_SIZE or the provided from cursor (or BATCH_SIZE when from = -1) to avoid duplicates near history start.
- Compute nextFrom/exhausted using the oldest index in the whole batch (to advance correctly) and return matching history.
- Update tests to reflect the new single-RPC behavior and adjusted nextFrom/exhausted semantics (remove multi-batch expectations and adjust request params and assertions accordingly).

This moves pagination control to the client (one batch per request) and simplifies server logic, avoiding complex loops and potential off-by-one/duplication issues.
Add two unit tests to tests/unit/history-route-filter.test.ts: one verifies that when from is much larger than BATCH_SIZE the handler uses the full BATCH_SIZE as fetchLimit (load-more behavior), and the other verifies that an empty ops query param is treated as no filter (legacy path) so response omits nextFrom and exhausted. Tests use mockGetAccountHistory and makeTuples to simulate responses.
@blazeapps007
Copy link
Copy Markdown
Contributor Author

@ety001, you can review now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants