Skip to content

Wire auto-pagination for query/ec2 services and LIMIT -> MaxResults pushdown #1

Description

@jeffreyaven

Describe the feature

Two related SQL-surface gaps for query/ec2 protocol services (e.g. aws.ec2.instance_types):

  1. Auto-pagination is not wired for query/ec2. Stage 2 only stamps config.pagination for
    rest-json/aws-json protocols (rule 18 in CLAUDE.md). For query/ec2 services stackql core falls
    back to Google-style token defaults (nextPageToken/pageToken) which never match AWS's
    NextToken, so traversal silently ends after page one. Docs also list NextToken as a
    WHERE-able param, implying manual pagination.
  2. LIMIT is not pushed to the API. A SELECT ... LIMIT 5 fetches the full (default-sized)
    page and trims client-side. The machinery to map LIMIT -> MaxResults already exists end to
    end in stackql core + any-sdk; this provider just doesn't emit the opt-in
    config.queryParamPushdown block.

Use Case

Users of query/ec2 services get silently truncated result sets: only the first page of a
multi-page Describe*/List* response is returned, with no error and no indication more rows
exist. The visible workaround - manually threading NextToken through repeated queries - defeats
the point of a SQL interface, and the sample queries in the docs actively steer users toward it
by listing NextToken/MaxResults as WHERE predicates.

Separately, LIMIT n queries over-fetch: the API returns its default page size (up to 1000 rows
for some ec2 ops) and stackql trims client-side, wasting latency and transfer when the user only
wanted a handful of rows.

Proposed Solution

Feature 1: pagination for query/ec2

Blocked on an any-sdk release cut, then a core bump. The blocker was stackql/any-sdk#117: the
schema-driven XML walker's output IS the rawBody that pagination inspects, and it dropped
sibling scalars including <nextToken>. Fixed on any-sdk HEAD (eff549b - write() in
pkg/stream_transform/schema_driven_xml.go passes scalar payload siblings through), but the
latest tag is v0.5.3-alpha11 and stackql core pins exactly that tag - the fix is unreleased.

  • any-sdk: tag a release containing eff549b (post-v0.5.3-alpha11)
  • stackql core: bump the any-sdk pin to that release
  • this repo (stage 2 generate-provider.mjs): lift the rest-json/json protocol gate in the
    pagination stamping; for query/ec2 emit config.pagination with request token NextToken
    (location query - ec2 upper-firsts the request locationName) and response token
    $.nextToken (the XML wire name, location body)
  • verify live: a multi-page ec2 Describe* traverses pages and preserves original params on
    re-injection (same verification as the DynamoDB ListTables case)
  • docs (cosmetic, independent): filter the param named by config.pagination.requestToken.key
    out of docgen sample queries (applies to already-wired services like dynamodb too);
    MaxResults stays visible

Feature 2: LIMIT -> MaxResults pushdown

No new plumbing needed for query/ec2 and rest-json. Core already extracts LIMIT into a neutral
PushdownIntent (internal/stackql/pushdown, wired via WithPushdownIntent), and any-sdk
v0.5.3-alpha11 contains the translation (applyPushdownTop, any-sdk PR boto#111). It is a no-op
unless the method carries config.queryParamPushdown; client-side LIMIT remains authoritative.
Ordering is safe for query/ec2: pushdown params are set at armoury-build time,
get_query_to_post_form_utf_8 runs later at dispatch, so MaxResults lands in the signed form
body.

  • this repo (stage 2): emit per-method opt-in config where the op has a MaxResults-style
    page-size param:
    yaml config: queryParamPushdown: top: paramName: MaxResults # per-op wire name (some APIs use Limit / maxResults) maxValue: <shape max>
  • handle the MaxResults minimum: many ops have a floor (DescribeInstanceTypes is min 5) and
    LIMIT 3 -> MaxResults=3 is an API validation error. any-sdk clamps max only. Options:
    minValue clamp in any-sdk, generator rounds up to the shape min (harmless - client-side
    LIMIT still trims), or only emit top where min is 1/absent. Botocore service-2.json
    carries min/max on the shape, so stage 1 has the data.
  • scope: query/ec2 + rest-json only initially. any-sdk ApplyPushdown emits query params
    exclusively; aws-json carries MaxResults/Limit as a body member, so pushing there needs
    body-located pushdown params in any-sdk (not yet supported).
  • precedence rule for an explicit WHERE MaxResults = ... colliding with a pushed LIMIT
    (pushdown currently overwrites the user's value via q.Set).

Interaction between the two (settle first)

MaxResults is a page size, not a total cap. Today (no ec2 pagination) LIMIT n ->
MaxResults=n yields exactly n rows in one call. Once pagination is wired, the core acquire loop
pages until the token runs dry, so LIMIT 5 would fetch every page at 5 rows per page - strictly
worse than not pushing. The HTTP acquire loop has no "stop when accumulated rows >= pushed limit"
early exit (the GraphQL path has the analogue via pushdown.SelectLimit).

  • stackql core: add LIMIT-satisfied early termination to the HTTP pagination loop
    (internal/stackql/execution/mono_valent_execution.go). Until that lands, either feature
    works alone; enabling both together regresses LIMIT queries on multi-page results.

Dependency summary

Dependency Repo Status
Walker drops sibling scalars (issue boto#117) stackql/any-sdk fixed on HEAD (eff549b), unreleased
any-sdk release + core pin bump stackql/any-sdk, stackql/stackql required for feature 1
queryParamPushdown translation (PR boto#111) stackql/any-sdk released in v0.5.3-alpha11 - no action
minValue clamp for top pushdown stackql/any-sdk optional (generator round-up is a workaround)
Body-located pushdown params (aws-json) stackql/any-sdk future scope
LIMIT-satisfied early exit in HTTP paging loop stackql/stackql required before enabling both features together

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

SDK version used

any-sdk v0.5.3-alpha11 (stackql core go.mod pin); walker fix present on any-sdk HEAD (eff549b, untagged)

Environment details (OS name and version, etc.)

N/A - provider generation pipeline / registry spec issue, platform-independent

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions