Describe the feature
Two related SQL-surface gaps for query/ec2 protocol services (e.g. aws.ec2.instance_types):
- Auto-pagination is not wired for query/ec2. Stage 2 only stamps
config.pagination for
rest-json/aws-json protocols (rule 18 in CLAUDE.md). For query/ec2 services stackql core falls
back to Google-style token defaults (nextPageToken/pageToken) which never match AWS's
NextToken, so traversal silently ends after page one. Docs also list NextToken as a
WHERE-able param, implying manual pagination.
LIMIT is not pushed to the API. A SELECT ... LIMIT 5 fetches the full (default-sized)
page and trims client-side. The machinery to map LIMIT -> MaxResults already exists end to
end in stackql core + any-sdk; this provider just doesn't emit the opt-in
config.queryParamPushdown block.
Use Case
Users of query/ec2 services get silently truncated result sets: only the first page of a
multi-page Describe*/List* response is returned, with no error and no indication more rows
exist. The visible workaround - manually threading NextToken through repeated queries - defeats
the point of a SQL interface, and the sample queries in the docs actively steer users toward it
by listing NextToken/MaxResults as WHERE predicates.
Separately, LIMIT n queries over-fetch: the API returns its default page size (up to 1000 rows
for some ec2 ops) and stackql trims client-side, wasting latency and transfer when the user only
wanted a handful of rows.
Proposed Solution
Feature 1: pagination for query/ec2
Blocked on an any-sdk release cut, then a core bump. The blocker was stackql/any-sdk#117: the
schema-driven XML walker's output IS the rawBody that pagination inspects, and it dropped
sibling scalars including <nextToken>. Fixed on any-sdk HEAD (eff549b - write() in
pkg/stream_transform/schema_driven_xml.go passes scalar payload siblings through), but the
latest tag is v0.5.3-alpha11 and stackql core pins exactly that tag - the fix is unreleased.
Feature 2: LIMIT -> MaxResults pushdown
No new plumbing needed for query/ec2 and rest-json. Core already extracts LIMIT into a neutral
PushdownIntent (internal/stackql/pushdown, wired via WithPushdownIntent), and any-sdk
v0.5.3-alpha11 contains the translation (applyPushdownTop, any-sdk PR boto#111). It is a no-op
unless the method carries config.queryParamPushdown; client-side LIMIT remains authoritative.
Ordering is safe for query/ec2: pushdown params are set at armoury-build time,
get_query_to_post_form_utf_8 runs later at dispatch, so MaxResults lands in the signed form
body.
Interaction between the two (settle first)
MaxResults is a page size, not a total cap. Today (no ec2 pagination) LIMIT n ->
MaxResults=n yields exactly n rows in one call. Once pagination is wired, the core acquire loop
pages until the token runs dry, so LIMIT 5 would fetch every page at 5 rows per page - strictly
worse than not pushing. The HTTP acquire loop has no "stop when accumulated rows >= pushed limit"
early exit (the GraphQL path has the analogue via pushdown.SelectLimit).
Dependency summary
| Dependency |
Repo |
Status |
| Walker drops sibling scalars (issue boto#117) |
stackql/any-sdk |
fixed on HEAD (eff549b), unreleased |
| any-sdk release + core pin bump |
stackql/any-sdk, stackql/stackql |
required for feature 1 |
queryParamPushdown translation (PR boto#111) |
stackql/any-sdk |
released in v0.5.3-alpha11 - no action |
minValue clamp for top pushdown |
stackql/any-sdk |
optional (generator round-up is a workaround) |
| Body-located pushdown params (aws-json) |
stackql/any-sdk |
future scope |
| LIMIT-satisfied early exit in HTTP paging loop |
stackql/stackql |
required before enabling both features together |
Other Information
No response
Acknowledgements
SDK version used
any-sdk v0.5.3-alpha11 (stackql core go.mod pin); walker fix present on any-sdk HEAD (eff549b, untagged)
Environment details (OS name and version, etc.)
N/A - provider generation pipeline / registry spec issue, platform-independent
Describe the feature
Two related SQL-surface gaps for query/ec2 protocol services (e.g.
aws.ec2.instance_types):config.paginationforrest-json/aws-json protocols (rule 18 in CLAUDE.md). For query/ec2 services stackql core falls
back to Google-style token defaults (
nextPageToken/pageToken) which never match AWS'sNextToken, so traversal silently ends after page one. Docs also listNextTokenas aWHERE-able param, implying manual pagination.
LIMITis not pushed to the API. ASELECT ... LIMIT 5fetches the full (default-sized)page and trims client-side. The machinery to map
LIMIT->MaxResultsalready exists end toend in stackql core + any-sdk; this provider just doesn't emit the opt-in
config.queryParamPushdownblock.Use Case
Users of query/ec2 services get silently truncated result sets: only the first page of a
multi-page Describe*/List* response is returned, with no error and no indication more rows
exist. The visible workaround - manually threading
NextTokenthrough repeated queries - defeatsthe point of a SQL interface, and the sample queries in the docs actively steer users toward it
by listing
NextToken/MaxResultsas WHERE predicates.Separately,
LIMIT nqueries over-fetch: the API returns its default page size (up to 1000 rowsfor some ec2 ops) and stackql trims client-side, wasting latency and transfer when the user only
wanted a handful of rows.
Proposed Solution
Feature 1: pagination for query/ec2
Blocked on an any-sdk release cut, then a core bump. The blocker was stackql/any-sdk#117: the
schema-driven XML walker's output IS the
rawBodythat pagination inspects, and it droppedsibling scalars including
<nextToken>. Fixed on any-sdk HEAD (eff549b-write()inpkg/stream_transform/schema_driven_xml.gopasses scalar payload siblings through), but thelatest tag is
v0.5.3-alpha11and stackql core pins exactly that tag - the fix is unreleased.eff549b(post-v0.5.3-alpha11)generate-provider.mjs): lift the rest-json/json protocol gate in thepagination stamping; for query/ec2 emit
config.paginationwith request tokenNextToken(location
query- ec2 upper-firsts the request locationName) and response token$.nextToken(the XML wire name, locationbody)re-injection (same verification as the DynamoDB ListTables case)
config.pagination.requestToken.keyout of docgen sample queries (applies to already-wired services like dynamodb too);
MaxResultsstays visibleFeature 2: LIMIT -> MaxResults pushdown
No new plumbing needed for query/ec2 and rest-json. Core already extracts LIMIT into a neutral
PushdownIntent(internal/stackql/pushdown, wired viaWithPushdownIntent), and any-sdkv0.5.3-alpha11contains the translation (applyPushdownTop, any-sdk PR boto#111). It is a no-opunless the method carries
config.queryParamPushdown; client-side LIMIT remains authoritative.Ordering is safe for query/ec2: pushdown params are set at armoury-build time,
get_query_to_post_form_utf_8runs later at dispatch, soMaxResultslands in the signed formbody.
page-size param:
yaml config: queryParamPushdown: top: paramName: MaxResults # per-op wire name (some APIs use Limit / maxResults) maxValue: <shape max>LIMIT 3->MaxResults=3is an API validation error. any-sdk clamps max only. Options:minValueclamp in any-sdk, generator rounds up to the shape min (harmless - client-sideLIMIT still trims), or only emit
topwhere min is 1/absent. Botocoreservice-2.jsoncarries min/max on the shape, so stage 1 has the data.
ApplyPushdownemits query paramsexclusively; aws-json carries
MaxResults/Limitas a body member, so pushing there needsbody-located pushdown params in any-sdk (not yet supported).
WHERE MaxResults = ...colliding with a pushed LIMIT(pushdown currently overwrites the user's value via
q.Set).Interaction between the two (settle first)
MaxResultsis a page size, not a total cap. Today (no ec2 pagination)LIMIT n->MaxResults=nyields exactly n rows in one call. Once pagination is wired, the core acquire looppages until the token runs dry, so
LIMIT 5would fetch every page at 5 rows per page - strictlyworse than not pushing. The HTTP acquire loop has no "stop when accumulated rows >= pushed limit"
early exit (the GraphQL path has the analogue via
pushdown.SelectLimit).(
internal/stackql/execution/mono_valent_execution.go). Until that lands, either featureworks alone; enabling both together regresses LIMIT queries on multi-page results.
Dependency summary
eff549b), unreleasedqueryParamPushdowntranslation (PR boto#111)v0.5.3-alpha11- no actionminValueclamp for top pushdownOther Information
No response
Acknowledgements
SDK version used
any-sdk v0.5.3-alpha11 (stackql core go.mod pin); walker fix present on any-sdk HEAD (eff549b, untagged)
Environment details (OS name and version, etc.)
N/A - provider generation pipeline / registry spec issue, platform-independent