Skip to content

Feat/cfn test v2 rqts runner - #1122

Open
ruslan-aws wants to merge 3 commits into
masterfrom
feat/cfn-test-v2-rqts-runner
Open

Feat/cfn test v2 rqts runner#1122
ruslan-aws wants to merge 3 commits into
masterfrom
feat/cfn-test-v2-rqts-runner

Conversation

@ruslan-aws

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

Adds an opt-in --v2 flag to cfn test that runs contract tests through the
RQTS (Resource Quality Test Suite) executor container instead of the existing
pytest-based suite. Without --v2, cfn test behavior is unchanged.

Motivation

RQTS is the container-based contract test runner already used for cloud
contract-test execution. This flag lets resource developers run the same
scenarios locally against a packaged handler artifact, so local results match
what the registry runs during RegisterType/TestType.

What's added

  • --v2 (default False): run the RQTS local test runner.
  • --rqts-image: override the executor image reference, for testing against a
    pre-release image. Defaults to the published
    public.ecr.aws/s5r7m5i4/cfn-rqts-executor-external:latest.
  • New rpdk.core.rqts package:
    • image.py — image resolution, and an anonymous docker pull attempted on
      every run (bounded to 3 attempts with a per-attempt timeout). If every
      attempt fails, a locally cached copy is used with a warning; only when no
      cached copy exists is the failure fatal.
    • preconditions.py — checks Docker availability (CLI on PATH plus a
      daemon ping), the built artifact package, and AWS credentials/region.
      Every unmet precondition is reported in a single error rather than failing
      on the first one.
    • argv.py — builds the docker run argv and the container environment.
    • runner.py — orchestrates the pipeline: artifact-type guard, preconditions,
      credential minting, image resolve/pull, container run with live output
      streaming, and exit-code mapping.

Behavior

The container is invoked in the executor's DirectJar handler mode:

docker run --rm \
  -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN -e AWS_REGION \
  -v <project-root>:/work \
  <image> \
  --extension contract-tests run-tests /work/<artifact>.zip \
  --direct-jar -r <region> -o /work/rqts-output

Notable points:

  • Credentials never enter the argv. The -e flags are name-only; values are
    supplied through the Docker client process environment. The full command line
    is therefore safe to log at DEBUG and never appears in ps output.
  • DirectJar mode, so the handler JAR is loaded in-process. No handler
    endpoint is probed and no host networking is configured.
  • Scenario selection is owned by the executor image, which gates scenarios by
    schema capability and resource-type namespace. The CLI passes no scenario or
    exclusion arguments.
  • Results are written to rqts-output/ under the project root. A zero container
    exit reports a pass; any non-zero exit raises the usual
    SysExitRecommendedError, which cli.py maps to exit code 1.
  • Resource types only. Hook projects are rejected with a clear message, and
    module projects short-circuit earlier as they do today.

Backward compatibility

  • The --v2 branch is additive and sits after the existing module
    short-circuit; the legacy path is untouched.
  • RqtsRunner is imported lazily inside the branch, so the pytest path pays no
    import cost.
  • Tests assert that without --v2 the pytest path runs, the runner is never
    constructed, and a non-zero pytest return still raises as before.
  • One user-visible side effect: --v2 makes the previously-undeclared --v
    abbreviation ambiguous with --verbose, so cfn test --v now errors with
    ambiguous option: --v could match --verbose, --v2. -v, -vv, --ve, and
    --verbose are unaffected.

Testing

  • 43 new tests in tests/rqts/, plus flag-registration and routing tests in
    tests/test_test.py. Hypothesis property tests cover image resolution, the
    bounded/anonymous pull, exact precondition aggregation, the bind mount,
    credentials never appearing in argv, DirectJar selection, region handling,
    absence of SAM Local/remote-lambda flags, absence of host networking, output
    directory, exit-code mapping, and absence of scenario/exclusion flags.
  • Docker and AWS are never actually invoked: subprocess.run, subprocess.Popen,
    and the credential helpers are patched throughout.
  • rpdk.core.rqts is at 100% statement and branch coverage.
  • Also verified end-to-end against a real Java resource project, confirming the
    image pull, container execution, streamed output, rqts-output/ results, and
    exit-code mapping.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Add an opt-in --v2 flag to cfn test that runs contract tests through
the RQTS (CTv2) executor container instead of the pytest-based suite.
Without --v2 the existing path is unchanged.

The new rpdk.core.rqts package resolves and pulls the pinned public
executor image with a bounded retry and a cached-image fallback,
aggregates Docker, artifact and credential preconditions into a single
error, builds the docker run argv, and orchestrates the run: mint
temporary credentials, stream container output, and map the container
exit code to the CLI result.

Credential values reach Docker only through the client process
environment. The argv carries name-only -e flags, so secrets never
appear in the command line or in debug logs. Handler invocation uses
DirectJar mode, so no host networking and no handler endpoint are
configured, and scenario selection is owned by the executor image.

Also add --rqts-image to override the image reference when testing
pre-release executor builds.
DEFAULT_HANDLER_PORT described the SAM Local handler endpoint, which the
DirectJar mode targeted by cfn test --v2 never uses: the handler JAR is
loaded in-process, so no port is bound and nothing referenced the value.
Bring the rqts package to full statement and branch coverage by
exercising the paths the existing tests bypassed:

- _run_docker reaching subprocess.run with a fixed, non-shell argv
- image_present_locally when docker inspect cannot run at all
- the Docker daemon ping exiting non-zero, raising, or timing out
- RqtsRunner.run aggregating unmet preconditions into one error
- build_docker_argv with explicit workdir, artifact and output overrides
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant