Introducing Harness Testing

[darklight3it]

📬 Issue #, if available:

N/A

✍️ Description of changes:

This PR adds Docker-based testing infrastructure using AWS's containerized-test-runner-for-aws-lambda, enabling automated testing of Lambda functions in a containerized environment that closely mirrors the AWS Lambda execution environment.

The PR introduces a test-dockerized Makefile target that runs test suites defined in test/dockerized/*.json files. These test suites specify handlers to test (from the examples directory), request payloads, and expected response assertions with optional jq transforms for validation.

The infrastructure reuses Lambda binaries from the /examples folder as test handlers, demonstrating the concept with an initial test case for basic-lambda. Additional tests and multi-concurrency scenarios can be added by creating new test suite JSON files.

A GitHub Actions workflow provides CI/CD integration for automated testing on pull requests.

Testing

Run dockerized tests locally:

make test-dockerized

Run RIE tests:

make test-rie
HANDLERS_TO_BUILD="basic-lambda basic-sqs" make test-rie

Build specific examples:

EXAMPLES="basic-lambda basic-lambda-concurrent" make build-examples

🔏 By submitting this pull request

• I confirm that I've ran cargo +nightly fmt.
• I confirm that I've ran cargo clippy --fix.
• I confirm that I've made a best effort attempt to update all relevant documentation.
• I confirm that my contribution is made under the terms of the Apache 2.0 license.on.
• I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jlizen]

Does this action exist? The ci is failing to find it, and I don't see references elsewhere on the Internet.

I wonder if the existing test-rie workflow is close to what you want here, and could be adapted?

Seems like the main thing missing would be a small script to translate the request mappings into curl requests that assert against the response? Along with some parameterizing to feed in the proper lambda binary and test assets to the dockerfile?

[jlizen]

(I like what you are trying to do here with the testing harness though, seems super handy... If the referenced action is in development / not yet public... Excited to see it!)

[darklight3it]

Sorry 😢 for this you miss some context here, and I had no time to write this in the appropriate channel. This stuff is agreed with @maxday.

That action is already used in the ruby repo and Maxime is actively working in making it public. The idea is to use the same harness test framework in all lambda runtimes.

I created a CR on that repo and this one is dependent on that to be working.

[darklight3it]

BTW I think the existing test-rie way of dealing with those tests will be removed in favor how the harness test standard that easily allow you to make assertions on the lambda response.

[FullyTyped]

There's already docker infra scripts written by Antonia.

[darklight3it]

I know but this is doing something different. It's integrating this stuff in the harness test GHA created by @maxday .

The idea is to port every RIC to use the same test framework.

Beside that this one also allows to use examples as task to be run inside the harness test framework.

[FullyTyped]

I don't think that is needed as we already do e2e tests on PR. I understand the need for uniformity, but I think other runtimes should move to this model, rather than introducing a language nobody wants to learn.

For rust in particular, we can inject a custom client into the runtime and get this without dealing with additional tooling, and we can also test the client by itself with just a custom server — this server simulator code is already available.

Ultimately, not my decision, but I think this is a waste of time and resources.

[maxday]

Thanks for your feedback @FullyTyped !
We believe there is a need for harness testing, as it could quickly detect some issues without having to deploy to a real Lambda ecosystem. It allows for a fast feedback loop during development.
We don't use a custom server, but the RIE, also maintained by AWS: https://github.com/aws/aws-lambda-runtime-interface-emulator

[maxday]

do we need this?

[darklight3it]

no, removing debug

[maxday]

I think you can now remove this as the action is now Dockerized, no need for python anymore

[darklight3it]

you're right, forgot it

[rarepolz]

nit: Looks like a duplicate checkbox + stray .on. at the end of the license line.

[darklight3it]

Sorry removing it from the PR.

[FullyTyped]

I wrote the server in this crate in other tests because it was far easier than using RIE and harness tests, and it allows you to evaluate the actual state of the runtime programmatically in a white-box approach.

There is a reason I did that way when I wrote those tests, Max.

~ Quinn

Let's maybe not assume ignorance next time we talk to contributors :)

[maxday]

I meant we don't use custom server in the GitHub action as we use the RIE sorry if that was not clear.

Also I'm definitely not assuming anything, I'm simply explaining why we are introducing the RIE and providing a link for those who aren't familiar with it and might find this PR later.