Skip to content

Add protocol support for AWS JSON-RPC #645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jonathan343 wants to merge 16 commits into
base: develop
Choose a base branch
from
Open

Add protocol support for AWS JSON-RPC #645

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5c1d94b
Add AWS JSON 1.0 and 1.1 traits
jonathan343 Feb 17, 2026
d6704da
smithy-json: support quoted non-finite numeric serde
jonathan343 Feb 26, 2026
2eb2023
codegen: improve generated protocol test assertions
jonathan343 Feb 26, 2026
ad43c77
smithy-http: preserve host prefixes when applying endpoints
jonathan343 Feb 26, 2026
0243826
smithy-aws-core: add awsJson client protocol support
jonathan343 Feb 26, 2026
5a4659c
codegen: add awsJson protocol generators and tests
jonathan343 Feb 26, 2026
7d332fb
codegen: use protocol-specific protocol test module names
jonathan343 Mar 3, 2026
3d2f6c5
Add docstring to _parse_http_protocol_values for clarity
jonathan343 Mar 4, 2026
e051943
smithy-json: combine equivalent number deserializer branches
jonathan343 Mar 4, 2026
3512866
smithy-aws-core: simplify awsJson error identification
jonathan343 Mar 7, 2026
76ac447
smithy-http: stop preserving host prefixes on endpoints
jonathan343 Mar 7, 2026
05c957e
codegen: skip awsJson endpoint trait protocol tests
jonathan343 Mar 7, 2026
4b07405
Handle modeled error namespace mismatches during lookup
jonathan343 Mar 11, 2026
306961c
Skip NaN protocol compliance cases in generated tests
jonathan343 Mar 12, 2026
7614b45
remove eventstream mix in until we get proper support
jonathan343 Mar 12, 2026
679c259
Address feedback
jonathan343 Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,12 @@ build-java: ## Builds the Java code generation packages.
cd codegen && ./gradlew clean build


test-protocols: ## Generates and runs the restJson1 protocol tests.
cd codegen && ./gradlew :protocol-test:build
uv pip install codegen/protocol-test/build/smithyprojections/protocol-test/rest-json-1/python-client-codegen
uv run pytest codegen/protocol-test/build/smithyprojections/protocol-test/rest-json-1/python-client-codegen
test-protocols: ## Generates and runs protocol tests for all supported protocols.
cd codegen && ./gradlew :protocol-test:clean :protocol-test:build
@set -e; for projection_dir in codegen/protocol-test/build/smithyprojections/protocol-test/*/python-client-codegen; do \
uv pip install "$$projection_dir"; \
uv run pytest "$$projection_dir"; \
done


lint-py: ## Runs linters and formatters on the python packages.
Expand Down
1 change: 1 addition & 0 deletions codegen/aws/core/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ extra["moduleName"] = "software.amazon.smithy.python.aws.codegen"
dependencies {
implementation(project(":core"))
implementation(libs.smithy.aws.traits)
implementation(libs.smithy.protocol.test.traits)
}
76 changes: 76 additions & 0 deletions ...e/src/main/java/software/amazon/smithy/python/aws/codegen/AwsJson10ProtocolGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package software.amazon.smithy.python.aws.codegen;

import java.util.Set;
import software.amazon.smithy.aws.traits.protocols.AwsJson1_0Trait;
import software.amazon.smithy.model.node.ArrayNode;
import software.amazon.smithy.model.node.ObjectNode;
import software.amazon.smithy.model.shapes.ShapeId;
import software.amazon.smithy.python.codegen.ApplicationProtocol;
import software.amazon.smithy.python.codegen.GenerationContext;
import software.amazon.smithy.python.codegen.HttpProtocolTestGenerator;
import software.amazon.smithy.python.codegen.SymbolProperties;
import software.amazon.smithy.python.codegen.generators.ProtocolGenerator;
import software.amazon.smithy.python.codegen.writer.PythonWriter;
import software.amazon.smithy.utils.SmithyInternalApi;

@SmithyInternalApi
public final class AwsJson10ProtocolGenerator implements ProtocolGenerator {
private static final Set<String> TESTS_TO_SKIP = Set.of(
// TODO: support the request compression trait
// https://smithy.io/2.0/spec/behavior-traits.html#smithy-api-requestcompression-trait
"SDKAppliedContentEncoding_awsJson1_0",
"SDKAppendsGzipAndIgnoresHttpProvidedEncoding_awsJson1_0",

// TODO: Fix for both REST-JSON and JSON-RPC
"AwsJson10ClientPopulatesDefaultValuesInInput",
"AwsJson10ClientSkipsTopLevelDefaultValuesInInput",
"AwsJson10ClientUsesExplicitlyProvidedMemberValuesOverDefaults",
"AwsJson10ClientPopulatesDefaultsValuesWhenMissingInResponse",
"AwsJson10ClientIgnoresNonTopLevelDefaultsOnMembersWithClientOptional",
"AwsJson10ClientIgnoresDefaultValuesIfMemberValuesArePresentInResponse",

// TODO: support client error-correction behavior when the server
// omits required values in modeled error responses.
"AwsJson10ClientErrorCorrectsWhenServerFailsToSerializeRequiredValues",
"AwsJson10ClientErrorCorrectsWithDefaultValuesWhenServerFailsToSerializeRequiredValues");
Comment on lines +45 to +48
Copy link
Copy Markdown
Contributor

@JordonPhillips JordonPhillips Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't you say you fixed this?

Copy link
Copy Markdown
Contributor Author

@jonathan343 jonathan343 Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, maybe you're thinking of the error namespace mismatch fallback behavior I added?


@Override
public ShapeId getProtocol() {
return AwsJson1_0Trait.ID;
}

@Override
public ApplicationProtocol getApplicationProtocol(GenerationContext context) {
var service = context.settings().service(context.model());
var trait = service.expectTrait(AwsJson1_0Trait.class);
var config = ObjectNode.builder()
.withMember("http", ArrayNode.fromStrings(trait.getHttp()))
.withMember("eventStreamHttp", ArrayNode.fromStrings(trait.getEventStreamHttp()))
.build();
return ApplicationProtocol.createDefaultHttpApplicationProtocol(config);
}

@Override
public void initializeProtocol(GenerationContext context, PythonWriter writer) {
writer.addDependency(AwsPythonDependency.SMITHY_AWS_CORE.withOptionalDependencies("json"));
writer.addImport("smithy_aws_core.aio.protocols", "AwsJson10ClientProtocol");
var serviceSymbol = context.symbolProvider().toSymbol(context.settings().service(context.model()));
var serviceSchema = serviceSymbol.expectProperty(SymbolProperties.SCHEMA);
writer.write("AwsJson10ClientProtocol($T)", serviceSchema);
}

@Override
public void generateProtocolTests(GenerationContext context) {
context.writerDelegator().useFileWriter("./tests/test_protocol.py", "tests.test_protocol", writer -> {
Comment thread
jonathan343 marked this conversation as resolved.
Outdated
new HttpProtocolTestGenerator(
context,
getProtocol(),
writer,
(shape, testCase) -> TESTS_TO_SKIP.contains(testCase.getId())).run();
});
}
}
63 changes: 63 additions & 0 deletions ...e/src/main/java/software/amazon/smithy/python/aws/codegen/AwsJson11ProtocolGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package software.amazon.smithy.python.aws.codegen;

import java.util.Set;
import software.amazon.smithy.aws.traits.protocols.AwsJson1_1Trait;
import software.amazon.smithy.model.node.ArrayNode;
import software.amazon.smithy.model.node.ObjectNode;
import software.amazon.smithy.model.shapes.ShapeId;
import software.amazon.smithy.python.codegen.ApplicationProtocol;
import software.amazon.smithy.python.codegen.GenerationContext;
import software.amazon.smithy.python.codegen.HttpProtocolTestGenerator;
import software.amazon.smithy.python.codegen.SymbolProperties;
import software.amazon.smithy.python.codegen.generators.ProtocolGenerator;
import software.amazon.smithy.python.codegen.writer.PythonWriter;
import software.amazon.smithy.utils.SmithyInternalApi;

@SmithyInternalApi
public final class AwsJson11ProtocolGenerator implements ProtocolGenerator {
private static final Set<String> TESTS_TO_SKIP = Set.of(
// TODO: support the request compression trait
// https://smithy.io/2.0/spec/behavior-traits.html#smithy-api-requestcompression-trait
"SDKAppliedContentEncoding_awsJson1_1",
"SDKAppendsGzipAndIgnoresHttpProvidedEncoding_awsJson1_1");

@Override
public ShapeId getProtocol() {
return AwsJson1_1Trait.ID;
}

@Override
public ApplicationProtocol getApplicationProtocol(GenerationContext context) {
var service = context.settings().service(context.model());
var trait = service.expectTrait(AwsJson1_1Trait.class);
var config = ObjectNode.builder()
.withMember("http", ArrayNode.fromStrings(trait.getHttp()))
.withMember("eventStreamHttp", ArrayNode.fromStrings(trait.getEventStreamHttp()))
.build();
return ApplicationProtocol.createDefaultHttpApplicationProtocol(config);
}

@Override
public void initializeProtocol(GenerationContext context, PythonWriter writer) {
writer.addDependency(AwsPythonDependency.SMITHY_AWS_CORE.withOptionalDependencies("json"));
writer.addImport("smithy_aws_core.aio.protocols", "AwsJson11ClientProtocol");
var serviceSymbol = context.symbolProvider().toSymbol(context.settings().service(context.model()));
var serviceSchema = serviceSymbol.expectProperty(SymbolProperties.SCHEMA);
writer.write("AwsJson11ClientProtocol($T)", serviceSchema);
}

@Override
public void generateProtocolTests(GenerationContext context) {
context.writerDelegator().useFileWriter("./tests/test_protocol.py", "tests.test_protocol", writer -> {
Comment thread
jonathan343 marked this conversation as resolved.
Outdated
new HttpProtocolTestGenerator(
context,
getProtocol(),
writer,
(shape, testCase) -> TESTS_TO_SKIP.contains(testCase.getId())).run();
});
}
}
2 changes: 1 addition & 1 deletion ...core/src/main/java/software/amazon/smithy/python/aws/codegen/AwsProtocolsIntegration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,6 @@
public class AwsProtocolsIntegration implements PythonIntegration {
@Override
public List<ProtocolGenerator> getProtocolGenerators() {
return List.of();
return List.of(new AwsJson10ProtocolGenerator(), new AwsJson11ProtocolGenerator());
}
}
51 changes: 46 additions & 5 deletions ...n/core/src/main/java/software/amazon/smithy/python/codegen/HttpProtocolTestGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
import java.util.logging.Logger;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import software.amazon.smithy.aws.traits.auth.SigV4Trait;
import software.amazon.smithy.codegen.core.CodegenException;
import software.amazon.smithy.codegen.core.Symbol;
import software.amazon.smithy.model.Model;
Expand Down Expand Up @@ -188,12 +189,14 @@ private void generateRequestTest(OperationShape operation, HttpRequestTestCase t
endpoint_uri="https://$L/$L",
transport = $T(),
retry_strategy=SimpleRetryStrategy(max_attempts=1),
${C|}
)
""",
CodegenUtils.getConfigSymbol(context.settings()),
host,
path,
REQUEST_TEST_ASYNC_HTTP_CLIENT_SYMBOL);
REQUEST_TEST_ASYNC_HTTP_CLIENT_SYMBOL,
(Runnable) this::writeSigV4TestConfig);
}));

// Generate the input using the expected shape and params
Expand Down Expand Up @@ -437,13 +440,15 @@ private void generateResponseTest(OperationShape operation, HttpResponseTestCase
headers=$J,
body=b$S,
),
${C|}
)
""",
CodegenUtils.getConfigSymbol(context.settings()),
RESPONSE_TEST_ASYNC_HTTP_CLIENT_SYMBOL,
testCase.getCode(),
CodegenUtils.toTuples(testCase.getHeaders()),
testCase.getBody().filter(body -> !body.isEmpty()).orElse(""));
testCase.getBody().filter(body -> !body.isEmpty()).orElse(""),
(Runnable) this::writeSigV4TestConfig);
}));
// Create an empty input object to pass
var inputShape = model.expectShape(operation.getInputShape(), StructureShape.class);
Expand Down Expand Up @@ -490,13 +495,15 @@ private void generateErrorResponseTest(
headers=$J,
body=b$S,
),
${C|}
)
""",
CodegenUtils.getConfigSymbol(context.settings()),
RESPONSE_TEST_ASYNC_HTTP_CLIENT_SYMBOL,
testCase.getCode(),
CodegenUtils.toTuples(testCase.getHeaders()),
testCase.getBody().orElse(""));
testCase.getBody().orElse(""),
(Runnable) this::writeSigV4TestConfig);
}));
// Create an empty input object to pass
var inputShape = model.expectShape(operation.getInputShape(), StructureShape.class);
Expand Down Expand Up @@ -531,7 +538,7 @@ private void assertResponseEqual(HttpMessageTestCase testCase, Shape operationOr
.findAny();

if (streamBinding.isEmpty()) {
writer.write("assert actual == expected\n");
writer.write("_assert_modeled_value(actual, expected)\n");
return;
}

Expand All @@ -556,7 +563,7 @@ assert isinstance(actual.$1L, AsyncByteStream)
compareMediaBlob(testCase, writer);
continue;
}
writer.write("assert actual.$1L == expected.$1L\n", memberName);
writer.write("_assert_modeled_value(actual.$1L, expected.$1L)\n", memberName);
}
}

Expand Down Expand Up @@ -607,10 +614,26 @@ private void writeClientBlock(
});
}

private void writeSigV4TestConfig() {
if (!service.hasTrait(SigV4Trait.class)) {
return;
}
writer.addImport("smithy_aws_core.identity", "StaticCredentialsResolver");
writer.write("""
region="us-east-1",
aws_access_key_id="test-access-key-id",
aws_secret_access_key="test-secret-access-key",
aws_credentials_identity_resolver=StaticCredentialsResolver(),
""");
}

private void writeUtilStubs(Symbol serviceSymbol) {
LOGGER.fine(String.format("Writing utility stubs for %s : %s", serviceSymbol.getName(), protocol.getName()));
writer.addDependency(SmithyPythonDependency.SMITHY_CORE);
writer.addDependency(SmithyPythonDependency.SMITHY_HTTP);
writer.addStdlibImport("dataclasses", "fields");
writer.addStdlibImport("dataclasses", "is_dataclass");
writer.addStdlibImport("math", "isnan");
writer.addImports("smithy_http.interfaces",
Set.of(
"HTTPRequestConfiguration",
Expand All @@ -621,6 +644,24 @@ private void writeUtilStubs(Symbol serviceSymbol) {
writer.addImport("smithy_core.aio.utils", "async_list");

writer.write("""
def _assert_modeled_value(actual: object, expected: object) -> None:
Copy link
Copy Markdown
Contributor

@JordonPhillips JordonPhillips Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of this. When you do a plain assert with two dataclasses, pytest will give you a nice result showing you the diff if they're not equal. This will fail at the first non-equal value. Also, since it only recurses on dataclasses it'll never find nan in a list or map.

In the spirit of moving things out of Java, it might be best to have a test-utils package that implements an assert that builds up an error. It could go in test dependencies so normal installs don't catch it.

Copy link
Copy Markdown
Contributor Author

@jonathan343 jonathan343 Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, this is more complicated that I thought. I'm gonna back track from this here so the scope is on JSON-RPC and I don't add complexity to this PR.

I played around with this approach for replacing NaN's with a custom object that won't fail on the ==. But I'm not a fan of having to rebuild the object just for this.

class _ModeledNaN:
    def __repr__(self) -> str:
        return "float('nan')"


class _NormalizedDocument:
    def __init__(self, value: object) -> None:
        self.value = value

    def __eq__(self, other: object) -> bool:
        return isinstance(other, _NormalizedDocument) and self.value == other.value

    def __repr__(self) -> str:
        return f"Document(value={self.value!r})"


_MODELED_NAN = _ModeledNaN()


def _normalize_modeled_value(value: Any) -> Any:
    if isinstance(value, float) and isnan(value):
        return _MODELED_NAN

    if isinstance(value, Document):
        return _NormalizedDocument(_normalize_modeled_value(value.as_value()))

    if is_dataclass(value) and not isinstance(value, type):
        return cast(Any, type(value))(
            **{
                field.name: _normalize_modeled_value(getattr(value, field.name))
                for field in fields(value)
                if field.init
            },
        )

    if isinstance(value, list):
        return [_normalize_modeled_value(member) for member in cast(list[Any], value)]

    if isinstance(value, tuple):
        return tuple(
            _normalize_modeled_value(member) for member in cast(tuple[Any, ...], value)
        )

    if isinstance(value, dict):
        return {
            key: _normalize_modeled_value(member)
            for key, member in cast(dict[Any, Any], value).items()
        }

    return value


def _assert_modeled_value(actual: Any, expected: Any) -> None:
    assert _normalize_modeled_value(actual) == _normalize_modeled_value(expected)

I'm gonna revert these changes and ignore the nan tests again. I'll create an issue for us to track the work for getting a proper solution.

Copy link
Copy Markdown
Contributor

@JordonPhillips JordonPhillips Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's a bit complex. What if you just to a recursive equalty check if it's a known nan test? I'm okay with the rest result being less good for those tests specifically. I think it should be pretty easy to traverse the document during codegen to see if there's a nan value.

if isinstance(expected, float) and isnan(expected):
assert isinstance(actual, float)
assert isnan(actual)
return

if is_dataclass(expected):
assert is_dataclass(actual)
for field in fields(expected):
_assert_modeled_value(
getattr(actual, field.name),
getattr(expected, field.name),
)
return

assert actual == expected


class $1L($2T):
""\"A test error that subclasses the service-error for protocol tests.""\"

Expand Down
6 changes: 0 additions & 6 deletions ...in/java/software/amazon/smithy/python/codegen/integrations/RestJsonProtocolGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,6 @@
public class RestJsonProtocolGenerator implements ProtocolGenerator {

private static final Set<String> TESTS_TO_SKIP = Set.of(
// These two tests essentially try to assert nan == nan,
// which is never true. We should update the generator to
// make specific assertions for these.
"RestJsonSupportsNaNFloatHeaderOutputs",
"RestJsonSupportsNaNFloatInputs",

// This requires support of idempotency autofill
"RestJsonQueryIdempotencyTokenAutoFill",

Expand Down
1 change: 1 addition & 0 deletions codegen/protocol-test/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,6 @@ repositories {

dependencies {
implementation(project(":core"))
implementation(project(":aws:core"))
implementation(libs.smithy.aws.protocol.tests)
}
44 changes: 44 additions & 0 deletions codegen/protocol-test/smithy-build.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,50 @@
"moduleVersion": "0.0.1"
}
}
},
"aws-json-1-0": {
"transforms": [
{
"name": "includeServices",
"args": {
"services": [
"aws.protocoltests.json10#JsonRpc10"
]
}
},
{
"name": "removeUnusedShapes"
}
],
"plugins": {
"python-client-codegen": {
"service": "aws.protocoltests.json10#JsonRpc10",
"module": "awsjson10",
"moduleVersion": "0.0.1"
}
}
},
"aws-json-1-1": {
"transforms": [
{
"name": "includeServices",
"args": {
"services": [
"aws.protocoltests.json#JsonProtocol"
]
}
},
{
"name": "removeUnusedShapes"
}
],
"plugins": {
"python-client-codegen": {
"service": "aws.protocoltests.json#JsonProtocol",
"module": "awsjson11",
"moduleVersion": "0.0.1"
}
}
}
}
}
Loading
Loading