Skip to content

test: Add benchmarks for KMS signer#3205

Merged
julienrbrt merged 1 commit intomainfrom
alex/kms_bench
Mar 28, 2026
Merged

test: Add benchmarks for KMS signer#3205
julienrbrt merged 1 commit intomainfrom
alex/kms_bench

Conversation

@alpe
Copy link
Copy Markdown
Contributor

@alpe alpe commented Mar 27, 2026

GCP:

export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
go test  -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
BenchmarkKmsSignerSign
BenchmarkKmsSignerSign-12             48          73557340 ns/op           0.00 MB/s       17751 B/op        210 allocs/op
BenchmarkKmsSignerSign-12             68          50129806 ns/op           0.00 MB/s       17225 B/op        206 allocs/op
BenchmarkKmsSignerSign-12             84          44575795 ns/op           0.00 MB/s       17093 B/op        205 allocs/op
BenchmarkKmsSignerSign-12             74          44234950 ns/op           0.00 MB/s       17310 B/op        206 allocs/op
BenchmarkKmsSignerSign-12             86          42962638 ns/op           0.00 MB/s       17190 B/op        206 allocs/op
BenchmarkKmsSignerSign-12             93          42389046 ns/op           0.00 MB/s       17245 B/op        206 allocs/op
BenchmarkKmsSignerSign-12             91          42135518 ns/op           0.00 MB/s       17217 B/op        205 allocs/op
BenchmarkKmsSignerSign-12             84          42199055 ns/op           0.00 MB/s       17257 B/op        205 allocs/op
BenchmarkKmsSignerSign-12             76          45042175 ns/op           0.00 MB/s       17194 B/op        205 allocs/op
BenchmarkKmsSignerSign-12             79          44681758 ns/op           0.00 MB/s       17157 B/op        205 allocs/op
PASS
ok      github.com/evstack/ev-node/pkg/signer/gcp       38.185s

AWS:

export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
go test  -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
BenchmarkKmsSignerSign
BenchmarkKmsSignerSign-12             72          51418837 ns/op           0.00 MB/s       33217 B/op        383 allocs/op
BenchmarkKmsSignerSign-12             75          52704365 ns/op           0.00 MB/s       32102 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             69          50641107 ns/op           0.00 MB/s       33246 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             70          52125871 ns/op           0.00 MB/s       31977 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             69          51551584 ns/op           0.00 MB/s       32001 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             73          50943109 ns/op           0.00 MB/s       35017 B/op        383 allocs/op
BenchmarkKmsSignerSign-12             61          54113689 ns/op           0.00 MB/s       32654 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             69          56350564 ns/op           0.00 MB/s       33990 B/op        383 allocs/op
BenchmarkKmsSignerSign-12             63          52215831 ns/op           0.00 MB/s       33327 B/op        382 allocs/op
BenchmarkKmsSignerSign-12             75          52353017 ns/op           0.00 MB/s       34353 B/op        382 allocs/op
PASS
ok      github.com/evstack/ev-node/pkg/signer/aws       42.782s

Summary by CodeRabbit

  • Tests
    • Added performance benchmarks for AWS KMS signing operations.
    • Added performance benchmarks for GCP KMS signing operations.

@github-actions
Copy link
Copy Markdown
Contributor

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed⏩ skipped✅ passed✅ passedMar 27, 2026, 11:21 AM

@claude
Copy link
Copy Markdown
Contributor

claude bot commented Mar 27, 2026

Claude finished @alpe's task in 1m 15s —— View job


Code Review

  • Gather context and read changed files
  • Review AWS benchmark test
  • Review GCP benchmark test
  • Post review feedback

Summary

The benchmarks are well-structured and follow good Go benchmarking practices. A few issues to address:


Issues

1. Copy-pasted comments point to the wrong provider (both files)

pkg/signer/aws/signer_benchmark_test.go lines 16–18 show GCP env vars and the ./pkg/signer/gcp path:

// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/...
// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
// go test  -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp

But the file is in package aws. These comments were copied from the GCP file and not updated.

Similarly, pkg/signer/gcp/signer_benchmark_test.go lines 16–18 show AWS env vars and ./pkg/signer/aws — the inverse problem.

Fix this →

2. GCP timeout is 1s vs AWS's 10s (pkg/signer/gcp/signer_benchmark_test.go:29)

signer, err := NewKmsSigner(b.Context(), keyName, &Options{
    Timeout: 1 * time.Second,  // very tight
    ...
})

The benchmark results show GCP latency up to ~73ms on a cold start. Under load or with any network hiccup, 1s could cause spurious failures. The AWS benchmark uses 10 * time.Second which is more generous. Consider aligning them or bumping GCP to at least 5 * time.Second.


Nits

  • Duplicated benchmarkHeaderData: The function body is identical in both packages. Since they're separate packages, sharing isn't straightforward without a new test helper package — acceptable duplication at this scale.

  • b.SetBytes with KMS: Calling b.SetBytes reports throughput as MB/s, but for KMS the bottleneck is the network round trip, not data transfer. The 0.00 MB/s in your benchmark output reflects this. It's not wrong, but slightly misleading.


What's good

  • Uses the idiomatic Go 1.24 b.Loop() API
  • b.ReportAllocs() + b.SetBytes() + b.ResetTimer() — all in the right places
  • Skips cleanly when env vars aren't set (correct for e2e-style benchmarks)
  • Realistic proto-marshalled header payload as input

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.42%. Comparing base (162cda6) to head (26180f1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3205   +/-   ##
=======================================
  Coverage   61.42%   61.42%           
=======================================
  Files         120      120           
  Lines       12449    12449           
=======================================
  Hits         7647     7647           
  Misses       3942     3942           
  Partials      860      860           
Flag Coverage Δ
combined 61.42% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 27, 2026

📝 Walkthrough

Walkthrough

Two new benchmark test files added for AWS and GCP KMS signers. Each benchmark reads cloud provider configuration from environment variables, creates a KMS signer with specified timeouts, marshals protobuf header data, and measures signing operation performance across multiple iterations.

Changes

Cohort / File(s) Summary
AWS KMS Signer Benchmark
pkg/signer/aws/signer_benchmark_test.go
Adds BenchmarkKmsSignerSign benchmark that reads AWS KMS credentials from environment variables, creates a signer with 10s timeout and zero max retries, generates deterministic protobuf-encoded header data, and benchmarks repeated signing operations with allocation tracking and byte reporting.
GCP KMS Signer Benchmark
pkg/signer/gcp/signer_benchmark_test.go
Adds BenchmarkKmsSignerSign benchmark that reads GCP KMS configuration from environment variables, creates a signer with 1s timeout and zero max retries, constructs protobuf header data with fixed field values, and benchmarks signing performance with allocation and byte metrics.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

Suggested reviewers

  • julienrbrt
  • tac0turtle

Poem

🐰 Hop, hop, hooray! 📊
Benchmarks now measure how fast we can sign,
With AWS and GCP both working in time,
Milliseconds counted, allocations tracked,
Performance profiling—the cryptographic path! 🔐

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description lacks the required 'Overview' section from the template and provides only benchmark commands and output without context, background, or rationale for the changes. Add an 'Overview' section explaining the purpose of the benchmarks, what they measure, and why they were added. Include context on whether this closes any issue.
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title 'test: Add benchmarks for KMS signer' accurately describes the main change—adding benchmark tests for KMS signer implementations in both GCP and AWS packages.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch alex/kms_bench

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
pkg/signer/aws/signer_benchmark_test.go (1)

60-83: Consider extracting shared test helpers.

The benchmarkHeaderData function is duplicated between the AWS and GCP benchmark files. This is acceptable for test isolation, but if more signer implementations are added, consider extracting this to a shared pkg/signer/testutil package.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/signer/aws/signer_benchmark_test.go` around lines 60 - 83, Extract the
duplicated benchmarkHeaderData helper into a shared test utility package (e.g.,
package testutil under pkg/signer/testutil) and update both AWS and GCP
benchmark files to call the shared helper; specifically, create a function
(preferably func BenchmarkHeaderData(tb testing.TB) []byte) that contains the
proto.Header construction and proto.Marshal logic currently in
benchmarkHeaderData, import pkg/signer/testutil in signer_benchmark_test.go (and
the GCP counterpart) and replace calls to the local benchmarkHeaderData(b) with
testutil.BenchmarkHeaderData(b), and ensure necessary imports (testing, proto,
require) are present in the new testutil package.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/signer/aws/signer_benchmark_test.go`:
- Around line 15-18: The comment block in signer_benchmark_test.go incorrectly
references GCP (env vars and package path) for the BenchmarkKmsSignerSign
benchmark; update those comments to show the AWS equivalents and package path
(e.g., replace GCP env var examples and "pkg/signer/gcp" with the AWS
environment variables used by the test and "pkg/signer/aws") so the doc comment
matches the AWS test implementation (refer to BenchmarkKmsSignerSign in this
file to verify the correct AWS variable names).

In `@pkg/signer/gcp/signer_benchmark_test.go`:
- Around line 15-18: The benchmark comment block in signer_benchmark_test.go
wrongly references AWS (EVNODE_E2E_AWS_KMS_KEY_ID, EVNODE_E2E_AWS_KMS_REGION and
./pkg/signer/aws); update that comment to GCP-specific environment variables and
path used by the GCP benchmark (e.g., replace AWS env var names with the GCP KMS
equivalents your test expects and change the package path to ./pkg/signer/gcp)
so the comment matches the actual BenchmarkKmsSignerSign implementation and
required GCP setup.

---

Nitpick comments:
In `@pkg/signer/aws/signer_benchmark_test.go`:
- Around line 60-83: Extract the duplicated benchmarkHeaderData helper into a
shared test utility package (e.g., package testutil under pkg/signer/testutil)
and update both AWS and GCP benchmark files to call the shared helper;
specifically, create a function (preferably func BenchmarkHeaderData(tb
testing.TB) []byte) that contains the proto.Header construction and
proto.Marshal logic currently in benchmarkHeaderData, import pkg/signer/testutil
in signer_benchmark_test.go (and the GCP counterpart) and replace calls to the
local benchmarkHeaderData(b) with testutil.BenchmarkHeaderData(b), and ensure
necessary imports (testing, proto, require) are present in the new testutil
package.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: c73fda29-18db-4c27-9f3b-41e7d10aadb6

📥 Commits

Reviewing files that changed from the base of the PR and between 162cda6 and 26180f1.

📒 Files selected for processing (2)
  • pkg/signer/aws/signer_benchmark_test.go
  • pkg/signer/gcp/signer_benchmark_test.go

Comment on lines +15 to +18
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Copy-paste error: comments describe GCP instead of AWS.

The comment block references GCP environment variables and the GCP package path, but this is the AWS benchmark file. The actual code correctly uses AWS variables.

📝 Suggested fix
 // BenchmarkKmsSignerSign test kms round trip
-// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
-// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
-// go test  -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
+// export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
+// export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
+// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
// export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/signer/aws/signer_benchmark_test.go` around lines 15 - 18, The comment
block in signer_benchmark_test.go incorrectly references GCP (env vars and
package path) for the BenchmarkKmsSignerSign benchmark; update those comments to
show the AWS equivalents and package path (e.g., replace GCP env var examples
and "pkg/signer/gcp" with the AWS environment variables used by the test and
"pkg/signer/aws") so the doc comment matches the AWS test implementation (refer
to BenchmarkKmsSignerSign in this file to verify the correct AWS variable
names).

Comment on lines +15 to +18
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
// export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Copy-paste error: comments describe AWS instead of GCP.

The comment block references AWS environment variables and the AWS package path, but this is the GCP benchmark file. The actual code correctly uses GCP variables.

📝 Suggested fix
 // BenchmarkKmsSignerSign test kms round trip
-// export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
-// export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
-// go test  -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
+// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
+// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
+// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_AWS_KMS_KEY_ID="arn:aws:kms:...."
// export EVNODE_E2E_AWS_KMS_REGION=eu-west-1
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/aws
// BenchmarkKmsSignerSign test kms round trip
// export EVNODE_E2E_GCP_KMS_KEY_NAME=projects/<project-id>/locations/<region>/keyRings/<keyring-name>/cryptoKeys/<key-name>/cryptoKeyVersions/1
// export EVNODE_E2E_GCP_KMS_CREDENTIALS_FILE=...
// go test -v -bench=BenchmarkKmsSignerSign -benchtime=3s -count=10 -run='^$' ./pkg/signer/gcp
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/signer/gcp/signer_benchmark_test.go` around lines 15 - 18, The benchmark
comment block in signer_benchmark_test.go wrongly references AWS
(EVNODE_E2E_AWS_KMS_KEY_ID, EVNODE_E2E_AWS_KMS_REGION and ./pkg/signer/aws);
update that comment to GCP-specific environment variables and path used by the
GCP benchmark (e.g., replace AWS env var names with the GCP KMS equivalents your
test expects and change the package path to ./pkg/signer/gcp) so the comment
matches the actual BenchmarkKmsSignerSign implementation and required GCP setup.

Copy link
Copy Markdown
Member

@julienrbrt julienrbrt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK because i don't have such keys.

@julienrbrt julienrbrt added this pull request to the merge queue Mar 28, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 28, 2026
@julienrbrt julienrbrt merged commit 1c2309a into main Mar 28, 2026
37 checks passed
@julienrbrt julienrbrt deleted the alex/kms_bench branch March 28, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants