Skip to content

chore(skills): refresh stale Relay signatures #469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
exactlyallan wants to merge 3 commits into
base: release/0.6
Choose a base branch
from
Open

chore(skills): refresh stale Relay signatures #469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
54b0d54
chore(skills): refresh stale Relay signatures
exactlyallan Jul 17, 2026
71dd8ce
docs(skills): refresh Relay signing inputs
exactlyallan Jul 17, 2026
5c3a8b5
Attach NVSkills validation signatures
svc-nvskills-signing Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 8 additions & 23 deletions skills/nemo-relay-get-started/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,9 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `nemo-relay-get-started`
- Evaluation date: 2026-07-15
- Evaluation date: 2026-07-17
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 15 evaluation tasks
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

## Agents Used
Expand All @@ -32,33 +29,21 @@ Reported benchmark dimensions:

Underlying evaluation signals used in this run:

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.
- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

The benchmark dataset contained 15 evaluation tasks:

- Positive tasks: 14 tasks where the skill was expected to activate.
- Negative tasks: 1 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
The evaluation dataset was not available in this report payload.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 97% (+0%) | 97% (+10%) |
| Correctness | 8 | 89% (+61%) | 76% (+38%) |
| Discoverability | 8 | 82% (+50%) | 74% (+35%) |
| Effectiveness | 8 | 74% (+50%) | 65% (+32%) |
| Efficiency | 8 | 72% (+34%) | 68% (+29%) |
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |
Comment on lines +32 to +46

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not report an overall benchmark pass when evaluation data is unavailable.

This block now states that the dataset and Tier 3 signals are unavailable and every result is N/A, but the report still claims Overall verdict: PASS and recommends publication in Lines 13 and 64-66. Qualify the result as incomplete/unscored, or explicitly limit the pass to Tier 1 static validation so missing evaluation data cannot be interpreted as successful benchmark evidence.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-get-started/BENCHMARK.md` around lines 32 - 46, The
benchmark report’s overall PASS and publication recommendation must not imply
successful evaluation when dataset and Tier 3 results are unavailable. Update
the verdict and recommendation sections associated with the report to mark the
benchmark incomplete/unscored, or explicitly scope PASS to Tier 1 static
validation only, while preserving the documented N/A results.


Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

Expand Down
5 changes: 3 additions & 2 deletions skills/nemo-relay-get-started/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,9 @@ proof, ask what outcome matters next and recommend exactly one plugin:
- PII Redaction -> sanitization of sensitive observability payloads
- Model Pricing -> cost estimates for managed LLM responses

Use the [plugin overview](https://docs.nvidia.com/nemo/relay/dev/configure-plugins/about)
to select the next component. Preview its smallest configuration, obtain
Use the
[plugin overview](https://docs.nvidia.com/nemo/relay/dev/configure-plugins/about) to
select the next component. Preview its smallest configuration, obtain
confirmation, and verify its behavior before layering in another plugin.

Use another handoff only after the user accepts or declines plugin progression,
Expand Down
28 changes: 8 additions & 20 deletions skills/nemo-relay-get-started/skill-card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ NVIDIA <br>
### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and engineers who are new to NeMo Relay and want to quickly evaluate it by following the least-complex try-now path for their existing environment, whether through the CLI, a maintained framework integration, or direct language-level instrumentation. <br>
Developers and engineers new to NeMo Relay who want to try the framework, select the simplest applicable quick-start path, and verify initial observable value before production setup. <br>

### Deployment Geography for Use: <br>
Global <br>
Expand All @@ -35,7 +35,7 @@ Mitigation: Review and scan skill before deployment. <br>


## Skill Output: <br>
**Output Type(s):** [Shell commands, Configuration instructions] <br>
**Output Type(s):** [Shell commands, Configuration instructions, Analysis] <br>
**Output Format:** [Markdown with inline bash code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>
Expand All @@ -46,9 +46,6 @@ Mitigation: Review and scan skill before deployment. <br>



## Evaluation Tasks: <br>
Evaluated against 15 evaluation tasks (14 positive skill-activation, 1 negative) in the astra-sandbox environment via NVSkills-Eval external profile. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
Expand All @@ -57,28 +54,19 @@ Reported benchmark dimensions: <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>
- `token_efficiency`: Compares token usage with and without the skill. <br>



## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 97% (+0%) | 97% (+10%) |
| Correctness | 8 | 89% (+61%) | 76% (+38%) |
| Discoverability | 8 | 82% (+50%) | 74% (+35%) |
| Effectiveness | 8 | 74% (+50%) | 65% (+32%) |
| Efficiency | 8 | 72% (+34%) | 68% (+29%) |
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |

## Skill Version(s): <br>
0.6.0-alpha.20260715 (source: git tag) <br>
0.7.0-alpha.20260717 (source: git tag) <br>
Comment on lines 68 to +69

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a blank line after the heading.

markdownlint reports that ## Skill Version(s) is not surrounded by blank lines. Add an empty line before the version value.

Proposed fix 
 ## Skill Version(s): <br>
+
 0.7.0-alpha.20260717 (source: git tag) <br>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Skill Version(s): <br>
0.6.0-alpha.20260715 (source: git tag) <br>
0.7.0-alpha.20260717 (source: git tag) <br>
## Skill Version(s): <br>
0.7.0-alpha.20260717 (source: git tag) <br>
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 68-68: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-get-started/skill-card.md` around lines 68 - 69, Add a
blank line immediately after the “Skill Version(s)” heading in the skill card,
before the version value, so the heading is surrounded by blank lines and
satisfies markdownlint.

Source: Linters/SAST tools


## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
Expand Down
2 changes: 1 addition & 1 deletion skills/nemo-relay-get-started/skill.oms.sig
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1yZWxheS1nZXQtc3RhcnRlZCIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJmYzZmZDlmMDY5NWMyODhkNDZjZGJhYTkxYTdmMDcyZjZkMjNjNTU0OWUxNWI4MTA5NmVmZGYzNzc5YTExZDk3IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRodWIiLAogICAgICAgICIuZ2l0IiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIgogICAgICBdCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjM2OGY3NjFiYTI2MzE0MTQ1MmRiYmE1NGQ0OTI2NjdkZjJlYmY1MjgyOGEyZDNlMzZjY2JjMTg1ZDUxNTMwMzQiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImQ1ODcyZDNhYjRmYzkyZDVhYTgwNmY5ZTlmNzk4MTJlOGE5MWZiOGIyMWJmNDJjNTk2NzQzMGRiM2EyMGIwYjkiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYjZkZGIwZTRhOTYxOWM3Y2E3YmIzNTdiZjYxM2FjZmMyYWEzM2E2ZjIyMjNiYzhhMGUwZTQwMWE4YjIyNTFiMCIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjY1ODEwNWE0MjFjYjE4NjFlYzNmZjY0NGZhMGY0ZWJkYTU2MmZkNjY1YjIyYTRkMWQwMGFhZDQ5YmEyZWMzNGMiLAogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvYnVpbHQtaW4taW50ZWdyYXRpb25zLXRyeS1ub3cubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3MThiYzE3NjAwYjQ5NmRlNWY3MDk1NzVjOWEwZmRiZjlhNTAxN2U2NmZhNGRhNWE1MDg1M2EyNDA2YTA3NjYyIiwKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2NsaS10cnktbm93Lm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiNzdkMDM0ZTMwYzViOWVkNTg5ZGFiZDU4MTdlZDZiNzZhZGFlMWZmNjhlMDdmNWM5NTc0MjUzOGE0NmVkNWY5NSIsCiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9tYW51YWwtbGFuZ3VhZ2UtdHJ5LW5vdy5tZCIKICAgICAgfSwKICAgICAgewogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjEyNjA2ZWRmYjc0OTNlN2ZiYmMxMzU0NDIyZTUyNWVhYzQ5MWVmNmMwMGE3YjNiMGVkZGYxNjhkMTRiMTczZWYiLAogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQCRqwiQFAEH0BrI5oWgXzm2P0sYOYySOB4YBDgrrNDmU6VVgHSdhlSyTa1JV7n5p4cCMDsOyblhQ3ePEJeDvaBYsYuZPGwwsTGtUpaHcI4DHiDT+pQQOVMNtXeIgi3Ncos4Vg==","keyid":""}]}}
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtby1yZWxheS1nZXQtc3RhcnRlZCIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICIzOWM3OWQxNzU5NjBiNjk0NGVhMDZjYTdkNGQzN2NkNTg0MzU4NTM3M2QxMTM1YmY5ODYwMWQ1MmIzYmU1NWM3IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiLAogICAgICAgICIuZ2l0aWdub3JlIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRodWIiCiAgICAgIF0sCiAgICAgICJtZXRob2QiOiAiZmlsZXMiCiAgICB9LAogICAgInJlc291cmNlcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJiNTg4OGQyNDlhOTQxNWE5N2RjNDQ0MmE1YWM5YTBiZTdmYzIxMDdjOGNlYTIyYmVkN2U2Y2Q5ZmNjZTQ4ODhkIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiZGlnZXN0IjogImMwZGEzM2ZlZjUyNGE0OTMxM2VmYWE1YTZlZmIzYmZlMDM2ZThlYWZkMTUzNWNkYjRmZWE4NWZmNjlhYmI3OWUiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImRpZ2VzdCI6ICJiNmRkYjBlNGE5NjE5YzdjYTdiYjM1N2JmNjEzYWNmYzJhYTMzYTZmMjIyM2JjOGEwZTBlNDAxYThiMjI1MWIwIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvYnVpbHQtaW4taW50ZWdyYXRpb25zLXRyeS1ub3cubWQiLAogICAgICAgICJkaWdlc3QiOiAiNjU4MTA1YTQyMWNiMTg2MWVjM2ZmNjQ0ZmEwZjRlYmRhNTYyZmQ2NjViMjJhNGQxZDAwYWFkNDliYTJlYzM0YyIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL2NsaS10cnktbm93Lm1kIiwKICAgICAgICAiZGlnZXN0IjogIjg4MzUyYmM0MTc0NDJkZjNjMTdlNjU2NDQ5OWI0OTE2ZDkzYzVmNThiMWJkZDUyOTY0Mjk4OGE2MGNkMTBjNTMiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9tYW51YWwtbGFuZ3VhZ2UtdHJ5LW5vdy5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICI3N2QwMzRlMzBjNWI5ZWQ1ODlkYWJkNTgxN2VkNmI3NmFkYWUxZmY2OGUwN2Y1Yzk1NzQyNTM4YTQ2ZWQ1Zjk1IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNraWxsLWNhcmQubWQiLAogICAgICAgICJkaWdlc3QiOiAiYTY2YWNlZTgwOGQ2NDBjNWU1ZGZlYjFjZmM3NTQwZmUyZjYyMWYzMGEzMTZjNzI4YjllYTVkYmUzMTI3ZmFhYSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0KICAgIF0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMFAEd20JaJMcW/5WmPmNHqJZaZmyR+ECo3sSsi58Dd4DUa+ajA6Q/GZ48K3mkRXmugIxAKRnLbgqfAUEwFJsVsrbSMxL1yii9Ih/YFqwnbVr1Uqq79KDda9XRkhFJkQLRWzmOg==","keyid":""}]}}
31 changes: 8 additions & 23 deletions skills/nemo-relay-plugin-observability/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,9 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
## Evaluation Summary

- Skill: `nemo-relay-plugin-observability`
- Evaluation date: 2026-07-15
- Evaluation date: 2026-07-17
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 20 evaluation tasks
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

## Agents Used
Expand All @@ -32,33 +29,21 @@ Reported benchmark dimensions:

Underlying evaluation signals used in this run:

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.
- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

The benchmark dataset contained 20 evaluation tasks:

- Positive tasks: 16 tasks where the skill was expected to activate.
- Negative tasks: 4 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
The evaluation dataset was not available in this report payload.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 100% (+0%) | 100% (+0%) |
| Correctness | 8 | 98% (+53%) | 93% (+31%) |
| Discoverability | 8 | 99% (+67%) | 85% (+42%) |
| Effectiveness | 8 | 91% (+47%) | 87% (+30%) |
| Efficiency | 8 | 92% (+51%) | 82% (+35%) |
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |
Comment on lines +32 to +46

@coderabbitai coderabbitai Bot Jul 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not report an overall PASS without evaluation evidence.

This section explicitly says the dataset and Tier 3 signals were unavailable, yet the report retains Overall verdict: PASS and a publication recommendation. Scope PASS to the validations actually completed, or mark the evaluation as incomplete/not evaluated.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-plugin-observability/BENCHMARK.md` around lines 32 - 46,
Update the Overall verdict and publication recommendation in BENCHMARK.md to
reflect that the evaluation dataset and Tier 3 signals were unavailable. Do not
report an overall PASS; scope the verdict to completed validations or mark the
evaluation as incomplete/not evaluated.


Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

Expand Down
Loading