Skip to content

chore(skills): refresh stale Relay signatures#469

Open
exactlyallan wants to merge 3 commits into
release/0.6from
fix/refresh-relay-skill-signatures
Open

chore(skills): refresh stale Relay signatures#469
exactlyallan wants to merge 3 commits into
release/0.6from
fix/refresh-relay-skill-signatures

Conversation

@exactlyallan

@exactlyallan exactlyallan commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Overview

Refresh two stale NeMo Relay skill signatures that are currently blocking publication through the NVIDIA skills catalog sync.

  • I confirm this contribution is my own work, or I have the right to submit it under this project's license.
  • I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

  • Remove the stale skill.oms.sig bundle from nemo-relay-get-started.
  • Remove the stale skill.oms.sig bundle from nemo-relay-plugin-observability.
  • Reflow existing Markdown in each affected SKILL.md without changing rendered
    instructions, ensuring NVCARPS includes both skills in its signing input set.
  • Leave all references, evaluations, cards, and benchmarks unchanged.
  • Trigger /nvskills-ci so the signing bot regenerates content-aligned bundles and validation artifacts.

Do not merge until the Attach NVSkills validation signatures bot commit lands and both affected skills pass content-integrity verification.

Where should the reviewer start?

Start with the two deleted signature bundles and the rendered-content-neutral Markdown reflows. NVSkills CI should restore each bundle with digests matching the current skill content.

Validation before opening:

  • Reproduced the catalog integrity failure against current NVIDIA/NeMo-Relay/main.
  • Confirmed exactly one mismatched signed file in nemo-relay-get-started.
  • Confirmed exactly five mismatched signed files in nemo-relay-plugin-observability.
  • Confirmed the only non-generated changes are rendered-content-neutral Markdown reflows in the two affected SKILL.md files.
  • git diff --check

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

Summary by CodeRabbit

  • Chores / Security

    • Refreshed the signature verification bundles for the Nemo Relay getting-started and observability skill packages to keep integrity checks current.
  • Documentation

    • Updated the getting-started and observability guides and skill cards with formatting-only tweaks, revised “use case” wording, and adjusted output/evaluation sections.
    • Refreshed benchmark metadata and evaluation reporting to reflect unavailable dataset details (results now shown as N/A) and updated version indicators.

@exactlyallan
exactlyallan requested a review from a team as a code owner July 17, 2026 16:44
@github-actions github-actions Bot added size:XS PR is extra small Maintenance CI or Build or general repository maintenance labels Jul 17, 2026
@exactlyallan

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Refreshes get-started and observability skill metadata, benchmark reports, documentation formatting, and signed DSSE bundle payloads. Evaluation results now report unavailable datasets or signals as N/A; no public declarations change.

Changes

Skill refresh

Layer / File(s) Summary
Get-started skill metadata and evaluation reporting
skills/nemo-relay-get-started/skill-card.md, skills/nemo-relay-get-started/BENCHMARK.md
Updates use-case and output metadata, removes evaluation-task details, records unavailable evaluation data as N/A, and advances the skill version.
Observability skill metadata and evaluation reporting
skills/nemo-relay-plugin-observability/skill-card.md, skills/nemo-relay-plugin-observability/BENCHMARK.md
Updates use-case references and output metadata, replaces detailed evaluation results with N/A, and advances the skill version.
Documentation formatting
skills/nemo-relay-get-started/SKILL.md, skills/nemo-relay-plugin-observability/SKILL.md
Reflows existing Markdown passages without changing their content or behavior.
Signed bundle refresh
skills/nemo-relay-get-started/skill.oms.sig, skills/nemo-relay-plugin-observability/skill.oms.sig
Updates DSSE payload digests, predicate data, and signatures while retaining the bundle structure and verification material.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title follows Conventional Commits and accurately summarizes the stale Relay signature refresh.
Description check ✅ Passed The description includes all required template sections and gives concrete details, reviewer guidance, and a related issue reference.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/refresh-relay-skill-signatures

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot added the DO NOT MERGE PR should not be merged; see PR for details label Jul 17, 2026
@github-actions github-actions Bot added size:S PR is small and removed size:XS PR is extra small labels Jul 17, 2026
@exactlyallan

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@coderabbitai coderabbitai Bot removed the DO NOT MERGE PR should not be merged; see PR for details label Jul 17, 2026
@exactlyallan exactlyallan added this to the 0.6 milestone Jul 17, 2026
@exactlyallan exactlyallan self-assigned this Jul 17, 2026
@willkill07

Copy link
Copy Markdown
Member

@exactlyallan could this target release/0.6 ?

@exactlyallan

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added size:M PR is medium and removed size:S PR is small labels Jul 17, 2026
@exactlyallan
exactlyallan changed the base branch from main to release/0.6 July 17, 2026 20:53
@exactlyallan
exactlyallan requested a review from lvojtku as a code owner July 17, 2026 20:53
exactlyallan and others added 3 commits July 17, 2026 13:56
Remove the two signature bundles that no longer match their skill content so NVSkills CI can regenerate content-aligned signatures without changing the underlying skills.

Signed-off-by: Allan Enemark <aenemark@nvidia.com>
Reflow existing Markdown without changing rendered instructions so NVSkills CI detects both affected skills and can regenerate their stale signature bundles.

Signed-off-by: Allan Enemark <aenemark@nvidia.com>
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@exactlyallan
exactlyallan force-pushed the fix/refresh-relay-skill-signatures branch from a5b5020 to 5c3a8b5 Compare July 17, 2026 20:56

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skills/nemo-relay-get-started/BENCHMARK.md`:
- Around line 32-46: The benchmark report’s overall PASS and publication
recommendation must not imply successful evaluation when dataset and Tier 3
results are unavailable. Update the verdict and recommendation sections
associated with the report to mark the benchmark incomplete/unscored, or
explicitly scope PASS to Tier 1 static validation only, while preserving the
documented N/A results.

In `@skills/nemo-relay-get-started/skill-card.md`:
- Around line 68-69: Add a blank line immediately after the “Skill Version(s)”
heading in the skill card, before the version value, so the heading is
surrounded by blank lines and satisfies markdownlint.

In `@skills/nemo-relay-plugin-observability/BENCHMARK.md`:
- Around line 32-46: Update the Overall verdict and publication recommendation
in BENCHMARK.md to reflect that the evaluation dataset and Tier 3 signals were
unavailable. Do not report an overall PASS; scope the verdict to completed
validations or mark the evaluation as incomplete/not evaluated.

In `@skills/nemo-relay-plugin-observability/skill-card.md`:
- Around line 40-42: Update the “Evaluation Agents Used” section in
skill-card.md by adding a blank line after the heading and formatting the agent
names `claude-code` and `codex` as inline code elements.
- Around line 34-35: Insert a blank line immediately after the headings “##
Skill Output: <br>” and “## Skill Version(s): <br>” in the skill card,
preserving the existing content and formatting otherwise.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 409b8863-4e50-48e9-a1fb-8e41e1d13ba2

📥 Commits

Reviewing files that changed from the base of the PR and between bd49cbf and 5c3a8b5.

📒 Files selected for processing (8)
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/skill-card.md
  • skills/nemo-relay-get-started/skill.oms.sig
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-plugin-observability/skill.oms.sig
📜 Review details
🧰 Additional context used
📓 Path-based instructions (9)
**/*.{md,rst,html,txt}

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)

**/*.{md,rst,html,txt}: Always spell NVIDIA in all caps. Do not use Nvidia, nvidia, nVidia, nVIDIA, or NV.
Use an NVIDIA before a noun because the name starts with an 'en' sound.
Do not add a registered trademark symbol after NVIDIA when referring to the company.
Use trademark symbols with product names only when the document type or legal guidance requires them.
Verify official capitalization, spacing, and hyphenation for product names.
Precede NVIDIA product names with NVIDIA on first mention when it is natural and accurate.
Do not rewrite product names for grammar or title-case rules.
Preserve third-party product names according to the owner's spelling.
Include the company name and full model qualifier on first use when it helps identify the model.
Preserve the official capitalization and punctuation of model names.
Use shorter family names only after the full name is established.
Spell out a term on first use and put the acronym in parentheses unless the acronym is widely understood by the intended audience.
Use the acronym on later mentions after it has been defined.
For long documents, reintroduce the full term if readers might lose context.
Form plurals of acronyms with s, not an apostrophe, such as GPUs.
In headings, common acronyms can remain abbreviated. Spell out the term in the first or second sentence of the body.
Common terms such as CPU, GPU, PC, API, and UI usually do not need to be spelled out for developer audiences.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*.{md,rst,html}

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-brand-terminology.md)

Link the first mention of a product name when the destination helps the reader.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*.{md,rst,txt}

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)

Spell NVIDIA in all caps. Do not use Nvidia, nvidia, or NV.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*.{md,rst}

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-guide.md)

**/*.{md,rst}: Format commands, code elements, expressions, package names, file names, and paths as inline code.
Use descriptive link text. Avoid raw URLs and weak anchors such as "here" or "read more."
Use title case consistently for technical documentation headings.
Introduce code blocks, lists, tables, and images with complete sentences.
Write procedures as imperative steps. Keep steps parallel and split long procedures into smaller tasks.
Prefer active voice, present tense, short sentences, contractions, and plain English.
Use can for possibility and reserve may for permission.
Use after for temporal relationships instead of once.
Prefer refer to over see when the wording points readers to another resource.
Avoid culture-specific idioms, unnecessary Latinisms, jokes, and marketing exaggeration in technical docs.
Spell out months in body text, avoid ordinal dates, and use clear time zones.
Spell out whole numbers from zero through nine unless they are technical values, parameters, versions, or UI values.
Use numerals for 10 or greater and include commas in thousands.
Do not add trademark symbols to learning-oriented docs unless the source, platform, or legal guidance explicitly requires them.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*.md

📄 CodeRabbit inference engine (.agents/skills/review-doc-style/assets/nvidia-style-technical-docs.md)

**/*.md: Use title case consistently in technical documentation headings
Avoid quotation marks, ampersands, and exclamation marks in headings
Keep product, event, research, and whitepaper names in their official title case
Use title case for table headers
Do not force social-media sentence case into technical docs
Format code elements, commands, parameters, package names, and expressions in monospace
Format directories, file names, and paths in monospace using backticks
Use angle brackets inside monospace for variables inside paths, such as /home/<username>/.login
Format error messages and strings in quotation marks, keeping literal code strings in code formatting when clearer
Format UI buttons, menus, fields, and labels in bold
Use angle brackets between UI labels for menu paths, such as File > Save As
Use italics for new terms on first use, sparingly and only when introducing the term
Use italics for publication titles
Format keyboard shortcuts in plain text, such as Press Ctrl+Alt+Delete
Use owner/repo link text for GitHub repositories, preferring [NVIDIA/NeMo](link) over prose references like 'the GitHub repo'
Introduce every code block with a complete sentence
Do not make a code block complete the grammar of the previous sentence
Do not continue a sentence after a code block
Use syntax highlighting when the format supports it for code blocks
Avoid the word 'snippet' unless the surrounding docs already use it as a term of art
Keep inline method, function, and class references consistent with nearby docs, omitting empty parentheses for prose readability when no call is shown
Use descriptive anchor text that matches the destination title when possible for links
Avoid raw URLs in running text
Avoid generic anchor text such as 'here,' 'this page,' and 'read more'
Include acronyms in link text when a linked term includes an acronym
Do not link long sentences or multiple sentences
Avoid links that pull readers away from a procedure unless the link is a p...

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/SKILL.md

📄 CodeRabbit inference engine (AGENTS.md)

SKILL.md files are skill entrypoints and must start with YAML frontmatter containing at least name and description.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md

⚙️ CodeRabbit configuration file

**/SKILL.md: Do not flag SKILL.md files for missing SPDX headers. Skill entrypoints intentionally start with YAML frontmatter instead.
Verify that every SKILL.md keeps valid YAML frontmatter with at least name and description fields before the Markdown body.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
**/*.{md,mdx}

📄 CodeRabbit inference engine (AGENTS.md)

Update README.md, fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.

**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such as RELEASING.md, not as user-facing docs pages or CHANGELOG.md
Keep stable user-facing wrappers at scripts/ root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, and grpc-v1 protocol details on separate pages

If links in documentation change, run just docs-linkcheck.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*.{md,markdown,mdx}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-get-started/skill-card.md
**/*

📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)

**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, use maintain-dynamic-plugins and include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, prefer uv run pre-commit run --files <changed files...>.
Before review or handoff, run uv run pre-commit run --all-files.

Files:

  • skills/nemo-relay-plugin-observability/SKILL.md
  • skills/nemo-relay-get-started/SKILL.md
  • skills/nemo-relay-get-started/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/BENCHMARK.md
  • skills/nemo-relay-plugin-observability/skill-card.md
  • skills/nemo-relay-plugin-observability/skill.oms.sig
  • skills/nemo-relay-get-started/skill.oms.sig
  • skills/nemo-relay-get-started/skill-card.md
🪛 markdownlint-cli2 (0.23.0)
skills/nemo-relay-plugin-observability/skill-card.md

[warning] 34-34: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)


[warning] 40-40: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)


[warning] 65-65: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

skills/nemo-relay-get-started/skill-card.md

[warning] 68-68: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🪛 SkillSpector (2.3.11)
skills/nemo-relay-get-started/SKILL.md

[warning] 33: [EA2] Autonomous Decision Making: Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.

Remediation: Add human-in-the-loop confirmation for destructive, irreversible, or high-impact operations. Never auto-execute commands that modify files, send data, or alter system state.

(Excessive Agency (EA2))

🔇 Additional comments (7)
skills/nemo-relay-get-started/skill-card.md (1)

12-12: LGTM!

Also applies to: 38-38, 48-49, 57-67

skills/nemo-relay-get-started/SKILL.md (1)

93-95: 📐 Maintainability & Code Quality

No frontmatter change needed. skills/nemo-relay-get-started/SKILL.md still has valid YAML frontmatter with both name and description.

			> Likely an incorrect or invalid review comment.
skills/nemo-relay-get-started/skill.oms.sig (1)

1-1: 🔒 Security & Privacy

No bundle drift to flag.

			> Likely an incorrect or invalid review comment.
skills/nemo-relay-plugin-observability/skill-card.md (1)

12-12: LGTM!

Also applies to: 28-31, 35-38, 54-63

skills/nemo-relay-plugin-observability/BENCHMARK.md (1)

10-12: LGTM!

skills/nemo-relay-plugin-observability/SKILL.md (1)

57-58: 📐 Maintainability & Code Quality

No action needed. skills/nemo-relay-plugin-observability/SKILL.md already starts with valid YAML frontmatter and non-empty name and description fields.

			> Likely an incorrect or invalid review comment.
skills/nemo-relay-plugin-observability/skill.oms.sig (1)

1-1: 🔒 Security & Privacy

Validate the DSSE bundle in a checkout that contains all referenced files The current checkout is missing at least one referenced resource (BENCHMARK.md), so the inline digest check cannot run as written. Keep the NVSkills validation-signature bot gate for the certificate chain and DSSE signature.

Comment on lines +32 to +46
- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

The benchmark dataset contained 15 evaluation tasks:

- Positive tasks: 14 tasks where the skill was expected to activate.
- Negative tasks: 1 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
The evaluation dataset was not available in this report payload.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 97% (+0%) | 97% (+10%) |
| Correctness | 8 | 89% (+61%) | 76% (+38%) |
| Discoverability | 8 | 82% (+50%) | 74% (+35%) |
| Effectiveness | 8 | 74% (+50%) | 65% (+32%) |
| Efficiency | 8 | 72% (+34%) | 68% (+29%) |
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not report an overall benchmark pass when evaluation data is unavailable.

This block now states that the dataset and Tier 3 signals are unavailable and every result is N/A, but the report still claims Overall verdict: PASS and recommends publication in Lines 13 and 64-66. Qualify the result as incomplete/unscored, or explicitly limit the pass to Tier 1 static validation so missing evaluation data cannot be interpreted as successful benchmark evidence.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-get-started/BENCHMARK.md` around lines 32 - 46, The
benchmark report’s overall PASS and publication recommendation must not imply
successful evaluation when dataset and Tier 3 results are unavailable. Update
the verdict and recommendation sections associated with the report to mark the
benchmark incomplete/unscored, or explicitly scope PASS to Tier 1 static
validation only, while preserving the documented N/A results.

Comment on lines 68 to +69
## Skill Version(s): <br>
0.6.0-alpha.20260715 (source: git tag) <br>
0.7.0-alpha.20260717 (source: git tag) <br>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a blank line after the heading.

markdownlint reports that ## Skill Version(s) is not surrounded by blank lines. Add an empty line before the version value.

Proposed fix
 ## Skill Version(s): <br>
+
 0.7.0-alpha.20260717 (source: git tag) <br>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## Skill Version(s): <br>
0.6.0-alpha.20260715 (source: git tag) <br>
0.7.0-alpha.20260717 (source: git tag) <br>
## Skill Version(s): <br>
0.7.0-alpha.20260717 (source: git tag) <br>
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 68-68: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-get-started/skill-card.md` around lines 68 - 69, Add a
blank line immediately after the “Skill Version(s)” heading in the skill card,
before the version value, so the heading is surrounded by blank lines and
satisfies markdownlint.

Source: Linters/SAST tools

Comment on lines +32 to +46
- No Tier 3 evaluation signal details were available in this report.

## Test Tasks

The benchmark dataset contained 20 evaluation tasks:

- Positive tasks: 16 tasks where the skill was expected to activate.
- Negative tasks: 4 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
The evaluation dataset was not available in this report payload.

## Results

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 100% (+0%) | 100% (+0%) |
| Correctness | 8 | 98% (+53%) | 93% (+31%) |
| Discoverability | 8 | 99% (+67%) | 85% (+42%) |
| Effectiveness | 8 | 91% (+47%) | 87% (+30%) |
| Efficiency | 8 | 92% (+51%) | 82% (+35%) |
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Do not report an overall PASS without evaluation evidence.

This section explicitly says the dataset and Tier 3 signals were unavailable, yet the report retains Overall verdict: PASS and a publication recommendation. Scope PASS to the validations actually completed, or mark the evaluation as incomplete/not evaluated.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-plugin-observability/BENCHMARK.md` around lines 32 - 46,
Update the Overall verdict and publication recommendation in BENCHMARK.md to
reflect that the evaluation dataset and Tier 3 signals were unavailable. Do not
report an overall PASS; scope the verdict to completed validations or mark the
evaluation as incomplete/not evaluated.

Comment on lines 34 to +35
## Skill Output: <br>
**Output Type(s):** [Configuration instructions, Code, Analysis] <br>
**Output Type(s):** [Configuration instructions, Code] <br>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add blank lines after the headings.

Insert a blank line after ## Skill Output: <br> and ## Skill Version(s): <br> to satisfy Markdown heading-spacing rules.

Also applies to: 65-66

🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 34-34: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-plugin-observability/skill-card.md` around lines 34 - 35,
Insert a blank line immediately after the headings “## Skill Output: <br>” and
“## Skill Version(s): <br>” in the skill card, preserving the existing content
and formatting otherwise.

Source: Linters/SAST tools

Comment on lines 40 to +42
## Evaluation Agents Used: <br>
- `claude-code` <br>
- `codex` <br>
- claude-code <br>
- codex <br>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Keep agent names in inline code and separate the heading with a blank line.

Format claude-code and codex as code elements, and add a blank line after the heading. As per coding guidelines, format code elements in monospace.

🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 40-40: Headings should be surrounded by blank lines
Expected: 1; Actual: 0; Below

(MD022, blanks-around-headings)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-relay-plugin-observability/skill-card.md` around lines 40 - 42,
Update the “Evaluation Agents Used” section in skill-card.md by adding a blank
line after the heading and formatting the agent names `claude-code` and `codex`
as inline code elements.

Sources: Coding guidelines, Linters/SAST tools

@exactlyallan exactlyallan added the DO NOT MERGE PR should not be merged; see PR for details label Jul 17, 2026
@exactlyallan

Copy link
Copy Markdown
Collaborator Author

Waiting for CI fixes to enable benchmark data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DO NOT MERGE PR should not be merged; see PR for details Maintenance CI or Build or general repository maintenance size:M PR is medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants