feat: Add agent-failure-investigator kit - #284
Open
yshsh218-crypto wants to merge 1 commit into
Open
Conversation
Contributor
WalkthroughSummaryMission briefing: adds “Agent Failure Investigator,” a browser-based deterministic trace investigation tool with framework adapters, rule-based classification, evidence and timeline reconstruction, reports, remediation, comparison, exports, benchmarks, documentation, and regression tests. ChangesTrace contracts and usage documentation
Trace ingestion and application shell
Rule catalog and investigation engine
Reports, visualization, and browser workflows
Benchmarking and regression validation
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
@yshsh218-crypto the pr is not up to the mark. Please check the CONTRIBUTING.MD file |
yshsh218-crypto
changed the title
Contributor
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
❌ Errors
🛑 Please fix the errors above before this PR can be merged. Refer to CONTRIBUTING.md and CLAUDE.md for the expected folder structure. |
Contributor
|
Failure recorded at 2026-07-27T16:53:22Z UTC. If this PR is not fixed within 4 weeks it will be automatically closed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 16
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/AGENT/agent-failure-investigator/bench/run.js`:
- Around line 7-13: Extract the VM sandbox creation and script-loading logic
from the bench runner into a shared harness helper, such as loadEngine(scripts),
and update run.js and scale.js to use it. Reuse the helper in tests/run-tests.js
where its bootstrap flow matches, preserving each caller’s existing script lists
and returned engine behavior.
In `@kits/AGENT/agent-failure-investigator/bench/scale.js`:
- Around line 23-29: Remove the duplicated VM bootstrap from the top-level setup
in scale.js and reuse the shared bootstrap logic established for bench/run.js.
Preserve loading the same rule files and js/engine.js into the sandbox, using
the existing shared symbol rather than maintaining a second implementation.
In `@kits/AGENT/agent-failure-investigator/css/style.css`:
- Around line 419-424: Update the color declarations in .remedy-why and the
additionally affected selectors to use the declared --text custom property
instead of undefined --fg, preserving their existing opacity and styling.
- Around line 405-442: Replace the repeated literal font stacks in .fw-badge,
.format-pill, .gnode-label, .gnode-tag, .remedy-tag, .cmp-pickers select,
.cmp-vs, and .cmp-table td with the existing var(--mono) and var(--display)
variables, preserving each selector’s current font-family role.
- Line 1: Update the Stylelint configuration used by
kits/medical-assistant/apps/.stylelintrc.json to make scss/at-rule-no-unknown
resolvable by explicitly registering the stylelint-scss plugin and its
dependency in the relevant package manifests, or replace the rule with core
at-rule-no-unknown. Ensure linting no longer reports the configured rule as
unknown.
In `@kits/AGENT/agent-failure-investigator/index.html`:
- Around line 82-85: Add an accessible name to the API-key input identified by
id="api-key" in the llm-row, using aria-label="Anthropic API key" or an
associated label element; keep the existing placeholder and control behavior
unchanged.
In `@kits/AGENT/agent-failure-investigator/js/adapters.js`:
- Around line 29-32: Update the native adapter’s transformation callback in
Adapter("native", ...) to merge accepted native input with emptyTrace(), filling
missing canonical trace sections while preserving provided values. Ensure
importTrace receives the normalized result instead of returning the claimed
native object unchanged.
- Around line 60-62: The status mapping at
kits/AGENT/agent-failure-investigator/js/adapters.js lines 60-62 must detect
timeout text before generic error matching, producing the canonical timeout
status; preserve success only when neither condition matches. At
kits/AGENT/agent-failure-investigator/js/adapters.js lines 160-164, normalize
provider statuses such as failed to error and all timeout variants to timeout,
ensuring both adapter paths emit only the documented status enum.
In `@kits/AGENT/agent-failure-investigator/js/app.js`:
- Around line 192-203: Escape the user-supplied e.ts value before interpolating
it into the SVG text and timeline chip HTML. Update the marks rendering and the
events.forEach block that creates tl-chip elements, reusing the existing
escapeHtml helper for timestamps while preserving the current empty-string
fallback.
In `@kits/AGENT/agent-failure-investigator/js/exporter.js`:
- Around line 79-91: Guard the result of window.open in the PDF export flow
before accessing w.document. If the popup is blocked and w is null, stop the
export gracefully and provide user feedback; otherwise preserve the existing
document.write and document.close behavior.
In `@kits/AGENT/agent-failure-investigator/js/report.js`:
- Around line 43-77: Update composeReport’s rootCause construction to safely
handle a result.primary key missing from ROOT_CAUSE_TEMPLATES, using the
existing generic fallback behavior rather than invoking an undefined template.
Preserve the current template-based message for known categories and the
no-primary response unchanged.
In `@kits/AGENT/agent-failure-investigator/rules/core.js`:
- Around line 9-11: Update definePlugin so it validates spec.id against existing
entries in the shared RULES array before registering; throw a clear error on
duplicate IDs, otherwise preserve the current push-and-return behavior.
In `@kits/AGENT/agent-failure-investigator/rules/wrong_tool.js`:
- Around line 11-29: Update the test method’s used-tool scoring to track the
tool or call index that produces the maximum affinity in bar, then use that
tracked call in the evidence message and refs instead of always using calls[0].
Preserve the winner comparison and ensure the reported score and referenced tool
remain consistent with the highest-affinity used call.
In `@kits/AGENT/agent-failure-investigator/tests/run-tests.js`:
- Around line 16-27: Refactor the VM bootstrap in tests/run-tests.js to reuse
the shared bootstrap mechanism introduced for bench/run.js and bench/scale.js
instead of creating its own context and evaluating SCRIPTS inline. Preserve the
existing script loading order and test behavior while removing the duplicated
ROOT, ctx, and vm.runInContext setup.
In `@kits/AGENT/README.md`:
- Line 5: Add language identifiers to every unlabeled fenced code block in
kits/AGENT/README.md and kits/AGENT/agent-failure-investigator/README.md, using
text for diagrams and layouts and bash for command examples, including the
duplicated fences at the referenced locations, so markdownlint passes.
- Around line 114-117: Update the setup instructions so users enter the kit
directory before invoking it: in kits/AGENT/README.md lines 114-117 and 123-127,
add the directory change before opening index.html and ensure test and benchmark
commands run from that directory; apply the same changes to
kits/AGENT/agent-failure-investigator/README.md lines 114-117 and 123-127.
Preserve each README’s human-readable setup guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 76779e03-e49e-4082-9f92-25fa44b50257
⛔ Files ignored due to path filters (3)
kits/AGENT/agent-failure-investigator.zipis excluded by!**/*.zipkits/AGENT/agent-failure-investigator/docs/architecture.svgis excluded by!**/*.svgkits/AGENT/architecture.svgis excluded by!**/*.svg
📒 Files selected for processing (27)
kits/AGENT/README.mdkits/AGENT/agent-failure-investigator/README.mdkits/AGENT/agent-failure-investigator/bench/forge.jskits/AGENT/agent-failure-investigator/bench/results.jsonkits/AGENT/agent-failure-investigator/bench/run.jskits/AGENT/agent-failure-investigator/bench/scale-results.jsonkits/AGENT/agent-failure-investigator/bench/scale.jskits/AGENT/agent-failure-investigator/css/style.csskits/AGENT/agent-failure-investigator/docs/adapters.mdkits/AGENT/agent-failure-investigator/docs/trace-schema.mdkits/AGENT/agent-failure-investigator/index.htmlkits/AGENT/agent-failure-investigator/js/adapters.jskits/AGENT/agent-failure-investigator/js/advisor.jskits/AGENT/agent-failure-investigator/js/app.jskits/AGENT/agent-failure-investigator/js/compare.jskits/AGENT/agent-failure-investigator/js/engine.jskits/AGENT/agent-failure-investigator/js/exporter.jskits/AGENT/agent-failure-investigator/js/graph.jskits/AGENT/agent-failure-investigator/js/report.jskits/AGENT/agent-failure-investigator/js/traces.jskits/AGENT/agent-failure-investigator/rules/core.jskits/AGENT/agent-failure-investigator/rules/hallucination.jskits/AGENT/agent-failure-investigator/rules/prompt.jskits/AGENT/agent-failure-investigator/rules/rag.jskits/AGENT/agent-failure-investigator/rules/tool_failure.jskits/AGENT/agent-failure-investigator/rules/wrong_tool.jskits/AGENT/agent-failure-investigator/tests/run-tests.js
Comment on lines
+7
to
+13
| const ROOT = path.join(__dirname, ".."); | ||
| const sandbox = { console, module: { exports: {} } }; | ||
| vm.createContext(sandbox); | ||
| [ | ||
| "rules/core.js", "rules/tool_failure.js", "rules/hallucination.js", | ||
| "rules/rag.js", "rules/prompt.js", "rules/wrong_tool.js", "js/engine.js" | ||
| ].forEach(f => vm.runInContext(fs.readFileSync(path.join(ROOT, f), "utf8"), sandbox, { filename: f })); |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Duplicate VM bootstrap logic across bench scripts.
This sandbox-creation + script-loading block is copy-pasted verbatim in bench/scale.js (lines 23-29), and a close variant lives in tests/run-tests.js (lines 16-27). Extracting a shared harness (e.g. a loadEngine(scripts) helper) would cut the triplicated boilerplate and give a single place to update if the rule-file list changes.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 12-12: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(ROOT, f), "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/bench/run.js` around lines 7 - 13,
Extract the VM sandbox creation and script-loading logic from the bench runner
into a shared harness helper, such as loadEngine(scripts), and update run.js and
scale.js to use it. Reuse the helper in tests/run-tests.js where its bootstrap
flow matches, preserving each caller’s existing script lists and returned engine
behavior.
Comment on lines
+23
to
+29
| const ROOT = path.join(__dirname, ".."); | ||
| const sandbox = { console, module: { exports: {} } }; | ||
| vm.createContext(sandbox); | ||
| [ | ||
| "rules/core.js", "rules/tool_failure.js", "rules/hallucination.js", | ||
| "rules/rag.js", "rules/prompt.js", "rules/wrong_tool.js", "js/engine.js" | ||
| ].forEach(f => vm.runInContext(fs.readFileSync(path.join(ROOT, f), "utf8"), sandbox, { filename: f })); |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Same VM bootstrap duplication as bench/run.js.
Identical to bench/run.js lines 7-13 — see consolidated note.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 28-28: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(ROOT, f), "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/bench/scale.js` around lines 23 - 29,
Remove the duplicated VM bootstrap from the top-level setup in scale.js and
reuse the shared bootstrap logic established for bench/run.js. Preserve loading
the same rule files and js/engine.js into the sandbox, using the existing shared
symbol rather than maintaining a second implementation.
| @@ -0,0 +1,442 @@ | |||
| /* Agent Failure Investigator — diagnostic console theme. | |||
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate the Stylelint config and check whether stylelint-scss is registered
fd -H -i '.stylelintrc*' .
fd -H stylelint.config.js .
rg -n 'scss/at-rule-no-unknown|stylelint-scss' -g '!node_modules'Repository: Lamatic/AgentKit
Length of output: 249
🏁 Script executed:
#!/bin/bash
set -u
echo "== Stylelint config files =="
for f in $(fd -H -i '.stylelintrc*' . | sort); do
echo "--- $f"
cat -n "$f"
done
for f in $(fd -H stylelint.config.js . | sort); do
echo "--- $f"
cat -n "$f"
done
echo "== Stylelint-related occurrences outside node_modules =="
rg -n 'stylelint|scss/at-rule-no-unknown|stylelint-scss|at-rule-no-unknown' -g '!node_modules' . || true
echo "== Package manifests with stylelint deps =="
fd -H 'package.json$' . -x sh -c 'echo "--- $1"; rg -n "stylelint|stylelint-scss" "$1" || true' sh {}Repository: Lamatic/AgentKit
Length of output: 2777
Make the SCSS rule dependency explicit.
kits/medical-assistant/apps/.stylelintrc.json enables scss/at-rule-no-unknown but no Stylelint dependency configuration is present in package manifests, so this rule can fail as unknown unless the lint tool/runtime adds stylelint-scss as a plugin/dependency. Add the plugin/config registration, or replace it with core at-rule-no-unknown.
🧰 Tools
🪛 Stylelint (17.14.0)
[error] 1-1: Unknown rule scss/at-rule-no-unknown. Did you mean at-rule-no-unknown?
(scss/at-rule-no-unknown)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/css/style.css` at line 1, Update the
Stylelint configuration used by kits/medical-assistant/apps/.stylelintrc.json to
make scss/at-rule-no-unknown resolvable by explicitly registering the
stylelint-scss plugin and its dependency in the relevant package manifests, or
replace the rule with core at-rule-no-unknown. Ensure linting no longer reports
the configured rule as unknown.
Source: Linters/SAST tools
Comment on lines
+405
to
+442
| .fw-strip { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; margin: 6px 0 22px; } | ||
| .fw-strip-label { font-size: 12px; color: var(--dim); letter-spacing: 0.06em; text-transform: uppercase; } | ||
| .fw-badge { font-family: "IBM Plex Mono", monospace; font-size: 12px; padding: 4px 10px; border: 1px solid var(--line); border-radius: 999px; color: var(--green, #3ecf8e); background: rgba(62, 207, 142, 0.06); } | ||
|
|
||
| .upload-label { display: inline-flex; align-items: center; cursor: pointer; } | ||
| .upload-label.small { font-size: 12px; padding: 4px 10px; margin-left: 6px; } | ||
| .format-pill { font-family: "IBM Plex Mono", monospace; font-size: 12px; padding: 5px 12px; border-radius: 999px; border: 1px solid var(--blue, #5aa9ff); color: var(--blue, #5aa9ff); } | ||
|
|
||
| #graph-scroll { overflow-x: auto; padding: 8px 0; } | ||
| .gnode-label { font-family: "Space Grotesk", sans-serif; font-size: 15px; font-weight: 600; } | ||
| .gnode-tag { font-family: "IBM Plex Mono", monospace; font-size: 10px; letter-spacing: 0.12em; } | ||
| .gnode-bad rect { animation: pulse-bad 1.6s ease-in-out infinite; } | ||
| @keyframes pulse-bad { 50% { stroke-opacity: 0.45; } } | ||
|
|
||
| .remedy { display: flex; gap: 12px; align-items: baseline; padding: 8px 0; border-bottom: 1px dashed var(--line); } | ||
| .remedy:last-child { border-bottom: none; } | ||
| .remedy-tag { font-family: "IBM Plex Mono", monospace; font-size: 12px; white-space: nowrap; padding: 3px 9px; border: 1px solid var(--green, #3ecf8e); color: var(--green, #3ecf8e); border-radius: 5px; } | ||
| .remedy-secondary .remedy-tag { border-color: var(--dim); color: var(--dim); } | ||
| .remedy-why { font-size: 13.5px; color: var(--fg); opacity: 0.85; } | ||
|
|
||
| .export-row { display: flex; gap: 10px; margin-top: 18px; } | ||
|
|
||
| .cmp-pickers { display: flex; align-items: flex-end; gap: 18px; flex-wrap: wrap; margin-bottom: 16px; } | ||
| .cmp-pickers select { background: transparent; color: var(--fg); border: 1px solid var(--line); border-radius: 6px; padding: 7px 10px; font-family: "IBM Plex Mono", monospace; font-size: 12.5px; max-width: 320px; } | ||
| .cmp-vs { font-family: "Space Grotesk", sans-serif; font-weight: 700; color: var(--dim); padding-bottom: 8px; } | ||
| .cmp-table { width: 100%; border-collapse: collapse; margin: 6px 0 18px; font-size: 13.5px; } | ||
| .cmp-table th { text-align: left; font-size: 11px; letter-spacing: 0.08em; text-transform: uppercase; color: var(--dim); padding: 6px 10px; border-bottom: 1px solid var(--line); } | ||
| .cmp-table td { padding: 8px 10px; border-bottom: 1px dashed var(--line); font-family: "IBM Plex Mono", monospace; } | ||
| .cmp-badge { font-size: 11px; padding: 2px 9px; border-radius: 999px; border: 1px solid; } | ||
| .cmp-better { color: var(--green, #3ecf8e); border-color: var(--green, #3ecf8e); } | ||
| .cmp-worse { color: var(--red, #ff4d4d); border-color: var(--red, #ff4d4d); } | ||
| .cmp-same { color: var(--dim); border-color: var(--line); } | ||
| .cmp-changed { color: var(--amber, #f2b544); border-color: var(--amber, #f2b544); } | ||
| .cmp-cols { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: 18px; } | ||
| .cmp-rule { font-size: 13px; padding: 6px 0; border-bottom: 1px dashed var(--line); } | ||
| .cmp-solved { color: var(--green, #3ecf8e); } | ||
| .cmp-new { color: var(--red, #ff4d4d); } | ||
| .cmp-open { color: var(--amber, #f2b544); } |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Consider reusing var(--mono) / var(--display) instead of repeating literal font stacks.
Lines 407, 411, 414-415, 421, 428, 430, 432 all hardcode "IBM Plex Mono", monospace / "Space Grotesk", sans-serif even though --mono/--display are already declared for exactly this purpose. Consolidating keeps the font stack a single source of truth.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/css/style.css` around lines 405 - 442,
Replace the repeated literal font stacks in .fw-badge, .format-pill,
.gnode-label, .gnode-tag, .remedy-tag, .cmp-pickers select, .cmp-vs, and
.cmp-table td with the existing var(--mono) and var(--display) variables,
preserving each selector’s current font-family role.
Comment on lines
+419
to
+424
| .remedy { display: flex; gap: 12px; align-items: baseline; padding: 8px 0; border-bottom: 1px dashed var(--line); } | ||
| .remedy:last-child { border-bottom: none; } | ||
| .remedy-tag { font-family: "IBM Plex Mono", monospace; font-size: 12px; white-space: nowrap; padding: 3px 9px; border: 1px solid var(--green, #3ecf8e); color: var(--green, #3ecf8e); border-radius: 5px; } | ||
| .remedy-secondary .remedy-tag { border-color: var(--dim); color: var(--dim); } | ||
| .remedy-why { font-size: 13.5px; color: var(--fg); opacity: 0.85; } | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Undefined custom property var(--fg) — did you mean --text?
--fg is never declared in :root (only --text/--dim exist). Since color is inherited, this currently self-recovers by inheriting the ambient text color, but it's a landmine if these selectors ever move to a differently-colored context — the kind of thing that only reveals itself during the actual mission, not the rehearsal.
🎨 Proposed fix
-.remedy-why { font-size: 13.5px; color: var(--fg); opacity: 0.85; }
+.remedy-why { font-size: 13.5px; color: var(--text); opacity: 0.85; }-.cmp-pickers select { background: transparent; color: var(--fg); border: 1px solid var(--line); border-radius: 6px; padding: 7px 10px; font-family: "IBM Plex Mono", monospace; font-size: 12.5px; max-width: 320px; }
+.cmp-pickers select { background: transparent; color: var(--text); border: 1px solid var(--line); border-radius: 6px; padding: 7px 10px; font-family: var(--mono); font-size: 12.5px; max-width: 320px; }Also applies to: 427-429
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/css/style.css` around lines 419 - 424,
Update the color declarations in .remedy-why and the additionally affected
selectors to use the declared --text custom property instead of undefined --fg,
preserving their existing opacity and styling.
Comment on lines
+9
to
+11
| const RULES = []; | ||
|
|
||
| const definePlugin = spec => { RULES.push(spec); return spec; }; |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Your mission, should you choose to accept it: guard against duplicate rule IDs.
definePlugin pushes blindly into the shared RULES array with no check that spec.id is unique. Since six separate files register into this one global, a typo or accidental double-registration (e.g. R14 defined twice) would silently corrupt scoring/report output with no error trail — self-destructing evidence, Ethan-Hunt style, except nobody notices.
♻️ Proposed fix
-const definePlugin = spec => { RULES.push(spec); return spec; };
+const definePlugin = spec => {
+ if (RULES.some(r => r.id === spec.id)) {
+ throw new Error(`Duplicate rule id: ${spec.id}`);
+ }
+ RULES.push(spec);
+ return spec;
+};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const RULES = []; | |
| const definePlugin = spec => { RULES.push(spec); return spec; }; | |
| const RULES = []; | |
| const definePlugin = spec => { | |
| if (RULES.some(r => r.id === spec.id)) { | |
| throw new Error(`Duplicate rule id: ${spec.id}`); | |
| } | |
| RULES.push(spec); | |
| return spec; | |
| }; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/rules/core.js` around lines 9 - 11,
Update definePlugin so it validates spec.id against existing entries in the
shared RULES array before registering; throw a clear error on duplicate IDs,
otherwise preserve the current push-and-return behavior.
Comment on lines
+11
to
+29
| test(trace) { | ||
| const query = lastUserMessage(trace); | ||
| const calls = trace.tool_calls || []; | ||
| const toolbox = trace.available_tools || []; | ||
| if (!query || !calls.length || toolbox.length < 2) return null; | ||
| const used = new Set(calls.map(c => c.tool)); | ||
| let bar = toolbox.filter(t => used.has(t.name)).reduce((m, t) => Math.max(m, affinity(query, t)), 0); | ||
| let winner = null; | ||
| toolbox.forEach(t => { | ||
| if (used.has(t.name)) return; | ||
| const s = affinity(query, t); | ||
| if (s > bar && s >= 2) { winner = t; bar = s; } | ||
| }); | ||
| if (!winner) return null; | ||
| return { | ||
| evidence: `The query matches "${winner.name}" (${bar} keyword overlaps: internal/policy/SLA-type terms) far better than the tool actually called, "${calls[0].tool}".`, | ||
| refs: [{ type: "tool", index: 0 }] | ||
| }; | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Evidence in R41 can point the finger at the wrong tool call.
bar (line 17) is the max affinity across all used tools, but the evidence string and refs (lines 26-27) always reference calls[0] as "the tool actually called." When more than one tool call exists and the highest-affinity used tool isn't calls[0], the report cites a bar score for a tool that isn't actually the one named — an incident investigator would be chasing the wrong lead. This mission has real stakes: misattributed evidence in a failure-investigator is worse than no evidence.
🕵️ Proposed fix — track which used call produced `bar`
test(trace) {
const query = lastUserMessage(trace);
const calls = trace.tool_calls || [];
const toolbox = trace.available_tools || [];
if (!query || !calls.length || toolbox.length < 2) return null;
const used = new Set(calls.map(c => c.tool));
- let bar = toolbox.filter(t => used.has(t.name)).reduce((m, t) => Math.max(m, affinity(query, t)), 0);
+ let bar = 0;
+ let bestUsedIdx = 0;
+ calls.forEach((c, i) => {
+ const t = toolbox.find(tb => tb.name === c.tool);
+ const s = t ? affinity(query, t) : 0;
+ if (s > bar) { bar = s; bestUsedIdx = i; }
+ });
let winner = null;
toolbox.forEach(t => {
if (used.has(t.name)) return;
const s = affinity(query, t);
if (s > bar && s >= 2) { winner = t; bar = s; }
});
if (!winner) return null;
return {
- evidence: `The query matches "${winner.name}" (${bar} keyword overlaps: internal/policy/SLA-type terms) far better than the tool actually called, "${calls[0].tool}".`,
- refs: [{ type: "tool", index: 0 }]
+ evidence: `The query matches "${winner.name}" (${bar} keyword overlaps) far better than the tool actually called, "${calls[bestUsedIdx].tool}".`,
+ refs: [{ type: "tool", index: bestUsedIdx }]
};
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| test(trace) { | |
| const query = lastUserMessage(trace); | |
| const calls = trace.tool_calls || []; | |
| const toolbox = trace.available_tools || []; | |
| if (!query || !calls.length || toolbox.length < 2) return null; | |
| const used = new Set(calls.map(c => c.tool)); | |
| let bar = toolbox.filter(t => used.has(t.name)).reduce((m, t) => Math.max(m, affinity(query, t)), 0); | |
| let winner = null; | |
| toolbox.forEach(t => { | |
| if (used.has(t.name)) return; | |
| const s = affinity(query, t); | |
| if (s > bar && s >= 2) { winner = t; bar = s; } | |
| }); | |
| if (!winner) return null; | |
| return { | |
| evidence: `The query matches "${winner.name}" (${bar} keyword overlaps: internal/policy/SLA-type terms) far better than the tool actually called, "${calls[0].tool}".`, | |
| refs: [{ type: "tool", index: 0 }] | |
| }; | |
| } | |
| test(trace) { | |
| const query = lastUserMessage(trace); | |
| const calls = trace.tool_calls || []; | |
| const toolbox = trace.available_tools || []; | |
| if (!query || !calls.length || toolbox.length < 2) return null; | |
| const used = new Set(calls.map(c => c.tool)); | |
| let bar = 0; | |
| let bestUsedIdx = 0; | |
| calls.forEach((c, i) => { | |
| const t = toolbox.find(tb => tb.name === c.tool); | |
| const s = t ? affinity(query, t) : 0; | |
| if (s > bar) { bar = s; bestUsedIdx = i; } | |
| }); | |
| let winner = null; | |
| toolbox.forEach(t => { | |
| if (used.has(t.name)) return; | |
| const s = affinity(query, t); | |
| if (s > bar && s >= 2) { winner = t; bar = s; } | |
| }); | |
| if (!winner) return null; | |
| return { | |
| evidence: `The query matches "${winner.name}" (${bar} keyword overlaps) far better than the tool actually called, "${calls[bestUsedIdx].tool}".`, | |
| refs: [{ type: "tool", index: bestUsedIdx }] | |
| }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/rules/wrong_tool.js` around lines 11 -
29, Update the test method’s used-tool scoring to track the tool or call index
that produces the maximum affinity in bar, then use that tracked call in the
evidence message and refs instead of always using calls[0]. Preserve the winner
comparison and ensure the reported score and referenced tool remain consistent
with the highest-affinity used call.
Comment on lines
+16
to
+27
| const ROOT = path.join(__dirname, ".."); | ||
| const ctx = { console }; | ||
| vm.createContext(ctx); | ||
| const SCRIPTS = [ | ||
| "js/traces.js", | ||
| "rules/core.js", "rules/tool_failure.js", "rules/hallucination.js", | ||
| "rules/rag.js", "rules/prompt.js", "rules/wrong_tool.js", | ||
| "js/adapters.js", "js/engine.js", "js/advisor.js", "js/compare.js", "js/report.js" | ||
| ]; | ||
| for (const f of SCRIPTS) { | ||
| vm.runInContext(fs.readFileSync(path.join(ROOT, f), "utf8"), ctx, { filename: f }); | ||
| } |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Third copy of the VM bootstrap pattern.
Same root cause as bench/run.js/bench/scale.js — see consolidated note.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 25-25: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(path.join(ROOT, f), "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/agent-failure-investigator/tests/run-tests.js` around lines 16 -
27, Refactor the VM bootstrap in tests/run-tests.js to reuse the shared
bootstrap mechanism introduced for bench/run.js and bench/scale.js instead of
creating its own context and evaluating SCRIPTS inline. Preserve the existing
script loading order and test behavior while removing the duplicated ROOT, ctx,
and vm.runInContext setup.
|
|
||
| **A forensic diagnostic tool for AI agents.** Upload a failed agent trace — from **LangGraph, OpenAI Agents SDK, CrewAI, AutoGen, Lamatic, or the native format** — and get a structured failure report with clickable evidence, a failure-propagation graph, a reconstructed timeline, a remediation playbook, and an exportable investigation document. | ||
|
|
||
| ``` |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Mission: identify the fenced-block languages. markdownlint reports unlabeled fences throughout both READMEs. Mark diagrams and layouts as text, and commands as bash, so documentation lint passes.
kits/AGENT/README.md#L5-L5: add language identifiers to this and the remaining unlabeled fences.kits/AGENT/agent-failure-investigator/README.md#L5-L5: add the same identifiers to the duplicated fences.
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)
[warning] 5-5: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
📍 Affects 2 files
kits/AGENT/README.md#L5-L5(this comment)kits/AGENT/agent-failure-investigator/README.md#L5-L5
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/README.md` at line 5, Add language identifiers to every unlabeled
fenced code block in kits/AGENT/README.md and
kits/AGENT/agent-failure-investigator/README.md, using text for diagrams and
layouts and bash for command examples, including the duplicated fences at the
referenced locations, so markdownlint passes.
Source: Linters/SAST tools
Comment on lines
+114
to
+117
| ``` | ||
| git clone <repo> | ||
| open index.html # or just double-click it | ||
| ``` |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Mission: enter the kit before invoking it. The documented commands run from the cloned repository root, while index.html, tests/, and bench/ live in kits/AGENT/agent-failure-investigator.
kits/AGENT/README.md#L114-L117: addcd kits/AGENT/agent-failure-investigatorbefore openingindex.html.kits/AGENT/README.md#L123-L127: ensure test and benchmark commands run after thatcd.kits/AGENT/agent-failure-investigator/README.md#L114-L117: add the same directory change.kits/AGENT/agent-failure-investigator/README.md#L123-L127: ensure test and benchmark commands run after thatcd.
As per coding guidelines, kits/*/README.md: Each kit must contain a README.md file with human-readable setup guide.
🧰 Tools
🪛 markdownlint-cli2 (0.23.0)
[warning] 114-114: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
📍 Affects 2 files
kits/AGENT/README.md#L114-L117(this comment)kits/AGENT/README.md#L123-L127kits/AGENT/agent-failure-investigator/README.md#L114-L117kits/AGENT/agent-failure-investigator/README.md#L123-L127
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/AGENT/README.md` around lines 114 - 117, Update the setup instructions
so users enter the kit directory before invoking it: in kits/AGENT/README.md
lines 114-117 and 123-127, add the directory change before opening index.html
and ensure test and benchmark commands run from that directory; apply the same
changes to kits/AGENT/agent-failure-investigator/README.md lines 114-117 and
123-127. Preserve each README’s human-readable setup guidance.
Source: Coding guidelines
Contributor
|
Hi @yshsh218-crypto! 👋 Before this PR can be reviewed by maintainers, please resolve all comments and requested changes from the CodeRabbit automated review. Steps to follow:
This helps keep the review process efficient for everyone. Thank you! 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
1. Select Contribution Type
kits/<category>/<kit-name>/)bundles/<bundle-name>/)templates/<template-name>/)2. General Requirements
kebab-caseand matches the flow IDREADME.md(purpose, setup, usage)3. File Structure (Check what applies)
config.jsonpresent with valid metadata (name, description, tags, steps, author, env keys)flows/<flow-name>/(where applicable) include:config.json(Lamatic flow export)inputs.jsonmeta.jsonREADME.md.env.examplewith placeholder values only (kits only)config.jsonnode graphs (changes via Lamatic Studio export)4. Validation
npm install && npm run devworks locally (kits: UI runs; bundles/templates: flows are valid)[kit] Add <name> for <use case>)