refactor: use gh aw logs for token analysis workflows by lpcox · Pull Request #1884 · github/gh-aw-firewall

lpcox · 2026-04-10T18:09:09Z

Summary

Replaces manual artifact downloading with gh aw logs --json in all 4 token analysis workflows. This aligns with the approach used by gh-aw's own copilot-token-audit and copilot-token-optimizer workflows.

Motivation

Our token workflows were manually discovering runs, downloading artifacts by name, and parsing raw token-usage.jsonl files. This broke when the artifact name changed from agent-artifacts to firewall-audit-logs (see #1883). The gh aw logs CLI handles all of this correctly — it knows the correct artifact names (including backward compat), downloads and organizes data, and outputs structured JSON with pre-aggregated token metrics.

Changes

New shared component

shared/mcp/gh-aw.md — Installs the gh aw CLI (mirrors upstream gh-aw's component)

Token Usage Analyzers (Copilot + Claude)

Added shared/mcp/gh-aw.md import and steps: block with gh aw logs --engine <engine> --json
Rewrote prompts to read pre-downloaded structured JSON instead of manual artifact downloading
Removed *.blob.core.windows.net from network allowlist

Token Optimizers (Copilot + Claude)

Added shared/mcp/gh-aw.md import and steps: block with 7-day data pre-download
Replaced manual gh run download with pre-downloaded data filtering

Benefits

No artifact name dependency — eliminates the root cause of fix: token analyzers look for wrong artifact name #1883
Richer data — includes token totals, costs, turns, tool usage
Consistent with gh-aw's recommended pattern
Faster — data pre-downloaded in steps block instead of agent-time downloads

Refs: #1883, github/gh-aw#25683

Replace manual artifact downloading (gh run download --name firewall-audit-logs) with gh aw logs --json in all 4 token analysis workflows. This aligns with the approach used by gh-aw's own token-audit and token-optimizer workflows. Benefits: - Eliminates dependency on knowing the correct artifact name (the root cause of the bug fixed in PR #1883) - Gets pre-aggregated structured data (token counts, costs, turns) instead of parsing raw token-usage.jsonl - Handles artifact naming backward/forward compat automatically - Pre-downloads data in steps block (faster, more reliable) Changes: - copilot-token-usage-analyzer.md: replaced manual run discovery + artifact download with gh aw logs --engine copilot --json pre-step - claude-token-usage-analyzer.md: same for --engine claude - copilot-token-optimizer.md: added gh aw logs --engine copilot pre-step for 7-day data, removed manual artifact download - claude-token-optimizer.md: same for --engine claude - shared/mcp/gh-aw.md: new shared component to install gh-aw CLI (mirroring upstream gh-aw's shared/mcp/gh-aw.md) Refs: #1883, github/gh-aw#25683 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-10T18:10:13Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	85.85%	85.95%	📈 +0.10%
Statements	85.76%	85.85%	📈 +0.09%
Functions	87.54%	87.54%	➡️ +0.00%
Branches	78.56%	78.61%	📈 +0.05%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	86.3% → 86.6% (+0.36%)	85.9% → 86.2% (+0.35%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Refactors the Copilot/Claude token usage analyzer and optimizer workflows to rely on gh aw logs --json for structured, pre-aggregated token metrics instead of manually discovering runs and downloading/parsing artifacts. This reduces coupling to artifact naming/paths and aligns these workflows with the gh-aw recommended approach.

Changes:

Added a shared workflow component to install the gh aw CLI extension for pre-agent steps.
Updated 4 token-analysis workflows to pre-download structured run/log data (gh aw logs --engine … --json) and adjusted prompts to consume that JSON.
Removed the Azure blob domain from workflow network allowlists since artifacts are no longer downloaded directly.

Show a summary per file

File	Description
.github/workflows/shared/mcp/gh-aw.md	New shared component to install gh-aw and stage the binary for use in workflows.
.github/workflows/copilot-token-usage-analyzer.md	Switches analyzer data source to pre-downloaded `gh aw logs --json` output; updates prompt instructions accordingly.
.github/workflows/copilot-token-usage-analyzer.lock.yml	Compiled workflow reflecting the new import/steps and tightened allowed domains.
.github/workflows/copilot-token-optimizer.md	Switches optimizer artifact analysis to structured log JSON over a 7-day window; updates guidance.
.github/workflows/copilot-token-optimizer.lock.yml	Compiled workflow reflecting the new import/steps.
.github/workflows/claude-token-usage-analyzer.md	Switches analyzer data source to pre-downloaded `gh aw logs --json` output; updates prompt instructions accordingly.
.github/workflows/claude-token-usage-analyzer.lock.yml	Compiled workflow reflecting the new import/steps and tightened allowed domains.
.github/workflows/claude-token-optimizer.md	Switches optimizer artifact analysis to structured log JSON over a 7-day window; updates guidance.
.github/workflows/claude-token-optimizer.lock.yml	Compiled workflow reflecting the new import/steps.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 9/9 changed files
Comments generated: 9

Copilot · 2026-04-10T18:12:28Z

.github/workflows/shared/mcp/gh-aw.md

+      # Install gh-aw if not already available
+      if ! gh aw --version >/dev/null 2>&1; then
+        echo "Installing gh-aw extension..."
+        curl -fsSL https://raw.githubusercontent.com/github/gh-aw/refs/heads/main/install-gh-aw.sh | bash


The install step downloads and executes a script from refs/heads/main via curl | bash, which is a supply-chain risk and makes builds non-reproducible. Prefer pinning to a specific commit/tag (or release asset) and verifying integrity (e.g., checksum/signature) before execution.

Suggested change

curl -fsSL https://raw.githubusercontent.com/github/gh-aw/refs/heads/main/install-gh-aw.sh | bash

GH_AW_REPO="https://github.com/github/gh-aw.git"

GH_AW_COMMIT="0123456789abcdef0123456789abcdef01234567"

GH_AW_TMPDIR="$(mktemp -d)"

trap 'rm -rf "$GH_AW_TMPDIR"' EXIT

git clone "$GH_AW_REPO" "$GH_AW_TMPDIR/gh-aw"

git -C "$GH_AW_TMPDIR/gh-aw" checkout "$GH_AW_COMMIT"

gh extension install "$GH_AW_TMPDIR/gh-aw"

Copilot · 2026-04-10T18:12:28Z

.github/workflows/copilot-token-usage-analyzer.md

+        --engine copilot \
+        --start-date -1d \
+        --json \
+        -c 50 \


This step claims to download the last 24 hours of runs, but -c 50 caps the results and can silently omit runs if more than 50 exist in the window. Consider removing the cap or making it explicit in the messaging/logic (and/or using pagination/max-runs appropriate for the period).

Suggested change

-c 50 \

Copilot · 2026-04-10T18:12:28Z

.github/workflows/copilot-token-usage-analyzer.md

+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}


The workflow imports shared/mcp/gh-aw.md, which already sets GH_TOKEN using the GH_AW_* fallback chain, but this step overrides GH_TOKEN to secrets.GITHUB_TOKEN. If the repo relies on a higher-scope token (or runs on GHE with different requirements), this can break gh aw logs. Consider using the same fallback expression here, or omit GH_TOKEN so the component’s choice applies.

Suggested change

env:

GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copilot · 2026-04-10T18:12:29Z

.github/workflows/claude-token-usage-analyzer.md

+      echo "\U0001F4E5 Downloading Claude workflow logs (last 24 hours)..."
+
+      LOGS_EXIT=0
+      gh aw logs \
+        --engine claude \
+        --start-date -1d \
+        --json \
+        -c 50 \
+        > /tmp/gh-aw/token-audit/claude-logs.json || LOGS_EXIT=$?


This step claims to download the last 24 hours of runs, but -c 50 caps the results and can silently omit runs if more than 50 exist in the window. Consider removing the cap or making it explicit in the messaging/logic (and/or using pagination/max-runs appropriate for the period).

Copilot · 2026-04-10T18:12:29Z

.github/workflows/claude-token-usage-analyzer.md

+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}


The workflow imports shared/mcp/gh-aw.md, which already sets GH_TOKEN using the GH_AW_* fallback chain, but this step overrides GH_TOKEN to secrets.GITHUB_TOKEN. If the repo relies on a higher-scope token (or runs on GHE with different requirements), this can break gh aw logs. Consider using the same fallback expression here, or omit GH_TOKEN so the component’s choice applies.

Suggested change

env:

GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copilot · 2026-04-10T18:12:29Z

.github/workflows/copilot-token-optimizer.md

+
+      echo "\U0001F4E5 Downloading Copilot workflow logs (last 7 days)..."
+
+      LOGS_EXIT=0
+      gh aw logs \
+        --engine copilot \
+        --start-date -7d \
+        --json \
+        -c 50 \
+        > /tmp/gh-aw/token-audit/copilot-logs.json || LOGS_EXIT=$?
+
+      if [ -s /tmp/gh-aw/token-audit/copilot-logs.json ]; then
+        TOTAL=$(jq '.runs | length' /tmp/gh-aw/token-audit/copilot-logs.json)
+        echo "\u2705 Downloaded $TOTAL Copilot workflow runs (last 7 days)"


This step claims to download the last 7 days of runs, but -c 50 caps the results and can silently omit runs if more than 50 exist in the window. Consider removing the cap or making it explicit in the messaging/logic (and/or increasing the cap to match a 7-day lookback).

Suggested change

echo "\U0001F4E5 Downloading Copilot workflow logs (last 7 days)..."

LOGS_EXIT=0

gh aw logs \

--engine copilot \

--start-date -7d \

--json \

-c 50 \

> /tmp/gh-aw/token-audit/copilot-logs.json || LOGS_EXIT=$?

if [ -s /tmp/gh-aw/token-audit/copilot-logs.json ]; then

TOTAL=$(jq '.runs | length' /tmp/gh-aw/token-audit/copilot-logs.json)

echo "\u2705 Downloaded $TOTAL Copilot workflow runs (last 7 days)"

MAX_RUNS=50

echo "\U0001F4E5 Downloading Copilot workflow logs (last 7 days, up to $MAX_RUNS runs)..."

LOGS_EXIT=0

gh aw logs \

--engine copilot \

--start-date -7d \

--json \

-c "$MAX_RUNS" \

> /tmp/gh-aw/token-audit/copilot-logs.json || LOGS_EXIT=$?

if [ -s /tmp/gh-aw/token-audit/copilot-logs.json ]; then

TOTAL=$(jq '.runs | length' /tmp/gh-aw/token-audit/copilot-logs.json)

echo "\u2705 Downloaded $TOTAL Copilot workflow runs from the last 7 days (capped at $MAX_RUNS runs)"

if [ "$TOTAL" -ge "$MAX_RUNS" ]; then

echo "\u26a0\ufe0f Result count reached the configured cap of $MAX_RUNS runs; additional runs from the last 7 days may not be included"

fi

Copilot · 2026-04-10T18:12:30Z

.github/workflows/copilot-token-optimizer.md

+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}


The workflow imports shared/mcp/gh-aw.md, which already sets GH_TOKEN using the GH_AW_* fallback chain, but this step overrides GH_TOKEN to secrets.GITHUB_TOKEN. If the repo relies on a higher-scope token (or runs on GHE with different requirements), this can break gh aw logs. Consider using the same fallback expression here, or omit GH_TOKEN so the component’s choice applies.

Suggested change

env:

GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copilot · 2026-04-10T18:12:30Z

.github/workflows/claude-token-optimizer.md

+        --engine claude \
+        --start-date -7d \
+        --json \
+        -c 50 \


This step claims to download the last 7 days of runs, but -c 50 caps the results and can silently omit runs if more than 50 exist in the window. Consider removing the cap or making it explicit in the messaging/logic (and/or increasing the cap to match a 7-day lookback).

Suggested change

-c 50 \

Copilot · 2026-04-10T18:12:30Z

.github/workflows/claude-token-optimizer.md

+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}


The workflow imports shared/mcp/gh-aw.md, which already sets GH_TOKEN using the GH_AW_* fallback chain, but this step overrides GH_TOKEN to secrets.GITHUB_TOKEN. If the repo relies on a higher-scope token (or runs on GHE with different requirements), this can break gh aw logs. Consider using the same fallback expression here, or omit GH_TOKEN so the component’s choice applies.

Suggested change

env:

GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

github-actions · 2026-04-10T18:14:36Z

Smoke Test Results — run 24257149635

✅ GitHub MCP: "fix: token analyzers look for wrong artifact name" / "chore: upgrade gh-aw to v0.68.0 and auto-discover lock files"
✅ Playwright: github.com title contains "GitHub"
✅ File write: /tmp/gh-aw/agent/smoke-test-claude-24257149635.txt created and verified
✅ Bash: file read back successfully

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-04-10T18:15:40Z

Smoke test report
PR titles:

chore: upgrade gh-aw to v0.68.0 and auto-discover lock files
feat: add historical benchmark storage and trend reporting
GitHub MCP merged PR review: ✅
safeinputs-gh PR query: ❌
Playwright title contains "GitHub": ✅
Tavily web search returned results: ❌
File write (/tmp/gh-aw/agent/...): ✅
Bash cat verification: ✅
Discussion query + mystical discussion comment: ❌
Build (npm ci && npm run build): ✅
Overall status: FAIL

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-04-10T18:16:48Z

Security Review

✅ Net security improvement

Removal of *.blob.core.windows.net from network allowlists (claude-token-usage-analyzer.md, copilot-token-usage-analyzer.md, and their .lock.yml files):

 network:
   allowed:
     - github
-    - "*.blob.core.windows.net"

This removes a wildcard Azure blob storage domain from the agent's egress allowlist, reducing the network attack surface. Good change.

⚠️ Supply chain risk: unpinned `curl | bash` in new shared component

File: .github/workflows/shared/mcp/gh-aw.md

curl -fsSL https://raw.githubusercontent.com/github/gh-aw/refs/heads/main/install-gh-aw.sh | bash

This downloads and executes a shell script from a mutable branch reference (refs/heads/main). This is inconsistent with the security posture used everywhere else in these workflows, which pins all GitHub Actions to specific commit SHAs:

# gh-aw-manifest: {...,"actions":[
#   {"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd",...},
#   {"repo":"github/gh-aw-actions/setup","sha":"0acfb4a691fe207cd8bc982ea5cb9d750d57a702",...}
# ]}

Risk: If github/gh-aw's main branch receives a malicious push (compromised maintainer, supply chain attack, accidentally merged PR), the next run of any of the 4 affected workflows would execute arbitrary code on the runner with access to these secrets: COPILOT_GITHUB_TOKEN, GH_AW_GITHUB_MCP_SERVER_TOKEN, GH_AW_GITHUB_TOKEN, GITHUB_TOKEN.

The if ! gh aw --version guard only helps if gh-aw is already installed — in a fresh runner environment this fallback would trigger unconditionally.

Suggested fix: Pin to a specific release tag or commit SHA that matches the version already expected by the workflow (v0.68.0 per the manifest):

# Option 1: Pin to a release tag
curl -fsSL https://raw.githubusercontent.com/github/gh-aw/v0.68.0/install-gh-aw.sh | bash

# Option 2: Pin to a specific commit SHA (more robust)
curl -fsSL https://raw.githubusercontent.com/github/gh-aw/COMMIT_SHA/install-gh-aw.sh | bash

No changes were made to core AWF security components (src/host-iptables.ts, containers/agent/setup-iptables.sh, src/squid-config.ts, src/docker-manager.ts, seccomp profile, or capability configuration). The risk is isolated to the pre-agent CI steps in these 4 token analysis workflows.

Generated by Security Guard for issue #1884 · ● 168.8K · ◷

github-actions · 2026-04-10T18:17:41Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Java Maven required --Dmaven.repo.local workaround as ~/.m2 was owned by root (no sudo available in sandbox).

Generated by Build Test Suite for issue #1884 · ● 610.1K · ◷

lpcox requested a review from Mossaka as a code owner April 10, 2026 18:09

Copilot AI review requested due to automatic review settings April 10, 2026 18:09

Copilot started reviewing on behalf of lpcox April 10, 2026 18:09 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

lpcox mentioned this pull request Apr 10, 2026

📊 cli-proxy token usage impact: preliminary observations #1885

Open

5 tasks

github-actions bot added the smoke-claude label Apr 10, 2026

github-actions bot added the build-test label Apr 10, 2026

lpcox merged commit 377d62d into main Apr 10, 2026
61 of 64 checks passed

lpcox deleted the refactor/token-workflows-use-gh-aw-logs branch April 10, 2026 18:31

-        curl -fsSL https://raw.githubusercontent.com/github/gh-aw/refs/heads/main/install-gh-aw.sh | bash
+        GH_AW_REPO="https://github.com/github/gh-aw.git"
+        GH_AW_COMMIT="0123456789abcdef0123456789abcdef01234567"
+        GH_AW_TMPDIR="$(mktemp -d)"
+        trap 'rm -rf "$GH_AW_TMPDIR"' EXIT
+        git clone "$GH_AW_REPO" "$GH_AW_TMPDIR/gh-aw"
+        git -C "$GH_AW_TMPDIR/gh-aw" checkout "$GH_AW_COMMIT"
+        gh extension install "$GH_AW_TMPDIR/gh-aw"

Conversation

lpcox commented Apr 10, 2026

Summary

Motivation

Changes

New shared component

Token Usage Analyzers (Copilot + Claude)

Token Optimizers (Copilot + Claude)

Benefits

Uh oh!

github-actions bot commented Apr 10, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 10, 2026

Uh oh!

github-actions bot commented Apr 10, 2026

Uh oh!

github-actions bot commented Apr 10, 2026

Security Review

✅ Net security improvement

⚠️ Supply chain risk: unpinned curl | bash in new shared component

Uh oh!

github-actions bot commented Apr 10, 2026

🏗️ Build Test Suite Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

⚠️ Supply chain risk: unpinned `curl | bash` in new shared component