Skip to content

README.md

Latest commit

 

History

History
429 lines (319 loc) · 14 KB

README.md

File metadata and controls

429 lines (319 loc) · 14 KB

ECR

Capabilities

Supports many LLM providers

Anthropic, DeepSeek, Gemini, Llama, OpenAI

Github Action

You can run as a GitHub Action to have it as part of your CI/CD workflow. https://github.com/marketplace/actions/ethereum-security-reviewer

Create a workflow in your repository, e.g., .github/workflows/security-review.yml, use the security-review.yml.example file as an example.

Add your API key(s) and other environmental variables to your repository secrets:

  • Go to your repository settings
  • Go to Secrets and Variables > Actions
  • Create a new Repository Secret, with the Name ANTHROPIC_API_KEY and the Secret your Anthropic API key.
  • Create a PR to see it in action. Example output: https://github.com/fredrik0x/go-ethereum/pull/2

Quick Demo

To view how the tool performs in standalone mode, you can build a docker image.

Create a GitHub Personal Access Token (Classic) with read:repo_hook checked: https://github.com/settings/tokens

Create an Anthropic API key: https://console.anthropic.com/settings/keys

docker build -t ethereum-code-reviewer:latest .
docker run --rm ethereum-code-reviewer:latest https://github.com/bluealloy/revm/pull/2085 --llm-provider anthropic --anthropic-api-key <anthropic-api-key> --github-token <github-pat-classic>

Output should look similar to:

Analyzing PR #2085
Repository: bluealloy/revm
Initial analysis found 3 potential vulnerabilities. Performing verification...

🛡️ Security Review Report
Vulnerabilities Detected: Yes

Summary:
After verification: 3 confirmed vulnerabilities out of 3 initially found.

Cost Information: $0.108381 (25052 input + 2215 output tokens, claude-sonnet-4-5-20250929)

Detailed Findings:

HIGH Severity Issue
Description: G1_MSM_BASE_GAS_FEE has been changed from 12000 to 1200, which is a 10x reduction in gas cost for G1 multi-scalar multiplication operations
Recommendation: Revert G1_MSM_BASE_GAS_FEE to 12000 to match EIP-2537 specification
Confidence: 95%

HIGH Severity Issue
Description: MAP_FP2_TO_G2_BASE_GAS_FEE is set to 0x23800 (145408) instead of the expected 23800, representing a ~6x increase in gas cost
Recommendation: Change MAP_FP2_TO_G2_BASE_GAS_FEE from 0x23800 to 23800 to match EIP-2537 specification
Confidence: 90%

MEDIUM Severity Issue
Description: G2_MSM uses incorrect constants - G2_ADD_ADDRESS, G2_ADD_BASE_GAS_FEE, and G2_ADD_INPUT_LENGTH instead of G2_MSM specific constants
Recommendation: Use correct G2_MSM constants instead of G2_ADD constants in g2_msm.rs
Confidence: 85%
::warning::Security vulnerabilities detected with 95% confidence

Dependencies

Python 3.8 or higher

pip install -e .

Environment Setup

The tool uses flags or environment variables for API keys and configuration. The system prompts are adjusted as environment variables.

  1. Using a .env file:

    # Copy the example env file
cp .env.example .env

# Edit .env with your API keys
GITHUB_TOKEN=your_github_token_here
ANTHROPIC_API_KEY=your_anthropic_key_here
OPENAI_API_KEY=your_openai_key_here
VOYAGE_API_KEY=your_voyage_key_here  # Required for Anthropic with docs
...

  2. Setting environment variables directly:

    export GITHUB_TOKEN=your_github_token_here
export ANTHROPIC_API_KEY=your_anthropic_key_here
export OPENAI_API_KEY=your_openai_key_here
export VOYAGE_API_KEY=your_voyage_key_here
...

The tool will automatically load variables from .env if present, falling back to system environment variables or flags.

Multi-judging

If more than one provider is enabled, weighted multi-judging is enabled with voting between providers. The multi-judge mode:

  • Runs analysis with all available LLMs in parallel
  • Uses weighted voting to determine if vulnerabilities exist
  • If vulnerabilities are detected by consensus, uses Anthropic to synthesize a unified report
  • Provides higher confidence through cross-validation between models
  • Is automatically enabled when using multiple providers (one must be Anthropic)

You can set the weight of each provider by using these variables: ANTHROPIC_WEIGHT, OPENAI_WEIGHT, GEMINI_WEIGHT, DEEPSEEK_WEIGHT, LLAMA_WEIGHT

Using Historical Vulnerability or Specification Context

The tool supports using historical vulnerability documentation to improve its analysis through RAG. The embedding providers are:

  • OpenAI: Uses text-embedding-3-small model, automatically used with OpenAI's GPT.
  • Voyage AI: Required when using Anthropic's Claude with documentation.
  1. Create a directory for your vulnerability documentation:
mkdir docs
  1. Add your documentation files (supports .md files):
docs/
  ├── vuln1.md
  ├── vuln2.md
  └── vuln3.md
  1. Use the documentation in the analysis:
# Using Anthropic (requires Voyage AI key)
python -m pr_security_review https://github.com/org/repo/pull/1 \
  --llm-provider anthropic \
  --docs-dir ./docs \
  --voyage-api-key your_voyage_api_key

# Using OpenAI (automatically uses OpenAI embeddings)
python -m pr_security_review https://github.com/org/repo/pull/1 \
  --llm-provider openai \
  --docs-dir ./docs

This will then:

  1. Generate embeddings for your documentation (cached for reuse)
  2. Find relevant context for each PR analysis using vector similarity
  3. Include this context in the security review

CLI

One-Shot examples using Anthropic

Single PR Review

# Using environment variables
export GITHUB_TOKEN=your_github_personal_access_token
export ANTHROPIC_API_KEY=your_anthropic_key
python -m pr_security_review https://github.com/org/repo/pull/1

# Optionally post results as a PR comment
python -m pr_security_review https://github.com/org/repo/pull/1 --post-comment

# With vulnerability documentation (requires Voyage AI)
export VOYAGE_API_KEY=your_voyage_key
python -m pr_security_review https://github.com/org/repo/pull/1 \
  --docs-dir ./docs

# Or using command line arguments
python -m pr_security_review https://github.com/org/repo/pull/1 \
  --github-token your_github_token \
  --llm-provider anthropic \
  --anthropic-api-key your_anthropic_key \
  --model claude-sonnet-4-5-20250929 \  # optional
  --docs-dir ./docs \   # optional
  --voyage-api-key your_voyage_key        # required if using docs-dir

Analyzing Recent PRs

You can also analyze the last X PRs from a repository to get a broader security overview:

# Analyze the last 10 PRs from a repository (default)
python -m pr_security_review --recent-prs owner/repo

# Analyze the last 5 PRs from a repository
python -m pr_security_review --recent-prs owner/repo --pr-count 5

# With specific LLM provider and API key
python -m pr_security_review --recent-prs owner/repo \
  --pr-count 15 \
  --llm-provider openai \
  --openai-api-key YOUR_KEY \
  --github-token YOUR_GITHUB_TOKEN

This feature is particularly useful for:

  • Security audits of recent changes
  • Identifying patterns in security issues across PRs
  • Bulk analysis of repositories
  • Getting a security overview before releases

The tool will:

  1. Fetch the last X PRs from the repository (both open and closed)
  2. Analyze each PR with code changes for security vulnerabilities
  3. Provide a summary report showing which PRs have security issues
  4. Display total cost information for the batch analysis

Single File Analysis

You can analyze individual files:

# Analyze a specific file
python -m pr_security_review --file https://github.com/org/repo/blob/main/src/auth.py

Single Commit Analysis

You can also analyze individual commits:

# Analyze a specific commit
python -m pr_security_review --repository NethermindEth/Nethermind --analyze-commit <SHA>

Monitor

You can use the commit monitoring feature that track repositories for new commits and automatically analyze them for security vulnerabilities.

Add repositories manually
# Add a repository to monitor (e.g., Nethermind's master branch)
python -m pr_security_review --monitor-add https://github.com/NethermindEth/nethermind --monitor-branches master

# Check for new commits once
python -m pr_security_review --monitor-check

# Start continuous monitoring (checks every 5 minutes by default)
python -m pr_security_review --monitor-continuous

Common Commands

# List all monitored repositories
python -m pr_security_review --monitor-list

# Remove a repository from monitoring
python -m pr_security_review --monitor-remove https://github.com/NethermindEth/nethermind

# Analyze a specific commit
python -m pr_security_review --analyze-commit abc123 --repository owner/repo

# Continuous monitoring with custom interval (in seconds)
python -m pr_security_review --monitor-continuous --monitor-interval 600

Message Queue

You can use a supported AMQP queue (such as rabbitmq) to receive incoming review requests, and to submit results. Use the AMQP_URL, QUEUE_NAME and RESPONSE_QUEUE_NAME to set this up.

GitHub Application

You can also run the tool as a GitHub App, which allows for command-based triggering of security reviews on PRs. Example: https://github.com/fredrik0x/go-ethereum/pull/1

  1. Create a new GitHub App:

    • Go to your GitHub Settings > Developer settings > GitHub Apps
    • Click "New GitHub App"
    • Fill in the required information:
      • Name: Choose a name for your app
      • Homepage URL: Your repository URL
      • Webhook URL: The URL where your app will be running
      • Webhook secret: Generate a secure random string

  2. Set Repository Permissions:

    • Pull requests: Read & write (for reading PR changes and posting comments)
    • Repository contents: Read (for accessing code)
    • Repository collaborators: Read (for checking user permissions)

  3. Subscribe to Events:

    • Pull request (for automatic analysis of new PRs)
    • Issue comment (for command-based triggering)

  4. After creating the app:

    • Note down the App ID
    • Generate and download a private key
    • Install the app on your repositories

  5. Run the app as a webhook server (or use Dockerfile.github):

python -m pr_security_review \
  --github-app \
  --github-app-id YOUR_APP_ID \
  --github-private-key-path path/to/private-key.pem \
  --github-webhook-secret YOUR_WEBHOOK_SECRET \
  --llm-provider openai \  # or anthropic
  --openai-api-key YOUR_OPENAI_KEY \  # or --anthropic-api-key for Anthropic
  --docs-dir ./docs \  # optional
  --voyage-api-key YOUR_VOYAGE_KEY  # required if using Anthropic with docs

The app will now:

  • Automatically analyze new PRs and PR updates
  • Allow repository admins/maintainers to trigger analysis by commenting /security-review on any PR

Web Application

To run the application in web server mode, you must at a minimum set the following flags or environment variables:

GITHUB_TOKEN=
ANTHROPIC_API_KEY=
GOOGLE_CLIENT_ID=
FLASK_SECRET_KEY=
AUTHORIZED_EMAILS=the.google.email.address.you.want.to.be.admin
WEB_APP_PORT=
BASE_URL=

Email Notifications

You can receive an email whenever a new finding is found by setting the AWS SES environmental variables.

Telegram Notifications

You can receive telegram notifications when a review has found a potential vulnerability (or all reviews)

# Configure Telegram notifications in the .env file with findings web server
TELEGRAM_BOT_TOKEN=your_telegram_bot_token
TELEGRAM_CHAT_ID=your_telegram_chat_id
FINDINGS_SERVER_URL=https://your-server-url.com  # Optional: defaults to localhost if not set
FINDINGS_SERVER_PORT=8000  # Optional: defaults to 8000
NOTIFY_CLEAN_COMMITS=true  # Optional: notify for clean commits with no vulnerabilities

# Start monitoring with Telegram notifications
python -m pr_security_review --monitor-continuous

This will:

  1. Start a findings server that hosts detailed vulnerability reports
  2. Send concise summaries to Telegram with links to detailed reports
  3. Each link contains comprehensive information about the vulnerability, including:
    • Detailed explanations of the issue
    • Impact analysis
    • Fix recommendations with code examples
    • Additional resources when available
  4. Enable Telegram bot commands:
    • /lastcommits - Shows the latest commits for each monitored repository
    • /listrepos - Lists all repositories currently being monitored
  5. Optionally sends notifications for clean commits (commits with no security vulnerabilities)

The /lastcommits command provides a quick overview of the most recent commit in each branch of monitored repositories, including:

  • Repository name and branch
  • Commit hash with link to GitHub
  • Author name
  • Commit date and time
  • Commit message snippet

The /listrepos command shows a simple list of all repositories being monitored, including:

  • Repository name
  • Monitored branches

When the --notify-clean-commits flag is enabled (or NOTIFY_CLEAN_COMMITS=true environment variable is set), you'll also receive notifications for commits that have been analyzed and found to have no security vulnerabilities. These "clean commit" notifications include:

  • Repository name and branch
  • Commit hash with link to GitHub
  • Author name
  • Commit message

Database Management

You can use this database utility to managing your security findings:

# Check database connection
python -m pr_security_review.db_utils check

# View database statistics
python -m pr_security_review.db_utils stats

# Clean up expired findings
python -m pr_security_review.db_utils cleanup

# List recent findings for a repository
python -m pr_security_review.db_utils list --repo ethereum/go-ethereum

# View details of a specific finding
python -m pr_security_review.db_utils details <uuid>

License

MIT License - see LICENSE file for details.