Skip to content

Commit 7b35b59

Browse files
sbryngelsonclaude
andauthored
Add CI lint gate and local precheck command (#1122)
* Add CI lint gate and local precheck command - Add lint-gate job to test.yml that runs fast checks (formatting, spelling, linting) before expensive test matrix and HPC jobs start - Add concurrency groups to test.yml, coverage.yml, cleanliness.yml, and bench.yml to cancel superseded runs on new pushes - Add ./mfc.sh precheck command for local CI validation before pushing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix precheck.sh portability and usability issues - Add cross-platform hash function (macOS uses md5, Linux uses md5sum) - Validate -j/--jobs argument (require value, must be numeric) - Improve error messages with actionable guidance - Clarify that formatting has been auto-applied when check fails Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Gate benchmarks on Test Suite completion Add wait-for-tests job that polls GitHub API to ensure: - Lint Gate passes first (fast fail) - All Github test jobs complete successfully - Only then do benchmark jobs start This prevents wasting HPC resources on benchmarking code that fails tests, while preserving the existing maintainer approval gate. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Auto-install git pre-commit hook for precheck - Add .githooks/pre-commit that runs ./mfc.sh precheck before commits - Auto-install hook on first ./mfc.sh invocation (symlinks to .git/hooks/) - Hook only installs once; subsequent runs skip if already present - Developers can bypass with: git commit --no-verify Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Use dynamic CPU count in pre-commit hook Auto-detect available CPUs for parallel formatting: - Linux: nproc - macOS: sysctl -n hw.ncpu - Fallback: 4 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Show CPU count in pre-commit hook output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add precheck command to CLI and autocomplete Register precheck in commands.py so it appears in: - Shell tab completion - CLI documentation - ./mfc.sh precheck --help Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Auto-update installed shell completions on regeneration When completion scripts are auto-regenerated, also update the installed completions at ~/.local/share/mfc/completions/ if they exist. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Show source command when completions auto-update When installed shell completions are auto-updated, print a message with the appropriate source command for the user's detected shell. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Always check installed completions on every run Previously, installed completions only updated when source files changed and regeneration occurred. Now we also check if the installed completions are older than the generated ones (e.g., after git pull brings new pre-generated completions). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Prevent directory completion fallback in shell completions - Remove -o bashdefault from bash complete command to prevent falling back to directory completion when no matches found - Add explicit : (no-op) for zsh commands without arguments to prevent default file/directory completion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Auto-install completions and fix bash completion options - Auto-install completions on first mfc.sh run (via main.py) - Add -o filenames back to bash complete (needed for file completion) - Keep -o bashdefault removed to prevent directory fallback - Simplify code by having __update_installed_completions handle both install and update cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Auto-install completions from mfc.sh with shell rc setup Move completion auto-install to mfc.sh so it runs for ALL commands including help, precheck, etc. This ensures completions are always set up on first run. - Install completion files to ~/.local/share/mfc/completions/ - Add source line to .bashrc or fpath to .zshrc - Tell user to restart shell or source the file - main.py now only handles updates when generated files change Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Clarify verbose, debug, and debug-log flag documentation - -v/-vv/-vvv: output verbosity levels - --debug: build with debug compiler flags (for MFC Fortran code) - --debug-log/-d: Python toolchain debug logging (not MFC code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Cap pre-commit hook parallelism at 12 jobs Avoid hogging resources on machines with many cores. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 1d3286a commit 7b35b59

11 files changed

Lines changed: 394 additions & 21 deletions

File tree

.githooks/pre-commit

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
3+
# MFC pre-commit hook
4+
# Runs ./mfc.sh precheck before allowing commits
5+
# Bypass with: git commit --no-verify
6+
7+
# Only run if we're in the MFC repo root (where mfc.sh exists)
8+
if [ ! -f "$(git rev-parse --show-toplevel)/mfc.sh" ]; then
9+
exit 0
10+
fi
11+
12+
cd "$(git rev-parse --show-toplevel)"
13+
14+
# Auto-detect CPU count (capped at 12 to avoid hogging resources)
15+
JOBS=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
16+
[ "$JOBS" -gt 12 ] && JOBS=12
17+
18+
echo ""
19+
echo "mfc: Running precheck before commit (-j $JOBS)..."
20+
echo ""
21+
22+
if ./mfc.sh precheck -j "$JOBS"; then
23+
echo ""
24+
exit 0
25+
else
26+
echo ""
27+
echo "mfc: Commit blocked. Fix issues above or use 'git commit --no-verify' to bypass."
28+
echo ""
29+
exit 1
30+
fi

.github/workflows/bench.yml

Lines changed: 59 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,15 @@ on:
66
types: [submitted]
77
workflow_dispatch:
88

9+
concurrency:
10+
group: ${{ github.workflow }}-${{ github.ref }}
11+
cancel-in-progress: true
12+
913
jobs:
1014
file-changes:
1115
name: Detect File Changes
1216
runs-on: 'ubuntu-latest'
13-
outputs:
17+
outputs:
1418
checkall: ${{ steps.changes.outputs.checkall }}
1519
steps:
1620
- name: Clone
@@ -19,13 +23,65 @@ jobs:
1923
- name: Detect Changes
2024
uses: dorny/paths-filter@v3
2125
id: changes
22-
with:
26+
with:
2327
filters: ".github/file-filter.yml"
2428

29+
wait-for-tests:
30+
name: Wait for Test Suite
31+
runs-on: ubuntu-latest
32+
steps:
33+
- name: Wait for Test Suite to Pass
34+
env:
35+
GH_TOKEN: ${{ github.token }}
36+
run: |
37+
echo "Waiting for Test Suite workflow to complete..."
38+
SHA="${{ github.event.pull_request.head.sha || github.sha }}"
39+
40+
# Poll every 60 seconds for up to 3 hours
41+
for i in $(seq 1 180); do
42+
# Get the Test Suite workflow runs for this commit
43+
STATUS=$(gh api repos/${{ github.repository }}/commits/$SHA/check-runs \
44+
--jq '.check_runs[] | select(.name == "Lint Gate") | .conclusion' | head -1)
45+
46+
if [ "$STATUS" = "success" ]; then
47+
echo "Lint Gate passed. Checking test jobs..."
48+
49+
# Check if any Github test jobs failed
50+
FAILED=$(gh api repos/${{ github.repository }}/commits/$SHA/check-runs \
51+
--jq '[.check_runs[] | select(.name | startswith("Github")) | select(.conclusion == "failure")] | length')
52+
53+
if [ "$FAILED" != "0" ]; then
54+
echo "::error::Test Suite has failing jobs. Benchmarks will not run."
55+
exit 1
56+
fi
57+
58+
# Check if Github tests are still running
59+
PENDING=$(gh api repos/${{ github.repository }}/commits/$SHA/check-runs \
60+
--jq '[.check_runs[] | select(.name | startswith("Github")) | select(.conclusion == null)] | length')
61+
62+
if [ "$PENDING" = "0" ]; then
63+
echo "All Test Suite jobs completed successfully!"
64+
exit 0
65+
fi
66+
67+
echo "Tests still running ($PENDING pending)..."
68+
elif [ "$STATUS" = "failure" ]; then
69+
echo "::error::Lint Gate failed. Benchmarks will not run."
70+
exit 1
71+
else
72+
echo "Lint Gate status: ${STATUS:-pending}..."
73+
fi
74+
75+
sleep 60
76+
done
77+
78+
echo "::error::Timeout waiting for Test Suite to complete."
79+
exit 1
80+
2581
self:
2682
name: "${{ matrix.name }} (${{ matrix.device }}${{ matrix.interface != 'none' && format('-{0}', matrix.interface) || '' }})"
2783
if: ${{ github.repository=='MFlowCode/MFC' && needs.file-changes.outputs.checkall=='true' && ((github.event_name=='pull_request_review' && github.event.review.state=='approved') || (github.event_name=='pull_request' && (github.event.pull_request.user.login=='sbryngelson' || github.event.pull_request.user.login=='wilfonba'))) }}
28-
needs: file-changes
84+
needs: [file-changes, wait-for-tests]
2985
strategy:
3086
fail-fast: false
3187
matrix:

.github/workflows/cleanliness.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@ name: Cleanliness
22

33
on: [push, pull_request, workflow_dispatch]
44

5+
concurrency:
6+
group: ${{ github.workflow }}-${{ github.ref }}
7+
cancel-in-progress: true
8+
59
jobs:
610
file-changes:
711
name: Detect File Changes

.github/workflows/coverage.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@ name: Coverage Check
22

33
on: [push, pull_request, workflow_dispatch]
44

5+
concurrency:
6+
group: ${{ github.workflow }}-${{ github.ref }}
7+
cancel-in-progress: true
8+
59
jobs:
610
file-changes:
711
name: Detect File Changes

.github/workflows/test.yml

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,50 @@
11
name: 'Test Suite'
22

33
on: [push, pull_request, workflow_dispatch]
4-
4+
5+
concurrency:
6+
group: ${{ github.workflow }}-${{ github.ref }}
7+
cancel-in-progress: true
8+
59
jobs:
10+
lint-gate:
11+
name: Lint Gate
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Clone
15+
uses: actions/checkout@v4
16+
17+
- name: Setup Python
18+
uses: actions/setup-python@v5
19+
with:
20+
python-version: '3.12'
21+
22+
- name: Initialize MFC
23+
run: ./mfc.sh init
24+
25+
- name: Check Formatting
26+
run: |
27+
./mfc.sh format -j $(nproc)
28+
git diff --exit-code || (echo "::error::Code is not formatted. Run './mfc.sh format' locally." && exit 1)
29+
30+
- name: Spell Check
31+
run: ./mfc.sh spelling
32+
33+
- name: Lint Toolchain
34+
run: ./mfc.sh lint
35+
36+
- name: Lint Source - No Raw Directives
37+
run: |
38+
! grep -iR '!\$acc\|!\$omp' --exclude="parallel_macros.fpp" --exclude="acc_macros.fpp" --exclude="omp_macros.fpp" --exclude="shared_parallel_macros.fpp" --exclude="syscheck.fpp" ./src/*
39+
40+
- name: Lint Source - No Double Precision Intrinsics
41+
run: |
42+
! grep -iR 'double_precision\|dsqrt\|dexp\|dlog\|dble\|dabs\|double\ precision\|real(8)\|real(4)\|dprod\|dmin\|dmax\|dfloat\|dreal\|dcos\|dsin\|dtan\|dsign\|dtanh\|dsinh\|dcosh\|d0' --exclude-dir=syscheck --exclude="*nvtx*" --exclude="*precision_select*" ./src/*
43+
44+
- name: Lint Source - No Junk Code
45+
run: |
46+
! grep -iR -e '\.\.\.' -e '\-\-\-' -e '===' ./src/*
47+
648
file-changes:
749
name: Detect File Changes
850
runs-on: 'ubuntu-latest'
@@ -21,7 +63,7 @@ jobs:
2163
github:
2264
name: Github
2365
if: needs.file-changes.outputs.checkall == 'true'
24-
needs: file-changes
66+
needs: [lint-gate, file-changes]
2567
strategy:
2668
matrix:
2769
os: ['ubuntu', 'macos']
@@ -95,7 +137,7 @@ jobs:
95137
self:
96138
name: "${{ matrix.cluster_name }} (${{ matrix.device }}${{ matrix.interface != 'none' && format('-{0}', matrix.interface) || '' }})"
97139
if: github.repository == 'MFlowCode/MFC' && needs.file-changes.outputs.checkall == 'true'
98-
needs: file-changes
140+
needs: [lint-gate, file-changes]
99141
continue-on-error: false
100142
timeout-minutes: 480
101143
strategy:

mfc.sh

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,39 @@ fi
1010
# Load utility script
1111
. "$(pwd)/toolchain/util.sh"
1212

13+
# Auto-install git pre-commit hook (once, silently)
14+
if [ -d "$(pwd)/.git" ] && [ ! -e "$(pwd)/.git/hooks/pre-commit" ] && [ -f "$(pwd)/.githooks/pre-commit" ]; then
15+
ln -sf "$(pwd)/.githooks/pre-commit" "$(pwd)/.git/hooks/pre-commit"
16+
log "Installed git pre-commit hook (runs$MAGENTA ./mfc.sh precheck$COLOR_RESET before commits)."
17+
fi
18+
19+
# Auto-install shell completions (once)
20+
COMPLETION_DIR="$HOME/.local/share/mfc/completions"
21+
if [ ! -d "$COMPLETION_DIR" ]; then
22+
mkdir -p "$COMPLETION_DIR"
23+
cp "$(pwd)/toolchain/completions/mfc.bash" "$COMPLETION_DIR/"
24+
cp "$(pwd)/toolchain/completions/_mfc" "$COMPLETION_DIR/"
25+
26+
# Add to shell rc file based on current shell
27+
if [[ "$SHELL" == *"zsh"* ]]; then
28+
RC_FILE="$HOME/.zshrc"
29+
RC_LINE="fpath=(\"$COMPLETION_DIR\" \$fpath)"
30+
SOURCE_CMD="source $COMPLETION_DIR/_mfc"
31+
else
32+
RC_FILE="$HOME/.bashrc"
33+
RC_LINE="[ -f \"$COMPLETION_DIR/mfc.bash\" ] && source \"$COMPLETION_DIR/mfc.bash\""
34+
SOURCE_CMD="source $COMPLETION_DIR/mfc.bash"
35+
fi
36+
37+
if [ -f "$RC_FILE" ] && ! grep -q "$COMPLETION_DIR" "$RC_FILE" 2>/dev/null; then
38+
echo "" >> "$RC_FILE"
39+
echo "# MFC shell completion" >> "$RC_FILE"
40+
echo "$RC_LINE" >> "$RC_FILE"
41+
fi
42+
43+
log "Installed tab completions. Restart shell or run:$MAGENTA $SOURCE_CMD$COLOR_RESET"
44+
fi
45+
1346
# Print startup message immediately for user feedback
1447
log "Starting..."
1548

@@ -56,6 +89,10 @@ elif [ "$1" '==' "spelling" ] && [ "$2" != "--help" ] && [ "$2" != "-h" ]; then
5689
. "$(pwd)/toolchain/bootstrap/python.sh"
5790

5891
shift; . "$(pwd)/toolchain/bootstrap/spelling.sh" $@; exit 0
92+
elif [ "$1" '==' "precheck" ]; then
93+
. "$(pwd)/toolchain/bootstrap/python.sh"
94+
95+
shift; . "$(pwd)/toolchain/bootstrap/precheck.sh" $@; exit 0
5996
fi
6097

6198
mkdir -p "$(pwd)/build"

toolchain/bootstrap/precheck.sh

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
#!/bin/bash
2+
set -e
3+
set -o pipefail
4+
5+
# Function to display help message
6+
show_help() {
7+
echo "Usage: ./mfc.sh precheck [OPTIONS]"
8+
echo "Run the same fast checks that CI runs before expensive tests start."
9+
echo "Use this locally before pushing to catch issues early."
10+
echo ""
11+
echo "Options:"
12+
echo " -h, --help Display this help message and exit."
13+
echo " -j, --jobs JOBS Runs JOBS number of parallel jobs for formatting."
14+
echo ""
15+
exit 0
16+
}
17+
18+
# Cross-platform hash function (macOS uses md5, Linux uses md5sum)
19+
compute_hash() {
20+
if command -v md5sum > /dev/null 2>&1; then
21+
md5sum | cut -d' ' -f1
22+
elif command -v md5 > /dev/null 2>&1; then
23+
md5 -q
24+
else
25+
# Fallback: use cksum if neither available
26+
cksum | cut -d' ' -f1
27+
fi
28+
}
29+
30+
JOBS=1
31+
32+
while [[ $# -gt 0 ]]; do
33+
case "$1" in
34+
-j|--jobs)
35+
if [[ -z "$2" || "$2" == -* ]]; then
36+
echo "Precheck: -j/--jobs requires a value."
37+
exit 1
38+
fi
39+
if ! [[ "$2" =~ ^[0-9]+$ ]]; then
40+
echo "Precheck: jobs value '$2' is not a valid number."
41+
exit 1
42+
fi
43+
JOBS="$2"
44+
shift
45+
;;
46+
-h | --help)
47+
show_help
48+
;;
49+
*)
50+
echo "Precheck: unknown argument: $1."
51+
exit 1
52+
;;
53+
esac
54+
55+
shift
56+
done
57+
58+
FAILED=0
59+
60+
log "Running$MAGENTA precheck$COLOR_RESET (same checks as CI lint-gate)..."
61+
echo ""
62+
63+
# 1. Check formatting
64+
log "[$CYAN 1/4$COLOR_RESET] Checking$MAGENTA formatting$COLOR_RESET..."
65+
# Capture state before formatting
66+
BEFORE_HASH=$(git diff -- '*.f90' '*.fpp' '*.py' 2>/dev/null | compute_hash)
67+
if ! ./mfc.sh format -j "$JOBS" > /dev/null 2>&1; then
68+
error "Formatting check failed to run."
69+
FAILED=1
70+
else
71+
# Check if formatting changed any Fortran/Python files
72+
AFTER_HASH=$(git diff -- '*.f90' '*.fpp' '*.py' 2>/dev/null | compute_hash)
73+
if [ "$BEFORE_HASH" != "$AFTER_HASH" ]; then
74+
error "Code was not formatted. Files have been auto-formatted; review and stage the changes."
75+
echo ""
76+
git diff --stat -- '*.f90' '*.fpp' '*.py' 2>/dev/null || true
77+
echo ""
78+
FAILED=1
79+
else
80+
ok "Formatting check passed."
81+
fi
82+
fi
83+
84+
# 2. Spell check
85+
log "[$CYAN 2/4$COLOR_RESET] Running$MAGENTA spell check$COLOR_RESET..."
86+
if ./mfc.sh spelling > /dev/null 2>&1; then
87+
ok "Spell check passed."
88+
else
89+
error "Spell check failed. Run$MAGENTA ./mfc.sh spelling$COLOR_RESET for details."
90+
FAILED=1
91+
fi
92+
93+
# 3. Lint toolchain (Python)
94+
log "[$CYAN 3/4$COLOR_RESET] Running$MAGENTA toolchain lint$COLOR_RESET..."
95+
if ./mfc.sh lint > /dev/null 2>&1; then
96+
ok "Toolchain lint passed."
97+
else
98+
error "Toolchain lint failed. Run$MAGENTA ./mfc.sh lint$COLOR_RESET for details."
99+
FAILED=1
100+
fi
101+
102+
# 4. Source code lint checks
103+
log "[$CYAN 4/4$COLOR_RESET] Running$MAGENTA source lint$COLOR_RESET checks..."
104+
SOURCE_FAILED=0
105+
106+
# Check for raw OpenACC/OpenMP directives
107+
if grep -qiR '!\$acc\|!\$omp' --exclude="parallel_macros.fpp" --exclude="acc_macros.fpp" --exclude="omp_macros.fpp" --exclude="shared_parallel_macros.fpp" --exclude="syscheck.fpp" ./src/* 2>/dev/null; then
108+
error "Found raw OpenACC/OpenMP directives. Use macros instead."
109+
SOURCE_FAILED=1
110+
fi
111+
112+
# Check for double precision intrinsics
113+
if grep -qiR 'double_precision\|dsqrt\|dexp\|dlog\|dble\|dabs\|double\ precision\|real(8)\|real(4)\|dprod\|dmin\|dmax\|dfloat\|dreal\|dcos\|dsin\|dtan\|dsign\|dtanh\|dsinh\|dcosh\|d0' --exclude-dir=syscheck --exclude="*nvtx*" --exclude="*precision_select*" ./src/* 2>/dev/null; then
114+
error "Found double precision intrinsics. Use generic intrinsics."
115+
SOURCE_FAILED=1
116+
fi
117+
118+
# Check for junk code patterns
119+
if grep -qiR -e '\.\.\.' -e '\-\-\-' -e '===' ./src/* 2>/dev/null; then
120+
error "Found junk code patterns (..., ---, ===) in source."
121+
SOURCE_FAILED=1
122+
fi
123+
124+
if [ $SOURCE_FAILED -eq 0 ]; then
125+
ok "Source lint passed."
126+
else
127+
FAILED=1
128+
fi
129+
130+
echo ""
131+
132+
if [ $FAILED -eq 0 ]; then
133+
ok "All precheck tests passed! Safe to push."
134+
exit 0
135+
else
136+
error "Some precheck tests failed. Fix issues before pushing."
137+
exit 1
138+
fi

0 commit comments

Comments
 (0)