Skip to content

Reduce lint hot-path overhead#6826

Draft
Brett-Best wants to merge 28 commits into
realm:mainfrom
Brett-Best:codex/lint-hot-path-performance
Draft

Reduce lint hot-path overhead#6826
Brett-Best wants to merge 28 commits into
realm:mainfrom
Brett-Best:codex/lint-hot-path-performance

Conversation

@Brett-Best

@Brett-Best Brett-Best commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This PR reduces lint and correction hot-path overhead across rule execution, syntax classification, file caching, and parallel scheduling. In SwiftLint's OSS-check suite, the combined series improves end-to-end lint time by 9–35%.

  • move the remaining lint-mode rules and SwiftSyntax-mode custom rules off SourceKit by deriving their structure and syntax-kind information from SwiftSyntax
  • skip syntax-tree rewriting for correctable rules when a file has no violations
  • replace repeated full-tree and classification work with targeted scans, cheap pre-checks, and per-file caches
  • remove repeated per-token, per-node, per-rule, and per-file invariant work across frequently executed rules
  • reduce cache synchronization and violation post-processing overhead
  • use file-level parallelism more effectively by running rules serially per file when appropriate and dispatching the largest files first
  • fix sorted-index and SourceKit-compatible comment-range behavior

Motivation

Profiling a large repository showed that a substantial portion of runtime was spent outside rule-specific analysis: SourceKit editorOpen requests, repeated syntax-tree walks, classification of unrelated tokens, full-tree correction rewrites on clean files, filesystem and cache synchronization, repeated source-line and term-list construction, task-local bookkeeping, violation-region partitioning, and concurrent walks contending on the same syntax arena.

Most of this work is invariant for a rule type or file, can be cached, or can be avoided entirely after a cheap candidate check. This series moves invariant work out of repeated paths and makes expensive analysis conditional on a file having relevant content.

Details

Remove unnecessary SourceKit work

  • port indentation_width, literal_expression_end_indentation, multiline_function_chains, multiline_parameters_brackets, statement_position, and the vertical-whitespace brace rules to SwiftSyntax-backed implementations
  • add a shared SwiftSyntax-derived syntax map for rules that still need syntax-kind filtering
  • execute custom rules configured for SwiftSyntax mode without consulting SourceKit, including inherited default_execution_mode
  • preserve SourceKit-compatible line-comment ranges for LF, CRLF, and CR line endings without extending block comments

Avoid unnecessary correction and rule work

  • validate correctable rules first and skip rewriter construction and serialization when a file has no violations
  • skip adjusted body-line analysis when the physical body span is already within the configured threshold
  • gate comment_spacing regex work behind a cheap UTF-16 scan and match directly against the file's string view
  • hoist per-file invariants and add cheap preconditions in colon, comma, closure_end_indentation, and redundant_self
  • skip violation-region partitioning and identifier-set construction when no violations exist or the file has one enable-everywhere region
  • reuse the rule identifier already resolved by lint instead of copying its RuleDescription again in performLint

Cache and narrow repeated analysis

  • replace the per-file artifact cache's concurrent queue with an NSLock while retaining computation deduplication
  • cache comment byte ranges per file and compute them with one position-tracking visitor pass
  • speed up CommentLinesVisitor by avoiding empty-trivia re-lexing, reversed trivia copies, repeated token-kind checks, and per-token converter lookups
  • collect indentation_width syntax data in one tree walk
  • cache source lines once per file for vertical_parameter_alignment
  • precompute inclusive_language term lists once per file
  • cache allowedSymbolsAndAlphanumerics, minimum-Swift-version support, environment state, file emptiness, and targeted file metadata
  • avoid general syntax classification and full-tree traversal where a token, root-node, line, or regex-candidate scan is sufficient
  • memoize nested function-chain suffixes to avoid quadratic re-walking

Improve scheduling

  • skip collection dispatch when no collecting rule is enabled
  • when the number of files is at least the active processor count, run rules serially per file and rely on file-level parallelism; few-file invocations retain per-file rule parallelism
  • sort parallel lint work by descending file size so expensive files start early instead of leaving one core processing the largest file at the end

Correctness fixes

  • make RandomAccessCollection.firstIndexAssumingSorted(where:) return nil, rather than endIndex, when no element matches
  • honor custom-rule execution_mode and inherited default_execution_mode
  • extend bridged line-comment ranges correctly across LF, CRLF, and CR endings while keeping block-comment extents unchanged
  • invalidate cached comment byte ranges with the rest of a file's cached artifacts

Impact

Large multi-file lint runs should spend less CPU time in SourceKit, syntax-tree bookkeeping, filesystem access, cache synchronization, rule post-processing, and shared-arena contention. Clean --fix runs avoid reconstructing the syntax tree for every correctable rule. Single-file and editor integrations retain concurrent rule execution. No configuration migration is required, and SourceKit execution remains available where explicitly selected or required.

Validation

  • the earlier 13-commit tree passed 101 focused Swift tests covering the SourceKit-free ports, custom-rule modes, comment/line handling, file caching, collecting behavior, and affected built-in rules
  • the 15-commit follow-up applied cleanly and was pushed without waiting for another test run; CI provides validation for the final tree
  • metadata rewriting preserved the applied tree, and all 28 commits use Brett-Best <brettbest@live.com.au> as author and committer

git diff --check reports only the two intentional Markdown hard breaks in this PR's changelog entries.

@SwiftLintBot

SwiftLintBot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
2 Warnings
⚠️ Big PR
⚠️ This PR may need tests.
19 Messages
📖 Building this branch resulted in a binary size of 28437.34 KiB vs 28357.8 KiB when built on main (0% larger).
📖 Linting Aerial with this PR took 0.53 s vs 0.65 s on main (18% faster).
📖 Linting Alamofire with this PR took 0.71 s vs 0.91 s on main (21% faster).
📖 Linting Brave with this PR took 4.2 s vs 6.05 s on main (30% faster).
📖 Linting DuckDuckGo with this PR took 19.24 s vs 26.17 s on main (26% faster).
📖 Linting Firefox with this PR took 7.35 s vs 10.59 s on main (30% faster).
📖 Linting Kickstarter with this PR took 5.04 s vs 7.38 s on main (31% faster).
📖 Linting Moya with this PR took 0.34 s vs 0.35 s on main (2% faster).
📖 Linting NetNewsWire with this PR took 1.71 s vs 2.37 s on main (27% faster).
📖 Linting Nimble with this PR took 0.45 s vs 0.54 s on main (16% faster).
📖 Linting PocketCasts with this PR took 4.31 s vs 6.98 s on main (38% faster).
📖 Linting Quick with this PR took 0.25 s vs 0.35 s on main (28% faster).
📖 Linting Realm with this PR took 2.07 s vs 2.48 s on main (16% faster).
📖 Linting Sourcery with this PR took 1.14 s vs 1.47 s on main (22% faster).
📖 Linting Swift with this PR took 3.04 s vs 4.28 s on main (28% faster).
📖 Linting SwiftLintPerformanceTests with this PR took 0.16 s vs 0.17 s on main (5% faster).
📖 Linting VLC with this PR took 0.85 s vs 1.12 s on main (24% faster).
📖 Linting Wire with this PR took 10.12 s vs 15.33 s on main (33% faster).
📖 Linting WordPress with this PR took 6.74 s vs 9.82 s on main (31% faster).

Generated by 🚫 Danger

Brett-Best and others added 13 commits July 17, 2026 06:45
@Brett-Best @claude
Compare operands by token stream in identical_operands rule
57a4bc4 
Comparing via debugDescription(includeTrivia:false) builds a full
indented debug dump of both operand subtrees (keypath-driven child
names included) for every infix operator expression. Comparing the
lazy token streams short-circuits on the first mismatch and allocates
nothing. ~6% of a large lint run in profiling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Cache allowedSymbolsAndAlphanumerics in NameConfiguration
4d3e5e9 
The CharacterSet union was rebuilt through CoreFoundation on every
checked identifier; it only changes when allowedSymbols changes.
~83% of identifier_name's cost in profiling a large repo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Read comment ranges from token trivia in comment rules
371f1e8 
comment_spacing and period_spacing only need comment byte ranges, but
obtained them via file.syntaxClassifications, which runs SwiftSyntax's
general-purpose classifier over every token in the file. A direct
trivia walk over the cached syntax tree yields the same ranges without
classifying non-comment tokens. Both rules are migrated together since
they shared the classification cache; migrating one alone would move
the cost to the other.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Reduce per-rule-per-file overhead in the lint hot path
8869892 
- Skip Linter.collect's concurrentPerform when no CollectingRule is
  enabled: it dispatched rules.count no-op iterations per file, whose
  existential/RuleDescription copies and task-local bookkeeping cost
  ~14% of a large lint run.
- Memoize the minSwiftVersion support check per rule type; it compared
  freshly parsed version strings and copied the full RuleDescription
  per rule and file.
- Read SWIFTLINT_DISABLE_SOURCEKIT once; ProcessInfo.environment
  rebridges the entire environment on every access.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Return nil from firstIndexAssumingSorted when no element matches
cd1c45d 
The documentation promises nil when no element satisfies the predicate,
but the termination check tested the midpoint index instead of the
partition boundary, returning count in that case. Existing callers only
survived because they immediately formed (empty) suffix slices from the
result; subscripting callers crash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Free the remaining lint-mode rules from SourceKit
4dbe288 
statement_position, vertical_whitespace_closing_braces,
vertical_whitespace_opening_braces, and indentation_width resolve
syntax kinds through a shared SwiftSyntax-derived syntax map;
multiline_function_chains, multiline_parameters_brackets, and
literal_expression_end_indentation are ported to SwiftSyntax visitors
reproducing SourceKit's structure geometry. The shared map extends
comment tokens over their terminating newline to match sourcekitd's
extents, and rules build it only for files whose regexes match at all.

With no enabled rule requiring SourceKit, lint runs never pay the
per-file sourcekitd editorOpen round trip (~13% of CPU on a large
repo profiling run).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Run SwiftSyntax-mode custom rules without SourceKit
80581d4 
execution_mode/default_execution_mode existed in configuration and in
isEffectivelySourceKitFree, but validate() always used the
sourcekitd-backed syntax map regardless of mode — a swiftsyntax-mode
configuration would even hit the "Never call this for file that
sourcekitd fails" trap when SourceKit is unavailable. Dispatch on the
effective mode and resolve kinds from the SwiftSyntax-derived map,
built lazily only when a kind-filtered pattern matches.

Also propagate defaultExecutionMode through parent/child configuration
merging, which rebuilt CustomRulesConfiguration and silently dropped it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Reduce per-file filesystem and gating overhead
a0c7022 
- Cache validation read the full attribute dictionary (including an
  enumeration of all extended attributes) per file just for the
  modification date; ask for the single resource value instead.
- Read a file's emptiness once per file instead of once per rule and
  file: the check acquired the contents queue lock on every call.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Avoid full-tree work in mark, file_length, and trailing_whitespace
86d14bd 
- mark: materialize each token's leading trivia once instead of once
  per piece, and gate the regex on a cheap substring pre-check.
- file_length: compute everything from the root node without walking
  the entire tree.
- trailing_whitespace: scan lines for candidates first and only
  pre-compute comment/string-literal line information when any exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Extend only line comments over their newline in the bridged syntax map
3dbae1c 
sourcekitd includes the terminating newline in line comment tokens but
ends block comment tokens at the closing delimiter. Extending all
comment kinds suppressed violations following block comments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Cache comment byte ranges per file and compute them in one visitor pass
abdad1a 
Three rules walk the file for comment trivia; the result is now shared
through the file cache. Byte positions are tracked incrementally during
a single visitor pass: SyntaxProtocol.position and token-sequence
iteration climb the tree per call, which degrades sharply on deeply
nested expression trees.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Memoize chain suffixes in multiline_function_chains
36e3497 
visitPost fires for every call in a chain and each computation shares
its suffix with the enclosing call's, so long chains were re-walked
quadratically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Run rules serially per file when many files are linted in parallel
e7c9b0b 
Every rule walks the same file's syntax tree, and concurrent walks
contend on the shared syntax arena's reference counts, slowing all of
them down. With at least as many files as cores, file-level parallelism
already saturates the machine, so rules now run serially within each
file in that case. Few-file invocations (single-file lints, editor
integrations) keep concurrent rule execution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best
Brett-Best force-pushed the codex/lint-hot-path-performance branch from 4459be4 to e7c9b0b Compare July 16, 2026 20:46
Brett-Best and others added 15 commits July 17, 2026 16:29
@Brett-Best @claude
Skip rewriting in corrections when a file has no violations
df59b99 
Rewriter-based corrections reconstructed the entire syntax tree and
serialized it back to a string for every correctable rule and file,
even when there was nothing to correct — the dominant cost of --fix
runs on clean codebases. Detect violations first via validate(file:),
which reports a superset of what the rewriter corrects (the rewriter
additionally skips disabled regions), and only rewrite when needed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Resolve self-linting violations
cc03be7 
Extract trailing_whitespace's candidate collection into a helper to
reduce cyclomatic complexity, wrap an overlong line, and move
indentation_width's helper visitors into their own file to respect
the file length limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Extend line comments over CRLF and CR line endings in the bridged syn…
bfa8332 
…tax map

The SourceKit-free syntax map extended line comments over their
terminating newline only for LF line endings; the seven rules using it
now also run on platforms where checkouts commonly use CRLF.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Lint the largest files first in parallel runs
65bcd89 
With serial per-file rule execution, a huge file dispatched near the
end of a run occupies a single core while the remaining cores sit
idle. Sorting files by on-disk size before dispatch hides long
single-file runtimes behind the remaining work regardless of core
count. Reporting order for parallel runs is already
scheduler-dependent, so this changes no observable behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Add changelog entries
6727401 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Replace the per-file cache's concurrent queue with a lock
985868e 
The per-file artifact cache guarded every slot access with a concurrent
DispatchQueue, taking a barrier sync on writes and a plain sync on reads.
Each lint forces roughly a dozen slots per file across all rules, so the
queue's dispatch overhead was a measurable share of a run. An NSLock with
the same computed-outside-the-lock / DispatchGroup-dedup semantics is
cheaper for this short-critical-section workload. Also invalidate the
comment-byte-ranges slot, which was missing from invalidateAll.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Skip body-line analysis for bodies within the length threshold
681a364 
The body-length rules computed the comment/whitespace-adjusted line count
for every body, forcing the per-file linesWithTokens set and allocating a
Set of the body's line range each time. The effective count can never
exceed the physical brace-to-brace span, so bodies whose span is already
within the threshold cannot violate and skip the analysis entirely. Where
the count is still needed, it is now derived without materializing a Set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Speed up CommentLinesVisitor's per-token work
3f8d4fe 
Gate trivia materialization on a byte-length check so empty-trivia tokens
skip re-lexing, iterate trivia forward instead of copying reversed arrays,
compute the token kind only when needed to detect end-of-file, and resolve
line numbers through a per-line bounds cache instead of a converter lookup
per token. Comment-line results are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Collect indentation_width's syntax data in a single walk
934bfb4 
The rule ran separate tree walks for multi-line condition continuations
and multiline string-literal spans; merge them into one visitor pass.
Rename the helper file to match its renamed type.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Cache source lines in vertical_parameter_alignment
8a890f3 
SourceLocationConverter.sourceLines rebuilds the whole file as a fresh
[String] on every access; the rule invoked it once per parameter of every
multi-parameter declaration. Cache it once per file and compute grapheme
columns only for the parameters actually compared.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Precompute inclusive_language term lists once per file
2d27019 
allTerms and allAllowedTerms were re-derived (set union, lowercasing,
dedup, sort) on every visited token. Compute them once per file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Gate comment_spacing's regex behind a cheap pre-check
c0d4717 
Hoist the stringView access out of the per-comment loop and run the
violation regex only on comments a cheap UTF-16 unit scan cannot rule
out, matching against the file's string view in place instead of
re-materializing each comment body.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Optimize colon, comma, closure_end_indentation, redundant_self
ca60505 
Hoist per-file invariants out of per-node work and gate expensive checks
on cheap pre-conditions in these four rules.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Reduce per-rule violation post-processing in the lint loop
3befcf3 
Skip region partitioning and identifier-set construction when they cannot
affect the result (no violations, or a single enable-everywhere region).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best @claude
Avoid recomputing the rule identifier in performLint
f1b5834 
lint already resolves the identifier and passes it into the rule context;
performLint recomputed it. Reading it goes through the rule's description
witness, which copies the whole RuleDescription value, so this removes one
such copy per rule and file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Brett-Best

Brett-Best commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Some other benchmarks which also include the fix jpsim/SourceKitten#846

Linting Aerial with this PR took 0.29 s vs 0.43 s on `main` (32% faster).
Linting Alamofire with this PR took 0.43 s vs 0.58 s on `main` (25% faster).
Linting Brave with this PR took 2.51 s vs 3.76 s on `main` (33% faster).
Linting DuckDuckGo with this PR took 10.73 s vs 13.83 s on `main` (22% faster).
Linting <<work project>> with this PR took 3.97 s vs 7.52 s on `main` (47% faster).
Linting Firefox with this PR took 3.99 s vs 5.59 s on `main` (28% faster).
Linting Kickstarter with this PR took 2.77 s vs 4.47 s on `main` (38% faster).
Linting Moya with this PR took 0.19 s vs 0.28 s on `main` (32% faster).
Linting NetNewsWire with this PR took 0.98 s vs 1.3 s on `main` (24% faster).
Linting Nimble with this PR took 0.24 s vs 0.32 s on `main` (25% faster).
Linting PocketCasts with this PR took 2.43 s vs 6.02 s on `main` (59% faster).
Linting Quick with this PR took 0.17 s vs 0.23 s on `main` (26% faster).
Linting Realm with this PR took 1.31 s vs 2.58 s on `main` (49% faster).
Linting Sourcery with this PR took 0.63 s vs 1.16 s on `main` (45% faster).
Linting Swift with this PR took 1.64 s vs 2.39 s on `main` (31% faster).
Linting SwiftLintPerformanceTests with this PR took 0.13 s vs 0.14 s on `main` (7% faster).
Linting VLC with this PR took 0.51 s vs 0.69 s on `main` (26% faster).
Linting Wire with this PR took 5.6 s vs 8.17 s on `main` (31% faster).
Linting WordPress with this PR took 3.72 s vs 5.16 s on `main` (27% faster).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Brett-Best @SwiftLintBot