feat: always record per-declaration heartbeat costs with no performance degradation - #14712
Open
dennj wants to merge 1 commit into
Open
feat: always record per-declaration heartbeat costs with no performance degradation#14712dennj wants to merge 1 commit into
dennj wants to merge 1 commit into
Conversation
|
Mathlib CI status (docs):
|
Collaborator
|
Reference manual CI status:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes the compiler always record how many heartbeats each declaration costs and report it: compiling a module writes a .hb.json file next to the .olean (the way .ilean files are always produced), and the language server attaches the count to each 'goals accomplished' diagnostic for clients that opt in via a new heartbeatSupport client capability, so editors can display the cost of every proof as it is written.
Lean already maintains the heartbeat counter unconditionally. Every small allocation increments it to enforce maxHeartbeats. Elaboration even measures each declaration's usage in withRestoreOrSaveFull for incremental reuse, but the number is thrown away. The only way to recover it today is trace.profiler with trace.profiler.useHeartbeats, which costs 1.3×–5× compile time because it must retain and pretty-print the entire trace tree (work proportional to trace nodes, with per-declaration attribution requiring trace.profiler.output.pp and parsing declaration names back out of human-readable messages).
This PR instead reads the already-maintained counter at declaration boundaries only, so the work is proportional to declarations: two counter reads and one array push per declaration and phase (elab, kernel), pushed into one per-file sink and serialized once per module. On the elaboration side it reuses the measurement withRestoreOrSaveFull already makes, adding no counter reads at all. Auxiliary declarations (matchers, equation lemmas, generated code, deriving-generated instances) roll up to the user-written declaration that caused them, tracked by a CostOwner state machine threaded through the command and declaration elaborators.
Measured on a 615-module Mathlib slice with an interleaved, position-controlled A/B against master: wall-clock difference -0.2% (indistinguishable from noise), CPU time +0.6%. Counts are bit-for-bit reproducible across runs (they are allocation counts, not timings), which is what makes them usable as a CI metric on ordinary noisy runners.
What does this PR enable?
The motivation is that the Lean community's most visible cost metric is line count, and line count invites golfing: a readable, reusable proof can be replaced by a shorter brute-force one, and every existing signal reports the change as neutral or better while compile time quietly grows. The two proofs below prove the same statement; the numbers are this PR's own measurements:
theorem length_range_200 : (List.range 200).length = 200 :=
List.length_range -- 1.4k heartbeats
set_option maxRecDepth 10000 in
theorem length_range_200' : (List.range 200).length = 200 := by
decide -- 238k heartbeats, 170× the cost
Making the number always available lets both humans and AI tools see this difference at the moment the proof is written. As a demonstration, this vscode-lean4 branch renders the count next to each goals-accomplished marker using the new diagnostic field: https://github.com/Latinum-Formal-Methods/vscode-lean4-heartbeats
For a project like Mathlib, the per-module JSON also enables a per-declaration cost diff on every PR: baselines come from the build cache (mathlib's lake exe cache needs a one-line change to distribute the sidecar files), so no second compilation is needed. Declarations that fail or time out currently record nothing, and examples are not annotated in the editor since all examples in a namespace share one internal name.