Skip to content

Add AST-based public API checker and fix documentation gaps it found#5267

Draft
nlohmann wants to merge 4 commits into
developfrom
claude/todo-191-plan-508110
Draft

Add AST-based public API checker and fix documentation gaps it found#5267
nlohmann wants to merge 4 commits into
developfrom
claude/todo-191-plan-508110

Conversation

@nlohmann

@nlohmann nlohmann commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the AST-based public API extraction tool requested in #3691 (todo #191): a tool whose primary job is to programmatically enumerate the actual public API surface of the library from C++ semantics via libclang, independent of documentation status.

  • tools/api_checker/extract_api.py — walks the libclang AST of the six public class templates plus free functions/operators in nlohmann::, and derives an overload-disambiguating identity from each declaration's raw source text (not libclang's USR — see below for why).
  • tools/api_checker/check_docs.py — flags public entries missing an @sa documentation link, and @sa comments that leaked onto non-public entities.
  • tools/api_checker/diff_api.py — overload-aware breaking/feature diff between two refs, with a fast path against committed history snapshots.
  • tools/api_checker/check_macros.py — advisory cross-check of documented macros against #define sites.
  • tools/api_checker/snapshot_release.py + tools/api_checker/history/ — backfills one immutable API-surface snapshot per released v3.x tag (v3.1.0–v3.12.0), so diff_api.py can compare any two releases without live extraction.
  • tools/api_checker/POLICY.md — what counts as public API, what's excluded, and what stability is guaranteed.
  • docs/mkdocs/docs/home/api_changes.md — a new, per-release, per-function reference of public API changes (v3.1.0 through v3.12.0), generated from the history snapshots. Complements, not replaces, the existing release notes.

Documentation backlog fixed

Running the tool against the current tree surfaced a real, previously invisible backlog, which this PR also fixes:

  • ~25 new API doc pages, each with a compiled and output-verified example: all of ordered_map's public methods, json_sax's constructor/destructor/operator=, byte_container_with_subtype's comparison operators, and several orphaned type aliases.
  • Missing @sa comments on ~25 existing declarations that already had a doc page.
  • Stale or outright incorrect "Version history" entries on several pages, found by diffing consecutive release pairs and checking whether the resulting change was actually reflected in the target page's history section (e.g. ordered_map::at/count/find/insert were dated to the wrong release; operator ValueType/operator= were missing entries for real, later signature changes).

Notable bugs found and fixed along the way

Both found by testing against real release tags rather than trusting the extractor in isolation:

  1. An earlier identity-key design based on libclang's USR encoded the enclosing class template's own arity — basic_json gaining one new defaulted template parameter changed literally every member's USR, producing ~230 false "changed" entries for a release with zero real breaking changes. Replaced with raw source-text signature capture, which is immune to this.
  2. v3.11.0/v3.11.1 used a different (pre-rename) ABI inline-namespace scheme (json_v3_11_0, no _abi segment) than every other release, which the ABI-tag-stripping regex didn't recognize — making those two releases look like ~100% API churn in diff_api.py. Fixed and backfilled a format_version bump so old and new identity schemes can never be silently compared.

Both are documented in extract_api.py's docstrings and tools/api_checker/history/README.md.

CI

.github/workflows/check_api_docs.yml runs extract_api.py + check_docs.py on every PR, advisory-only for now (continue-on-error, since the backlog for code this PR didn't touch may not be fully at zero), plus a blocking drift check against the committed tools/api_checker/api_surface.json.

Test plan

  • python3 tools/api_checker/extract_api.py --self-test passes
  • python3 tools/api_checker/extract_api.py finds 341 public API entries; check_docs.py reports only the one accepted, documented gap (insert_iterator)
  • python3 tools/api_checker/check_macros.py passes cleanly
  • All 21+ new/changed doc examples compile and their output matches via docs/mkdocs/docs/Makefile's %.test target
  • docs/mkdocs/scripts/check_structure.py passes (nav/link structure, including the new api_changes.md page)
  • diff_api.py cross-checked by hand against real git diff output for v3.11.2→v3.11.3, v3.11.3→v3.12.0, and v3.12.0→HEAD
  • CI run on this PR (pending)

🤖 Generated with Claude Code

@github-actions github-actions Bot added the L label Jul 11, 2026
nlohmann and others added 2 commits July 11, 2026 18:59
@nlohmann @claude
Add AST-based public API checker and fix documentation gaps it found (#…
f23b3c6 
…3691)

Adds tools/api_checker/: extract_api.py derives the public API surface directly
from the libclang AST (independent of documentation status), check_docs.py flags
public entries missing @sa links (and @sa on non-public ones), diff_api.py does
an overload-aware breaking/feature diff between two refs, check_macros.py cross-
checks documented macros against #define sites, and snapshot_release.py backfills
immutable per-release surface snapshots into tools/api_checker/history/ (v3.1.0
through v3.12.0) so diff_api.py can compare releases without live extraction.
POLICY.md documents what counts as public API and what stability is guaranteed.

Running this tooling against the current tree found and fixed a real documentation
backlog: ~25 new API doc pages (ordered_map's methods, json_sax's ctor/dtor/
operator=, byte_container_with_subtype's comparison operators, several orphaned
type aliases), each with a compiled and output-verified example, plus missing
@sa comments and stale/incorrect Version History entries on several existing
pages (found by diffing consecutive release pairs and checking whether the
resulting change was actually reflected in the target page's history section).

Also adds docs/home/api_changes.md, a per-release, per-function reference of
public API changes (v3.1.0 through v3.12.0) generated from the history/
snapshots, complementing (not replacing) the existing release notes.

Along the way, found and fixed several extractor bugs by testing against real
release tags rather than trusting the algorithm in isolation -- most notably an
identity-key scheme based on libclang's USR that encoded the enclosing class
template's own arity, and a since-renamed ABI inline-namespace pattern
(json_v3_11_0 vs. today's json_abi_v3_11_2) that neither of two earlier regex
attempts stripped correctly. Both are documented in extract_api.py's docstrings
and tools/api_checker/history/README.md so the failure mode doesn't recur
silently.

.github/workflows/check_api_docs.yml runs extract_api.py + check_docs.py in CI,
advisory-only for now (documented backlog may not be at zero for entities this
PR didn't touch), plus a blocking drift check on the committed
tools/api_checker/api_surface.json.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann @claude
Fix cross-environment API-surface drift and lint findings from CI
0663907 
extract_api.py's own extraction wasn't deterministic across machines, which
CI's drift check caught immediately: JSON_HAS_RANGES auto-detects via the
standard library's __cpp_lib_ranges feature-test macro, which isn't reliably
gated to C++20 mode by every stdlib -- undefined under -std=c++17 with macOS's
libc++, but defined under the identical flag with the Ubuntu stdlib CI uses,
so parse()/accept()/from_*() extracted different signatures purely depending
on which machine ran the extraction. Pinned to -DJSON_HAS_RANGES=0: the
deterministic and safe choice, since pinning to 1 was tried first and found to
fail to parse on a stdlib without full <ranges> support even when the macro
claims otherwise.

Also found and fixed a second, independent source of the same class of drift:
get_identity_name() used cursor.spelling verbatim for CONVERSION_FUNCTION
cursors, which libclang renders as its own internally-canonicalized form of
the return type rather than what's literally written. Confirmed for
json_pointer::operator string_t() spelling differently on two machines
pinned to the identical libclang==18.1.1 wheel, with the JSON_HAS_RANGES fix
above ruled out as the cause. Now derived from the cursor's own raw source
text instead, immune to libclang's dependent-type resolution differences and
incidentally more readable than the libclang-internal forms it replaces.

Bumped SURFACE_FORMAT_VERSION to 3 and regenerated all 27 history snapshots
and the committed api_surface.json; both fixes are documented in
tools/api_checker/history/README.md's format-history log.

Also fixes diff_api.py's format_version guard, which only compared the two
loaded surfaces against each other and never against SURFACE_FORMAT_VERSION
(what this build actually understands) -- two surfaces on the same,
newer-than-expected format_version would have silently passed the guard.

Remaining fixes are the concretely actionable findings from Codacy's review
of the new tools/api_checker/ files: unused imports/variables, a stray
f-string with no placeholders. Left the docstring-formatting nitpicks
(pydocstyle D2xx/D4xx) and generic subprocess-usage notices alone -- the
former has no established convention elsewhere in this codebase's Python
tooling to conform to, and the latter are inherent to a dev tool that shells
out to git/clang with developer-controlled arguments, not user input.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann nlohmann force-pushed the claude/todo-191-plan-508110 branch from f048d7f to 0663907 Compare July 11, 2026 17:00
nlohmann and others added 2 commits July 11, 2026 23:43
@nlohmann
Merge remote-tracking branch 'origin/develop' into claude/todo-191-pl…
3940f4b 
…an-508110
@nlohmann @claude
Regenerate api_surface.json after merging develop
a27065b 
develop gained iterator+sentinel support for accept()/parse()/sax_parse()/
from_cbor()/from_msgpack()/from_ubjson()/from_bjdata()/from_bson() (#5205)
since this branch was created, which check_api_docs.yml's drift check caught
immediately: GitHub's pull_request trigger checks out the PR-vs-base merge
commit, not the PR branch in isolation, so CI was comparing the committed
api_surface.json (generated before that upstream change existed) against a
fresh extraction of a json.hpp that already had it.

This superseded an earlier, incorrect diagnosis of the same symptom (a
JSON_HAS_RANGES cross-environment pin) -- that fix remains in place since it's
still a real, independent determinism improvement for three other
JSON_HAS_RANGES-gated locations in type_traits.hpp, but it wasn't the actual
cause of this particular mismatch: these new overloads turned out to be
unconditional (SFINAE-gated via can_compare_ne, not preprocessor-gated), so
no environment-detection difference was involved here at all.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nlohmann