Skip to content

build: harden 1.0 release validation - #295

Merged
Mattsface merged 2 commits into
release/1.0.0from
build/v1-release-validation
Aug 6, 2026
Merged

build: harden 1.0 release validation#295
Mattsface merged 2 commits into
release/1.0.0from
build/v1-release-validation

Conversation

@Mattsface

Copy link
Copy Markdown
Member

Closes #288

Why

The release validator proved less than the 1.0 contract needs.

It clean-installed only the wheel, so a source distribution that cannot build, omits a package file, or loses a runtime dependency would have shipped unnoticed. It proved the strict HTTP default only through constructor signatures, so a behavioral regression in strict handling could pass while the signature still read strict_http: bool = True. Its error messages did not always name the failing artifact, and its terminology and usage example implied a wheel-only, 0.9.0-only tool.

Deterministic CI had two matching gaps: it still watched the stale release/0.9.0 branch, and it tested only Python 3.10 through 3.12 with no twine check at all. The existing branch-trigger test derived the release branch from the still-unbumped 0.9.0 package version, so release/0.9.0 satisfied it.

Nothing in this pull request publishes anything.

What

Validator (scripts/validate_release.py)

Clean-install coverage:

  • The wheel and the source distribution are each installed into their own throwaway virtual environment, never a shared one
  • Each environment upgrades pip, installs the exact local artifact, writes the smoke-test script into the temporary workspace, and runs it from outside the repository checkout
  • The smoke test asserts mlbstatsapi.__file__ resolves under the environment's purelib and that it is running inside a virtual environment, so the checkout cannot shadow the installed artifact
  • Log labels are explicit: installing wheel, running wheel smoke test, installing source distribution, running source distribution smoke test
  • The source distribution installs through normal pip build isolation

Strict behavior, not only signatures. The smoke test keeps the strict_http signature-default assertions and adds deterministic offline behavior driven by a fake injected Session. The fake builds a realistic requests.Response (status_code=403, reason="Forbidden", a Stats API-shaped URL derived from the actual request, and a small JSON body) and performs no network I/O:

Construction Expected
Mlb(session=fake) (no strict_http, so the real default runs) final 403 raises MlbHttpError
Mlb(session=fake, strict_http=True) final 403 raises MlbHttpError
Mlb(session=fake, strict_http=False) get_sports() returns [] and warns exactly once

For each raised error the smoke test verifies status_code == 403, reason == "Forbidden", method == "GET", the expected URL, and that response_data contains the JSON payload. For compatibility mode it verifies exactly one captured warning, that its category is MlbHttpCompatibilityWarning, and that the message mentions strict_http=False. Neither the exception string nor the warning string is frozen.

Direct adapter construction keeps its signature checks and now also confirms, for both v1 and v1.1, that the default and explicit strict adapters raise MlbHttpError for the fake final 403, and that explicit compatibility mode returns an empty MlbResult (status_code == 403, message == "Forbidden", data == {}) with the public warning.

Injected Session preservation now uses a real requests.Session subclass that records close(). It sets a custom User-Agent, a custom application header, and custom HTTPS and HTTP adapters mounted with max_retries=0, captures the exact adapter object identities, and after with Mlb(session=session): ... verifies the Session was never closed by the library, both headers are unchanged, both adapters are the exact same objects, and neither adapter carries the library retry policy. The smoke test closes the Session itself in a finally block.

Library-created Session validation is retained and extended: the User-Agent must equal python-mlb-statsapi/<installed-version>, create_retry_policy() must return a fresh Retry per call, and both mounted adapters must carry the documented retry values without freezing unrelated Requests internals.

The public API contract checks are unchanged in substance: the same package-root imports, constructor signatures, exception and warning inheritance, MlbResult behavior, legacy helpers, and no __all__ in 1.0. No accidentally exposed submodule was promoted and no legacy helper was removed.

Diagnostics and terminology:

  • Every validation error names the artifact and, where applicable, the field or path with expected and actual values
  • Reverted strict defaults fail with Mlb.strict_http must default to True for the 1.0 contract and MlbDataAdapter.strict_http must default to True for the 1.0 contract rather than a bare AssertionError
  • Wheel-only wording became "distribution artifact" wording where the logic now covers both
  • The usage example shows the default invocation plus --expected-version 1.0.0 instead of presenting 0.9.0 as the only expected version
  • REQUIRED_SDIST_PATHS grew from three entries to the eleven paths the generated archive actually carries (PKG-INFO, LICENSE, README.md, pyproject.toml, and the six mlbstatsapi modules plus mlbstatsapi/models/__init__.py), verified against tar -tzf. Tests, docs, and scripts are intentionally absent from the sdist and are not required.

Validator unit tests (tests/test_release_validation.py)

scripts/validate_release.py is imported with importlib since scripts/ is not a package. Synthetic wheel ZIPs and sdist tarballs stand in for real artifacts, clean installs are stubbed, and no test creates a virtual environment or makes a network request.

Failure coverage added: missing dist directory, no matching wheel, no matching source distribution, multiple stale wheels, multiple stale source distributions, incorrect wheel Name, incorrect wheel Version, incorrect Requires-Python, ambiguous wheel METADATA, each required sdist path missing in turn, failed wheel installation, failed sdist installation, and failed smoke test for both artifacts. Each assertion checks that the message names the artifact and the incorrect field or path with expected and actual values.

Other coverage: validate() clean-installs both artifacts in that order, each artifact gets a distinct virtual environment, the smoke script is written outside the checkout, the smoke-test source compiles, and the strict-default failure messages remain present in the smoke-test contract.

The documentation Python-example syntax test now covers README.md, docs/http-transport.md, docs/public-api.md, and every file in docs/releases/ instead of hard-coding 0.9.0.md. Current-version checks are scoped to README.md, docs/http-transport.md, docs/public-api.md, and docs/releases/1.0.0.md; historical notes are asserted to keep their own version-specific User-Agent statements, and no historical 0.7.1, 0.8.0, or 0.9.0 fact was rewritten.

The CI branch-trigger test now requires release/1.0.0 literally and rejects release/0.9.0, so it no longer depends on the version bump owned by #289.

CI (.github/workflows/build-and-test.yml)

  • Pull requests and pushes watch main and release/1.0.0; release/0.9.0 is removed; workflow_dispatch is kept
  • The offline matrix tests Python 3.10, 3.11, 3.12, 3.13, and 3.14, with fail-fast: false so a single-version failure is easy to isolate. Python 3.15 is deliberately excluded while it is a prerelease
  • The build job uses Python 3.14 and runs rm -rf dist, poetry build, python scripts/validate_release.py, then poetry run twine check dist/*. Both the validator and twine check are required, in that order
  • Twine 6.2.0 was added as a Poetry development dependency (pyproject.toml and poetry.lock), never a runtime dependency
  • The live MLB API workflow is untouched: same schedule, same manual trigger, still separate from deterministic CI

Documentation

docs/public-api.md, docs/releases/1.0.0.md, and README.md now state a minimum declared Python version of 3.10 and CI-validated versions 3.10 through 3.14, replacing the "3.10, 3.11, 3.12" wording and the note that 3.13 coverage was still outstanding. No upper Python bound was added, the runtime requirement stays >=3.10, and no document claims Python 3.15. docs/releases/1.0.0.md and the README development section also describe the two-artifact validator and twine check.

Python 3.14 incompatibility found and fixed

Adding 3.14 to the matrix surfaced one real failure, committed separately as test: normalize Union rendering across the Python matrix:

tests/test_public_api.py::test_mlb_public_method_signature[get_persons-...]
get_persons: (person_ids: str | List[int], **params)
         != (person_ids: Union[str, List[int]], **params)

Python 3.14 renders typing.Union[a, b] through inspect.formatannotation as str | List[int], while 3.10 through 3.13 render Union[str, List[int]]. The annotation object and the public API are unchanged; only the rendering differs. The signature manifest now normalizes the legacy spelling so one manifest is valid on every supported interpreter. This was not a dependency problem: every locked dependency, including pydantic-core 2.41.5 and cryptography 50.0.0, installs and runs on 3.14, so no dependency constraint needed relaxing and Python 3.14 was not removed from the matrix.

Tests

Focused suite (Python 3.12, Poetry environment):

poetry run pytest tests/test_release_validation.py tests/test_public_api.py \
  tests/test_http_contract.py tests/test_http_warnings.py tests/test_mlb_session.py -v -rxX
454 passed

Full deterministic suite:

poetry run pytest tests/ --ignore=tests/external_tests -v -rxX
610 passed, 29 warnings

The baseline on release/1.0.0 was 521 passing tests, so this adds 89.

Package checks:

poetry check                     exit 0 (pre-existing [tool.poetry] deprecation warnings only)
rm -rf dist && poetry build      wheel + sdist built
python3 scripts/validate_release.py
poetry run twine check dist/*    both artifacts PASSED
git diff --check                 clean

Validator output shows separate installation and smoke testing per artifact:

  installing wheel: python_mlb_statsapi-0.9.0-py3-none-any.whl
  running wheel smoke test against the installed artifact
smoke test passed for python-mlb-statsapi 0.9.0
  installing source distribution: python_mlb_statsapi-0.9.0.tar.gz
  running source distribution smoke test against the installed artifact
smoke test passed for python-mlb-statsapi 0.9.0
Release validation passed for python-mlb-statsapi 0.9.0

Local multi-version runs, each with the locked dependency versions installed into a fresh environment. GitHub Actions remains the authoritative matrix:

Python Offline suite
3.10.20 610 passed
3.11.15 610 passed
3.12.3 610 passed
3.13.14 610 passed
3.14.7 610 passed

scripts/validate_release.py was also run end to end under Python 3.14 (the build-job interpreter) and passed both artifact installs, and poetry install resolved the lock cleanly on 3.14.

Negative testing outside the automated suite. The smoke test was extracted and run against a deliberately broken installed copy to confirm the failure messages are understandable:

  • strict_http reverted to False in both constructors: AssertionError: Mlb.strict_http must default to True for the 1.0 contract
  • Strict handling disabled while the signature default stayed True: AssertionError: Mlb.strict_http must default to True for the 1.0 contract: a final 403 did not raise MlbHttpError

The second case is the one signature checks alone could not catch.

CI configuration audit:

rg -n "release/0.9.0|release/1.0.0|3.10|3.11|3.12|3.13|3.14|3.15|twine|publish|upload" \
  .github/workflows pyproject.toml docs/public-api.md docs/releases/1.0.0.md

release/1.0.0 appears in both the pull-request and push trigger lists, release/0.9.0 is gone, 3.10 through 3.14 are present, 3.15 appears only in the comment explaining its exclusion, and poetry run twine check dist/* is present.

Live MLB API tests were not run and were not merged into deterministic CI.

Version-bump boundary with #289

pyproject.toml still declares version = "0.9.0". The 1.0.0 bump belongs to #289.

The validator continues to read the expected version from pyproject.toml by default, so today it accepts correctly built 0.9.0-named artifacts that carry the already-implemented 1.0 behavior, and after #289 the same unmodified validator will require 1.0.0 artifacts. 1.0.0 appears only as a usage example in the docstring and as a synthetic expected-version fixture in tests; it is never hard-coded as the only valid artifact version. A unit test asserts the validator stays version-driven.

Publishing safety

No workflow contains poetry publish, twine upload, a PyPI or TestPyPI token, pypa/gh-action-pypi-publish, softprops/action-gh-release, gh release create, or git tag. A parametrized test enforces this across both workflows. Twine is used only for twine check, which reads metadata and uploads nothing. No release, tag, or publish action was performed.

Risk and impact

Minimal.

No library code changed. mlbstatsapi HTTP decision logic, constructor defaults, the exception hierarchy, the warning implementation, retry values, Session ownership behavior, endpoint models, endpoint signatures, and the package version are all untouched. The diff is limited to scripts/validate_release.py, two test modules, the offline CI workflow, the Twine development dependency, and current documentation. Historical release notes and the live-test workflow were not modified.

The realistic failure modes are CI-side. Validation now runs two clean installs instead of one, so the build job takes roughly twice as long in that step (about 11 seconds locally) and the sdist install needs network access for build isolation, which CI already has. Python 3.13 and 3.14 are new required jobs, so a future dependency that drops support for one of them will fail CI rather than pass silently, which is the intent. The smoke test reads Mlb._session and Mlb._strict_http and builds requests.Response._content; these are private, so a future internal refactor could require a validator update. That trade-off is already accepted for release validation and documentation is explicitly held to the public API instead.

If something does go wrong, the blast radius is a failing CI job or a failing release validation run. No published artifact and no runtime behavior can be affected by this change.

Intentionally left out

  • The package version bump to 1.0.0 (Prepare and publish release 1.0.0 #289)
  • Any publishing, tagging, or release creation
  • Python 3.15, which is a prerelease during this work
  • Merging the live MLB API tests into deterministic CI
  • Cleaning up the accidentally exposed package-root submodules, which docs/public-api.md already tracks as a separate focused issue
  • The pre-existing poetry check warnings about migrating [tool.poetry] metadata to [project], which would change package metadata layout and belongs in its own change
Open in Web Open in Cursor 

cursoragent and others added 2 commits August 6, 2026 02:59
Python 3.14 formats typing.Union[a, b] as "a | b" while Python 3.10-3.13
format "Union[a, b]", so the get_persons signature manifest entry failed
only on 3.14. The annotation object is unchanged; normalize the legacy
rendering so one manifest stays valid on every supported interpreter.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
The release validator only clean-installed the wheel and only proved the
strict HTTP default through constructor signatures, so a broken source
distribution or a behavioral regression in strict handling could pass.
Offline CI also still watched the stale release/0.9.0 branch and tested
only Python 3.10-3.12.

Validator:

- clean-install the wheel and the source distribution in separate
  throwaway virtual environments and run the same smoke test against both
- generalize wheel-only terminology to cover both distribution artifacts
- verify a final fake 403 raises MlbHttpError under the default and under
  explicit strict_http=True, for Mlb and for MlbDataAdapter on v1 and v1.1
- verify strict_http=False returns the historical empty result and emits
  exactly one MlbHttpCompatibilityWarning mentioning strict_http=False
- assert MlbHttpError status_code, reason, method, url, and response_data
  without freezing the exception or warning strings
- verify an injected requests.Session keeps its headers and its exact
  adapter objects, never receives the library retry policy, and is never
  closed by the library
- verify a library-created Session carries the installed-metadata
  User-Agent and the documented retry policy
- label reverted strict defaults explicitly instead of raising a bare
  AssertionError
- name the failing artifact, field, or path with expected and actual
  values in every validation error
- expand the required source-distribution paths to the files the archive
  intentionally carries

Tests:

- unit-cover the validator helpers with synthetic wheel ZIPs and sdist
  tarballs, including every required failure mode
- prove validate() clean-installs both artifacts in separate environments
- lock the documentation Python-example checks to every release-notes
  file while keeping current-version checks off historical notes
- require the release/1.0.0 CI trigger literally instead of deriving it
  from the still-unbumped package version
- cover the Python matrix, twine check, and publishing-safety contract

CI and docs:

- watch main and release/1.0.0; drop release/0.9.0
- test Python 3.10 through 3.14 and build on 3.14
- add twine as a development dependency and run twine check on both
  artifacts; nothing is uploaded
- state the 3.10 minimum and 3.10-3.14 CI coverage in the current docs

The package version stays 0.9.0; the 1.0.0 bump belongs to a separate
issue and the validator keeps reading the expected version from
pyproject.toml.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
@Mattsface
Mattsface marked this pull request as ready for review August 6, 2026 04:44
@Mattsface
Mattsface merged commit 0cf6c56 into release/1.0.0 Aug 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants