[TEST](pyspark) Batch per-test DataFrames into one collect#577
Open
Seth Fitzsimmons (sethfitz) wants to merge 12 commits into
Open
[TEST](pyspark) Batch per-test DataFrames into one collect#577Seth Fitzsimmons (sethfitz) wants to merge 12 commits into
Seth Fitzsimmons (sethfitz) wants to merge 12 commits into
Conversation
pytest-testmon tracks which tests cover which source files and skips unaffected tests on subsequent runs. Activated via a TESTMON Makefile variable so the default `make check` uses incremental selection while `make check TESTMON=` runs the full suite. Lock the dependency in the dev group, gitignore the local cache file, and thread $(TESTMON) through the test, test-all, and test-only targets. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Pull the shared `dimension` and `comparison` fields of the five vehicle selector subtypes into a `VehicleSelectorBase` parent, and thread `discriminator="dimension"` through the `VehicleSelector` annotated union. The discriminator turns the union into a Pydantic discriminated union, so it serializes as JSON Schema's `oneOf` + `discriminator` rather than `anyOf`. Regenerated segment_baseline_schema.json captures the new shape. This is a prerequisite for downstream tooling that walks discriminated unions structurally (e.g. PySpark codegen for segment's nested vehicle scoping). Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Replace the Tonga-based Division/DivisionArea/DivisionBoundary fixtures with Kauaʻi County samples that exercise admin_level, capital_division_ids, wikidata, and source license alongside the existing fields. Replace the Tonga-based Connector/Segment fixtures with a Vermooten Street junction in Pretoria that exercises access_restrictions with when.vehicle, speed_limits with when.heading, routes with ref, road_surface, and multi-source attribution. Reformat the TOML with 4-space indents and sorted keys to match sibling theme packages. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
overture-schema-codegen and overture-schema-common shipped with no requires-python at all, so their published metadata advertised support for every Python version. pip would resolve and install them on 3.9 and below, where they fail at import: both packages depend on overture-schema-system, which declares >=3.10 and uses 3.10-only syntax. The floor existed in practice and went unstated in the one place installers read. Both now declare >=3.10, matching the root workspace and the other eleven packages. Placement follows each file's local key ordering -- last in the alphabetized codegen table, between description and license in common, where its sibling overture-schema-system puts it. uv.lock is unchanged: >=3.10 is what uv already resolved against. Raising the floor to 3.11 is deliberately not part of this change; it waits on Python 3.10's end of life (2026-10-31). Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Add overture-schema-pyspark, a runtime PySpark validation package whose
per-model expression modules and conformance tests are generated from the
same Pydantic models that define the schema, plus an `overture-validate`
CLI for validating Parquet on disk or in S3. PySpark plugs in as a peer of
the Markdown output target: same ModelSpec extraction, same four-layer
architecture (Discovery -> Extraction -> Output Layout -> Rendering), a new
pipeline module.
Runtime (overture-schema-pyspark/src/overture/schema/pyspark/):
- validate.py -- public API: validate_model(), evaluate_checks(),
explain_errors(). The explain stage unpivots per-row check results into
one row per violation, preserving input columns for join-back.
- schema_check.py -- compares a Spark schema against the expected
StructType, reporting structural mismatches.
- check.py -- Check / ModelValidation dataclasses.
- cli.py -- `overture-validate FEATURE_TYPE PATH` runs the pipeline in a
single pass, keeping memory bounded for arbitrarily large inputs. Output
is one row per violation: feature ID, theme/type, field, check, message,
offending value.
- expressions/ -- shared building blocks (constraint_expressions,
column_patterns, _schema_structs). Per-model expression modules are
generated under expressions/generated/.
- _registry.py -- walks the generated tree at import time, exposing
REGISTRY and PARTITION_MAP keyed by each module's ENTRY_POINT.
Codegen output target (overture-schema-codegen/src/.../pyspark/):
ModelSpec
|
constraint_dispatch constraints -> ExpressionDescriptor /
ModelConstraintDescriptor
|
check_builder FieldShape tree -> Check / ModelCheck IR
(FieldPath = ScalarPath | ArrayPath | MapPath and
Guard sum types; resolves array nesting, map
key/value projection, variant gating)
schema_builder FieldShape tree -> SchemaField list (StructType)
test_data/ ModelSpec -> BASE_ROW_SPARSE / BASE_ROW_POPULATED,
scaffolds, invalid values
|
renderer Check IR -> per-model expression module
test_renderer Check IR -> per-model conformance test module
|
pipeline orchestrates, returns GeneratedModule list
Every constraint Pydantic enforces is dispatched to a PySpark expression:
field constraints (bounds, typed length split by attachment layer, stripped,
pattern, unique items, geometry type, JSON pointer), map key/value
constraints (projecting map_keys / map_values and descending into sub-model
values), NewType overrides (LinearlyReferencedRange), base-type overrides
(HttpUrl, EmailStr, BBox completeness -- matching Pydantic), and model
constraints (require_any_of, radio_group, require_if, forbid_if,
min_fields_set, require_any_true).
Validation degrades gracefully when columns are skipped or structurally
absent: every check declares the top-level columns it reads, and a check
over a missing column -- top-level, nested inside an array or map
(sources[].confidence resolves to sources), or model-level -- is dropped
before Spark planning rather than raising a raw AnalysisException.
ValidationResult.absent_columns surfaces what was dropped; the CLI backstop
classifies residual planning errors through the structured PySpark error
API, so a generator bug propagates as a traceback instead of being mistaken
for a missing column.
Discriminated unions (segment is the canonical hard case) split into
per-arm test files, and colliding (field, check) identities across arms are
disambiguated with symmetric suffixes computed over the unfiltered check
list, so per-arm test filtering cannot hide a collision the shared
expression module still carries.
The generated trees under expressions/generated/ and tests/generated/ are
regenerable output of `make generate-pyspark` and are not tracked in git;
`make check` and `make test-all` regenerate before running.
Tested against Spark 3.4.0 - 4.1.1; the lowest-direct CI cell verifies the
declared PySpark floor.
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
PySpark 3.4 (the declared floor) doesn't run on Java 21, the default JDK on ubuntu-latest runners -- it hits NoSuchMethodException on java.nio.DirectByteBuffer.<init>(long, int), removed in JDK 21. Pin the lowest-direct cell to Java 17 so the resolved pyspark==3.4.0 can actually start. The default cell (which resolves to a current pyspark 4.x) keeps the runner's default Java 21. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
…cation The absent-column CLI backstop used pyspark-4-only error APIs, failing the lowest-direct CI cell (pyspark==3.4.0): - absent_column called AnalysisException.getCondition(), added in 4.0 (renaming getErrorClass, which 4.0 deprecates via FutureWarning). Prefer getCondition() where present, fall back to getErrorClass() on 3.4. - test_cli built AnalysisExceptions with the 4.x camelCase errorClass / messageParameters kwargs. 3.4 uses snake_case, forbids a message alongside an error class, and cannot build an UNRESOLVED_COLUMN through the public constructor at all (its message template is JVM-side). Fake the two accessors absent_column reads instead. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
The version-baselining job maps each workspace package to a topological level; the new overture-schema-pyspark package was absent, so `compare` raised "Unknown package for level computation". Add it alongside the other top-level consumers (cli, codegen, annex). Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Collapse field_path's ScalarPath/ArrayPath/MapPath taxonomy into Direct
+ Iterated and extend the grammar to interleaved map/array markers. A
container nested directly inside another with no field name between
(list[list[X]], dict[K, list[X]], list[dict], nested maps, a map reached
through an array) becomes an anonymous iterating segment, so every such
shape is now representable and renders, validates, and mutates end to
end.
Replace the four array/map wrap functions with one _wrap_in_iteration
fold over RenderFrames, and add the flattening runtime pair
nested_map_{keys,values}_check. Remove the union-under-multi-list and
newtype-under-list guards -- those shapes render correctly now -- and
add a guard for a union variant field reached through iteration past its
discriminator element, where the ElementGuard would gate the wrong
element.
The generated-conformance-test path handles the new shapes: set_at_path
learns the {value}/{key} grammar and descends trailing containers; model
mutations gain a composite element_path for map-then-array and
array-then-map nesting; shapes with no in-place mutation are loud
capability gates, not silent misroutes.
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
A model-level constraint declared on a submodel reached through a plain
struct field -- @require_any_of on Details where Feature.details:
Details -- now anchors at the struct prefix instead of collapsing to the
row root. _model_constraint_target is the identity on the prefix; the
renderer qualifies every field and condition reference with that prefix
(F.col("details.foo")) and gates an optional struct on presence
(F.when(F.col("details").isNotNull(), ...)). ModelCheck.read_columns
reads the single top-level struct column.
_guard_struct_nested_anchor narrows to struct-nested discriminated
unions only: their variant ColumnGuards render the discriminator as a
top-level column, which a struct-qualified path cannot express. Plain
struct-nested model constraints are now supported and no longer raise.
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Rewrite the Check Builder section of the codegen design doc for the current Direct/Iterated FieldPath taxonomy: anonymous segments, the single _wrap_in_iteration fold, and the mixed map/array nesting cases (dict[K, list], list[dict], nested maps, map-in-array), replacing the retired ScalarPath/ArrayPath/MapPath description. Disambiguate iter_frames (named-only structural folding) from the renderer's per-segment _render_frames walk. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 20, 2026 21:37 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
test_column_patterns.py and test_constraint_expressions.py each built a DataFrame and collected once per test -- ~200 Spark jobs between them. Both now use a case registry: each _Case carries its input column, the check over it, and a predicate on the result; a module-scoped fixture packs every case's input into one wide single-row DataFrame, applies every check in a single select, and collects once. This is the batch-once pattern the generated conformance harness uses. Every prior assertion is preserved. Timings (remaining floor is Spark session startup): test_column_patterns.py 6.4s -> 4.0s (33 -> 37 cases) test_constraint_expressions.py 16.9s -> 5.1s (164 -> 165 cases) The extra cases come from splitting valid/invalid two-row tests into explicit per-input cases; no coverage dropped. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
force-pushed
the
pyspark-test-batching
branch
from
July 21, 2026 03:05
16977cb to
68eda76
Compare
Victor Schappert (vcschapp)
self-requested a review
July 21, 2026 17:42
Victor Schappert (vcschapp)
approved these changes
Jul 21, 2026
Victor Schappert (vcschapp)
left a comment
Victor Schappert (vcschapp)
left a comment
Collaborator
There was a problem hiding this comment.
1.5X-3X test speed-up. I'll take it!
Seth Fitzsimmons (sethfitz)
force-pushed
the
generalize-field-path
branch
from
July 21, 2026 18:05
0d507b8 to
7f190e1
Compare
Seth Fitzsimmons (sethfitz)
requested review from
a team and
John McCall (lowlydba)
as code owners
July 21, 2026 18:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #572, stacked on its
generalize-field-pathbranch (the column_patterns cases use thenested_map_{keys,values}_checkhelpers #572 adds). I'll rebase onto the integration branch once #572 merges.Closes #576.
What changed
test_column_patterns.pyandtest_constraint_expressions.pyeach built a DataFrame andcollect()ed once per test — roughly 200 Spark jobs between them. Both now use a case registry: each_Casecarries its input column, the check over it, and a predicate on the result; a module-scoped fixture packs every case's input into one wide single-row DataFrame, applies every check in a singleselect, and collects once. This is the batch-once pattern the generated conformance harness already uses.Effect
Every prior assertion is preserved — the distinct asserted literals all survive and the parametrized cases stay green.
test_column_patterns.pytest_constraint_expressions.pyThe extra cases come from splitting valid/invalid two-row tests into explicit per-input cases; no coverage is dropped. The remaining floor is Spark session startup.