feat: integration train — Waves 0–5 dependency-ordered manifest + verifier#14
Closed
estebanzimanyi wants to merge 16 commits into
Closed
feat: integration train — Waves 0–5 dependency-ordered manifest + verifier#14estebanzimanyi wants to merge 16 commits into
estebanzimanyi wants to merge 16 commits into
Conversation
estebanzimanyi
force-pushed
the
feat/integration-train
branch
2 times, most recently
from
b25000c to
06f058a
Compare
estebanzimanyi
changed the title
estebanzimanyi
force-pushed
the
feat/object-model
branch
from
6cd5d5b to
fae2912
Compare
libclang collapses the PG-vendored Jsonb (struct JsonbPair) and JsonPath (struct JsonPathParseItem) pointer types to int during header parsing, the same mechanism already handled for GSERIALIZED/GBOX/Interval/DateADT. Add both to the type-recovery map so the temporal-JSONB (tjsonb) and jsonbset functions carry their real Jsonb*/JsonPath* signatures in the generated IDL that every binding consumes (verified: jsonbset_value_n result recovers to Jsonb**).
Quadbin is a typedef of uint64 (meos_quadbin.h). Unlike H3Index — whose name is erased to int by the host-collision prefix rename, so the existing recovery (slot cType == int) catches it — Quadbin's name SURVIVES as the cType while only the canonical collapses to int (its underlying uint64 is the part that erased). So the recovery must also fire when the slot spells the typedef name, not just int. Add "Quadbin": "uint64_t" to the type-recovery map AND generalize the recovery to match either the collapsed `int` spelling or the surviving `original` typedef name, rewriting both cType and the collapsed canonical to uint64_t. Every catalog-driven binding now binds the quadbin cell as a 64-bit integer (bigint), like H3Index, instead of disagreeing on the collapsed int (JMEOS mapped the Quadbin cType to a Pointer; the Spark generator mapped the collapsed int to Integer).
The binding codegens consume each function's `shape` (array-return length, written-back output arrays, parameter nullability) from the IDL. Generate it from the headers instead of the hand-maintained meta stub, which had drifted to three functions and emitted no shape for the other 2900+ -- leaving every codegen to mis-handle the split/space-split/mvtgeom out-arrays and the nullable srs/maxt/state parameters. - parser/shapeinfer.py: infer arrayReturn/outputArrays from the parameter forms. A non-const `TYPE **` paired with a by-pointer `int *count` is a written-back out-array; a by-value `int count` marks a read-only in-array. - parser/nullable.py: read parameter nullability from the C Doxygen `@param ... may be NULL` source of truth and fold it into shape.nullable for the params present on each function. - run.py: wire both into the IDL build. Adds unit tests for both modules.
Follow each function's @csqlfn -> wrapper @sqlfn -> SQL name chain and attach the resulting SQL name to the catalog. The vendored-source root is overridable via MDB_SRC_ROOT so the @sqlfn (and @InGroup) extraction can be pointed at the same pinned MobilityDB checkout as the headers, keeping the generated catalog reproducibly equivalent to that pin.
A copy-paste @csqlfn in meos/src can point an ever/always spatial-relationship function (named <e|a><verb>_...) at the opposite-prefix MobilityDB-C wrapper — e.g. eintersects_tgeo_geo is tagged `@csqlfn #Aintersects_tgeo_geo`, so the chain resolves its @sqlfn to aIntersects instead of eIntersects. That silently drops the real (tgeo,geo) overload from the eIntersects group in a binding's overload dispatcher, leaving only a wrong-subtype backing reachable — which then raises a runtime "The temporal value must be of type tcbuffer" on a tgeompoint (observed in the MobilitySpark BerlinMOD bench, q17). The parser is faithful — it propagates whatever the source tags say — so this is a source defect to fix in meos/src, not a parser bug. Add lint_ea_sqlfn() and report the mismatches at catalog-gen so they surface loudly instead of shipping a wrong @sqlfn silently. Flags 5 live mistags (eintersects_tgeo_geo, etouches_tpoint_geo/_tcbuffer_geo/_tcbuffer_cbuffer, econtains_geo_trgeo); relayed to the source maintainers to correct the @csqlfn tags.
A single MEOS function can back more than one MobilityDB-C wrapper: the ever/always pair (eDisjoint/aDisjoint) shares one ea_* function, and the shift/scale/shift_scale trio shares one C function. Such functions carry one @csqlfn with comma- or space-separated #Wrapper() references that may continue across doxygen lines. _meos_to_mdb reads the whole tag value and returns every wrapper, and attach_sqlfn_map records all the SQL names a function exposes (sqlfn/sqlfnAll, mdbC/mdbCAll).
Add lint_sqlfn_case_collisions next to lint_ea_sqlfn: it groups every @sqlfn (and sqlfnAll) name by its lower-case form and flags any group with more than one spelling (e.g. tDistance vs tdistance), wired into run.py as a loud warning. Such names are the SAME SQL function — PostgreSQL folds unquoted identifiers to lower case — so the clash is invisible in SQL and pg_regress. But the binding name is case-sensitive, and case-insensitive engines (Spark SQL) register every spelling under one UDF, so one silently shadows the other (e.g. the trgeometry tdistance shadowing the tgeo tDistance dispatch, returning NULL). Surfacing it at catalog generation lets the casing straggler be fixed at the MEOS-C @sqlfn source before it reaches any binding.
Parse the @InGroup meos_<group> tag from each MEOS-C function's source comment (meos/src) and attach it as a 'group' field on every catalog function. Those groups are the structure of the MEOS reference manual / doxygen XML (meos_setspan_accessor, meos_temporal_comp_ever, meos_geo_constructor, ...), so every catalog-driven binding can organize its generated surface the same way the manual does — a function is found in the same place across all tools. The meos_internal_* groups are carried through too, so a binding can filter the non-user-facing surface. 2155 user-facing functions across 175 groups.
Adds parser/enrich.py: a clang-free pass that derives, for every function and opaque type, the metadata a service generator (OpenAPI/MCP/gRPC) needs but that C headers do not contain: - category (constructor/predicate/io/accessor/transformation/...) - typeEncodings: per opaque type, its text/MF-JSON/WKB in/out functions - network.exposable: whether the function maps to a stateless endpoint, with a precise reason when it does not (array/out-param, no encoder/ decoder, lifecycle, index) - wire: per-parameter and return request/response representation Matches the canonical C spellings libclang emits (struct-qualified names, unsigned char, int-returning predicates) and treats enums as scalars. Runs before merge_meta so every derived field is overridable from meta/meos-meta.json. Wired into run.py as step 2/3, documented in docs/enrichment.md, and tested on real signatures in tests/test_enrich.py (stdlib unittest, no libclang/pytest needed). Validated against the live MobilityDB master catalog: 2672 functions, 1790 (67%) stateless-exposable.
Adds generator/openapi.py + generate_openapi.py: projects the enriched MEOS catalog (network/wire/typeEncodings from the service-projection pass) onto an OpenAPI 3.1 document.
- one RPC-style POST /{function} per stateless-exposable function
(a function is to MEOS what a process is to OGC API - Processes)
- opaque values cross the wire as strings in their typeEncodings
(text/MF-JSON/HexWKB), surfaced as reusable component schemas
- enums become string component schemas with the real C constant names
- x-meos-{category,decode,encode,in,out,encodings} make the document
self-describing for a downstream server/MCP/gRPC generator
- pure dict -> dict (no libclang, no MEOS runtime), deterministic output
Logically depends on the catalog being enriched. Validated against the
live MobilityDB master catalog: 2672 functions -> 1790 operations over
14 component schemas, all $refs resolve. Documented in docs/openapi.md;
tested in tests/test_openapi.py (stdlib unittest).
…bilityDB#5 follow-up) Adds 'generator/movfeat.py' and 'generate_movfeat_openapi.py' as a second OpenAPI projection alongside the generic RPC-style projection in 'generator/openapi.py'. The MovFeat projection maps the subset of MEOS functions that have an OGC API – Moving Features (OGC 22-003r3) analogue onto the OGC-defined REST resource hierarchy under '/collections/{collectionId}/items/{featureId}/…'. Each route carries 'x-meos-{function,decode,encode}' extensions so a downstream OGC server (MobilityAPI, in this ecosystem) can dispatch each call to the right MEOS function without re-deriving the mapping. Reuses the _value_schema / _type_schema / _enum_schema helpers from 'generator/openapi.py' so component schemas and the MeosError response are byte-identical across the two projections. Smoke-tested locally on the catalog produced by 'run.py' against MobilityDB master headers: 10 paths, 5/6 unique MEOS function dependencies present in the catalog (temporal_as_mfjson, temporal_from_mfjson, tpoint_speed, temporal_derivative, tpoint_cumulative_length, tpoint_azimuth). Closes the 'OGC API – Moving Features resource projection' natural follow-up named in PR MobilityDB#5's body. The MobilityAPI ingestion plan (MobilityAPI MobilityDB#3, step 5) consumes this projection.
Runs the full regenerate path on every PR touching the parser, generator, meta files, or generate_openapi.py: - libclang sysroot install (matches MobilityAPI vendor-drift) - clone MobilityDB master for MEOS headers - run.py + generate_openapi.py - openapi-spec-validator against OpenAPI 3.1 - upload meos-openapi.json as an artefact Catches OpenAPI 3.1 violations the moment a generator change introduces them, instead of letting downstream consumers (MobilityAPI vendor-drift, PyMEOS-CFFI codegen, MobilityDuck binding generator) discover them later. Named as a 'natural follow-up' in PR MobilityDB#5's body.
Makes ecosystem-wide 100% parity provable at one point instead of asserted from per-PR isolation-green. meta/integration-train.json is the PR dependency DAG + per-wave gates + merge order; verify-train.sh composes the train and runs each wave's gate (PASS only when just-run green here, else BLOCKED with the exact gate it needs). Operationalizes MobilityDB discussion #895 (wave-based merge plan). Stacked on feat/object-model (the catalog anchor): Wave 0 verifies here (2699 fns, PR MobilityDB#10 21/21, from_mfjson + constructors uniform); Waves 1-3 are honestly gated on the MEOS-1.4 bump (the single universal unblock).
estebanzimanyi
force-pushed
the
feat/object-model
branch
from
fae2912 to
cb63cf4
Compare
estebanzimanyi
force-pushed
the
feat/integration-train
branch
from
06f058a to
ae88636
Compare
estebanzimanyi
force-pushed
the
feat/object-model
branch
from
cb63cf4 to
2c3c44e
Compare
Member
Author
|
The per-artifact contracts this coordinated are now in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #11. The original PR's branch lives on
MobilityDB/MEOS-API:feat/integration-train(predates the fork-branch-only policy). This PR rebuilds it from the fork with the additive Wave 4 + Wave 5 manifest entries:meta/integration-train.json#/waves[4]meta/integration-train.json#/waves[5]docs/integration-train.mdmeta/integration-train.jsonWave 4 — what's in it
Each binding regenerates against the Wave-0 MEOS-1.4 C surface via the
meos-idl.jsoncatalog. Each is bump-independent within its repo; the cross-binding gate is byte-identical regeneration (proves single SoT).Wave 5 — what's in it
Three sub-layers, all anchor-defined and downstream of Wave 4:
[[apache-polaris-readiness]]) ships when substrate lands.Branch base
This PR is on
feat/integration-train(fork) and uses the same base as #11:feat/object-model(MEOS-API PR #10). Stack relationship unchanged.Action requested
MEOS-APIfork atestebanzimanyi:feat/integration-train, matching the standard fork-only pattern.