Skip to content

Formal evaluation semantics, SPARQL/XPath conformance, XSLT focus, Iterate#25

Open
namedgraph wants to merge 11 commits into
mainfrom
feat/formal-semantics-evaluation
Open

Formal evaluation semantics, SPARQL/XPath conformance, XSLT focus, Iterate#25
namedgraph wants to merge 11 commits into
mainfrom
feat/formal-semantics-evaluation

Conversation

@namedgraph

Copy link
Copy Markdown
Member

Summary

Turns formal-semantics.md from a type catalog into an actual semantics, and aligns the implementation and test suite with it. Follow-up to the architecture evaluation (architecture-evaluation.md, first commit).

  • Formal definition (§3.8): abstract syntax, semantic domains (environment, focus, world), the big-step judgment ρ, φ ⊢ ⟨e, σ⟩ ⇓ ⟨v, ρ′, σ′⟩ with the full rule set, operator interpretation with effect classes, and metatheory (termination, determinism, ForEach-concurrency soundness). §3.8 is the definition; the §§3.2–3.7 prose restates it.
  • Document model (§2): syntactic form discrimination (operation call / URI reference / RDF data / generic object / sequence / scalar), the {"@id": …} URI reference form, scalar coercion table, normative JSON-LD reserved-key list, base-IRI rule.
  • Evaluation rules pinned: eager depth-first evaluation with declared quoted operands, sequence scoping (fresh variable scope per sequence and per ForEach iteration), error taxonomy, effect ordering. Resolves the bulk of tests/SPEC_GAPS.md (42 formerly UNCLEAR(spec) tests un-skipped).
  • SPARQL 1.1 / XPath conformance — no divergences: Str, Concat, Replace, EncodeForURI, STRUUID now follow their W3C signatures exactly (simple literals materialized as plain rdflib literals, as rdflib's own SPARQL engine does). Replace gains the flags argument and XPath $N replacement syntax with err:FORX* conditions.
  • XSLT focus: ForEach establishes (item, position, size); new Position and Last operations per fn:position()/fn:last(); Value's focus-item lookup closed to Binding + mapping.
  • Iterate ported from REST-VKG (XSLT 3.0 xsl:iterate-inspired): params / quoted operation / quoted next-iteration / structured break, normative 1000-iteration cap (keeps the algebra terminating). The operation catalog is now a strict superset of the Java/XML implementation's.
  • Linked Data response contract (§4.3–4.4): HTTP operations are RDF-specific and symmetric; conneg is transparent; non-RDF responses raise ValueError.
  • Interpreter fixes: mutable default arguments across all operations (variable leakage between documents in one process), boolxsd:integer coercion bug, Substitute's invalid _: N BNode serialization, Execute dropping the variable environment, scope-leaking list evaluation.

⚠️ Breaking changes

Deliberate, conformance-driven; a major version bump is warranted:

  • Str: language tags are no longer preserved (STR("hallo"@de)"hallo", per SPARQL); Str(BNode) now raises TypeError; results are simple literals (no datatype), not xsd:string.
  • EncodeForURI, STRUUID: return simple literals instead of xsd:string (datatype-sensitive comparisons change).
  • Concat: result kind now follows SPARQL CONCAT rules (was always xsd:string); shared language tags are carried.
  • Replace: replacement syntax is now XPath ($1 for groups; bare $/\ are errors — previously literal); zero-length-matching patterns and invalid flags now raise ValueError; language-tagged pattern/replacement now TypeError; result kind follows the first argument.
  • URI(BNode) and Substitute with a BNode binding: now TypeError (previously returned garbage).
  • Value: attribute lookup on arbitrary objects removed — focus-item shapes are closed to Binding + mapping (ValueError otherwise).
  • Current (and context-Value) outside ForEach: now ValueError (previously returned the default empty context).
  • Variable scoping: sequences and ForEach iterations now push/pop scopes — bindings no longer leak across iterations or out of nested sequences (previously leaked when an outer scope existed).
  • Scalar coercion: JSON booleans now coerce to xsd:boolean (previously xsd:integer via a bool-before-int bug); null is now a TypeError (previously the literal "None").
  • {"@id": "..."} objects: an object whose only member is @id now evaluates to a URI (the new URI reference form) instead of passing through as JSON-LD data.
  • ForEach context: operations now receive a Focus object rather than the bare item — third-party Operation subclasses that read self.context directly must unwrap it (all in-tree operations updated).
  • Filter: non-integer expressions raise TypeError (was NotImplementedError).
  • HTTP/SPARQL responses: non-RDF or unparseable responses raise ValueError (previously AttributeError/raw rdflib parser errors leaked).

Test plan

  • uv run pytest: 260 passed, 8 skipped (live-service/LLM-dependent), 20 deselected (network/sparql/ldh markers) — up from 145/50 on main.
  • uv run ruff check src tests: clean.
  • New suites: document model & scoping, client response contract (stub opener, offline), Position/Last, Iterate (incl. the 1000-cap and cursor threading), Concat.

🤖 Generated with Claude Code

namedgraph and others added 11 commits July 12, 2026 17:29
…plan

Evaluates the composed-operations premise, the formal semantics, the JSON
DSL, the Python codebase and the rdflib data model; diffs the operation
set and execution semantics against the Java/XML sibling (REST-VKG) and
lays out a three-tier plan: spec unification, two-way operation parity,
and Python code health.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d tests

Spec (formal-semantics.md rewrite):
- Add evaluation semantics: eager depth-first evaluation, quoted operands
  (ForEach.operation, Execute.operation), sequence forms with fresh variable
  scopes, context rules, effect classes and ordering guarantees, and a
  normative error taxonomy.
- Add the document model: optional envelope ({"@web-algebra": "1", "program":
  [...]}), the URI reference form ({"@id": ...}), normative JSON-LD
  discrimination keys, scalar coercion table, base-IRI rule.
- Rebuild the catalog with per-operation JSON argument tables; add missing
  Concat and ExtractOntology entries; fix Unit vs bottom on Variable; pin
  every decision tracked in tests/SPEC_GAPS.md (Str datatype, EncodeForURI
  charset, Replace dialect, STRUUID format, Substitute and Values rules,
  Merge set-union, Bindings order, Filter positional signature, ForEach
  output shape, Value/Current context rules, Execute narrative, Extract*
  endpoint role). ldh-* moves to an informative appendix.

Implementation alignment:
- Fix mutable default arguments across the interpreter and all operations
  (variable_stack/context leak between documents in one process, e.g. the
  MCP server).
- Sequence forms and ForEach iterations now push/pop a variable scope
  (replacing copy() semantics that leaked writes when an outer scope existed).
- Implement the URI reference form and Operation.unwrap_document; unwrap the
  envelope in main.py.
- Execute threads the variable environment (was silently dropped); Current
  errors without an iteration context; Value supports mapping context items;
  URI and Substitute reject BNodes (Substitute previously emitted invalid
  "_: N" syntax); Filter raises TypeError for non-integer expressions;
  boolean scalars coerce to xsd:boolean (bool-before-int); null forms raise
  TypeError; remove dead _serialize_for_json_context.

Tests: 42 UNCLEAR(spec) skips un-skipped and authored from the new spec;
new test_document.py covers the envelope, URI reference form, coercions and
scoping. 221 passed, 8 skipped (was 145/50).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The spec no longer documents divergences from W3C functions; operations
named after SPARQL/XPath functions follow those definitions by normative
reference, signatures included. Simple literals are materialized as plain
rdflib literals (no datatype) exactly as rdflib's own SPARQL engine does.

- Str: per `simple literal STR(literal ltrl)` / `STR(IRI rsrc)` — returns
  the lexical form / codepoint representation as a simple literal; language
  tags are no longer carried over; BNode is a type error.
- Replace: per REPLACE()/fn:replace — adds the optional flags argument
  (s m i x q), XPath replacement syntax ($N group references, \$ and \\
  escapes), err:FORX000* conditions as ValueError (invalid flags/pattern/
  replacement, zero-length-matching pattern), result kind follows the first
  argument, and pattern/replacement/flags must be simple literals.
- Concat: per CONCAT() result-kind rules — all xsd:string inputs yield
  xsd:string, a shared language tag is carried, anything else yields a
  simple literal.
- EncodeForURI, STRUUID: return simple literals per their signatures.
- SELECT/CONSTRUCT/DESCRIBE accept simple-literal queries via a new
  Operation.is_string_literal predicate (RDF 1.1 equivalence), so Str
  output composes into query arguments.

tests/SPEC_GAPS.md records the one honest implementation gap: XPath-only
regex constructs unsupported by Python re surface as ValueError. New
test_concat.py; Str/Replace suites rewritten against the W3C behavior.
238 passed, 8 skipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1.5.0 bump (510f4dd) updated pyproject.toml without regenerating the
lockfile's own package entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the optional {"@web-algebra": ..., "program": [...]} prolog: a
document is simply a single form or a program array. Removes
Operation.unwrap_document, the main.py unwrap step, the spec section and
error-table row, the system-prompt mention, and the envelope tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
formal-semantics.md §3.8 now contains the formal definition the document's
title promised: abstract syntax, semantic domains (Env, Focus, World), the
big-step judgment ρ, φ ⊢ ⟨e, σ⟩ ⇓ ⟨v, ρ′, σ′⟩ with the full rule set
(scalar/URI-ref/object/data/sequence/call plus the state-accessing forms
Variable, Value, Current, Position, Last, ForEach, Execute), the operator
interpretation δ_op with effect classes, and metatheory notes (termination
by structural induction, determinism modulo declared non-determinism, why
concurrent ForEach iterations are observationally sound). The prose of
§§3.2–3.7 is now the restatement; §3.8 wins on conflict.

The context (§3.5) generalizes to the focus — the triple (item, position,
size), exactly XSLT's dynamic context. ForEach establishes it per
iteration; new operations Position and Last expose it per XPath
fn:position()/fn:last() as xsd:integer literals; Current and unprefixed
Value lookups read the focus item. Outside any focus all three raise
ValueError.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the getattr fallback: "attribute of that name" was host-language
reflection, not a defined path step — it had no possible Java translation
and no focus-item shape that ForEach can produce needs it. The item shapes
are now closed (Binding → bound term, mapping → member value; anything
else raises ValueError), which makes Value fully well-defined and
REST-VKG-portable.

Also note in the catalog that Value has xsl:sequence semantics (value
as-is, no string conversion) — xsl:value-of is expressible as
Str(Value(...)).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sistence, Value-domain precision

Linked Data operations (§4.4) are now specified as RDF-specific and
symmetric: they read and write RDF graphs; content negotiation is handled
transparently by the implementation; a non-RDF response — unsupported
media type, missing Content-Type, or a body that does not parse as the
negotiated format — raises ValueError. §4.3 states the counterpart for
the SPARQL operations. client.py conforms: missing Content-Type no longer
crashes with AttributeError, and RDF-labelled bodies that fail to parse
raise ValueError instead of leaking rdflib parser errors; new stub-opener
tests pin the contract offline.

Also: Result values declared materialized and re-iterable (§1.1); the
Value domain's former JSON summand split into Object (generic-object
results over Values) and Data (RDF data forms: Term holes + raw JSON)
with their conversion boundaries stated (§3.1); scalar coercion row
reworded to "number parsed as floating-point"; Position type/operation
name overload noted in §1.1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The one operation the Java/XML Web Algebra had that the JSON side lacked,
inspired by XSLT 3.0's xsl:iterate. JSON serialization:

  {"@op": "Iterate", "args": {
    "params": {name: form, ...},           eager, bound as loop variables
    "operation": <quoted form or array>,   the loop body
    "next-iteration": {name: form, ...},   quoted; evaluated after each body
                                           in the iteration's environment
                                           (loop params + body bindings) and
                                           rebinds the parameters
    "break": {"name": ..., "equals"|"not-equals": ...}}}

Without next-iteration exactly one iteration runs; break compares the
named loop variable's lexical form after rebinding (missing variable
compares as "", REST-VKG parity); the result is the sequence of iteration
values (Units dropped); Iterate establishes no focus, so an enclosing
ForEach focus stays visible — matching the Java implementation, which
preserves the current binding across withVariable.

The iteration count is capped at a normative 1000 (as in REST-VKG), which
keeps the algebra terminating: §3.8 gains the (ITERATE) rule with a loop
helper measured by CAP − k, and the metatheory note is updated. §5 records
the Java implementation's fused merged-graph return and totalLimit as its
specialization (Merge(Iterate(...)) expresses the fusion here).

Spec §3.3/§3.6/§4.1/§3.8/§5, implementation, spec-derived tests (11 cases
incl. the cap and cursor threading), README and system prompt updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Major release: the formal evaluation semantics, SPARQL/XPath conformance
for the string operations, the XSLT focus (Position/Last), Iterate, the
@id URI reference form, and the interpreter fixes in this branch are
deliberately breaking (see PR #25 for the full list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tracts

Implements the remaining tractable Tier 3 items from architecture-evaluation.md
(3.1 mutable defaults and the _serialize_for_json_context deletion already
landed in the semantics work).

Exception taxonomy (3.3): new exceptions.py with WebAlgebraError and the
interpreter-level subclasses UnknownOperationError, InvalidFormError,
VariableNotFoundError, NoFocusError. Each ALSO inherits the built-in the spec's
error table (§3.7) mandates, so the normative contract and every existing
pytest.raises(TypeError|ValueError) assertion still hold, while callers gain
`except WebAlgebraError` to tell an ill-formed document from an unrelated bug.
Applied at the dispatch/evaluation/variable/focus sites (operation.py, value,
current, position, last); per-operation argument-type validation stays plain
TypeError per §3.7. HTTP transport failures are deliberately NOT wrapped —
§3.7 pins them unwrapped. §3.7 gains a sentence documenting the hierarchy.

HTTP-client dedup (3.4): new ClientOperation mixin builds the cert-configured
client once; GET/POST/PUT/PATCH (LinkedDataClient), SELECT/CONSTRUCT/DESCRIBE
(SPARQLClient) and ldh-AddFile (FileClient) drop their duplicated
model_post_init. SPARQLString keeps its own (OpenAI, different constructor).

Honest contracts (3.5): removed nine dead mcp_run methods from operations that
never claimed MCPTool (Bindings, Current, Execute, Filter, ForEach, Str, URI,
Value, Variable) — the server only dispatches mcp_run on MCPTool instances, so
those were unreachable. Establishes the invariant "mcp_run exists iff MCPTool
is claimed." (If Str/URI should be MCP tools, that is a deliberate feature add:
claim MCPTool + implement mcp_run.) ForEach/Iterate.execute now state plainly
that they are interpreter-level special forms with no pure form (§4.1).

Deferred to their own PR (large refactors coupled to the future parallel-ForEach
work, not rushed onto the 2.0 branch): extracting the interpreter into an
immutable ExecutionContext object (3.2), and generating inputSchema from
pydantic argument models (3.5 tail).

265 passed (5 new exception tests), ruff clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant