Optimize writable entity lookup and preserve same-row alias updates

[crdv7]

Summary

This change avoids redundant graphid lookups when SET, REMOVE, or
DELETE writes an entity already read by MATCH.

For writable statements, AGE now carries the matched tuple's ctid as a
hidden internal lookup hint. The executor validates the tuple's snapshot
visibility and graphid before using it, and falls back to the existing
graphid lookup when the hint is missing or stale. Graphid remains the logical
entity identity.

The same executor path also contained a correctness issue: when two aliases
in one result row represented the same entity, a later SET or REMOVE
could rebuild the entity from stale alias-local properties and overwrite an
earlier change. This change refreshes the current properties and keeps
same-graphid aliases and paths synchronized.

The resulting lookup order is:

validated CTID fetch
    -> usable non-partial B-tree index whose first key is entity id
    -> sequential scan when no suitable id index exists

No Cypher syntax or user-visible result columns are added.

Implementation

Hidden CTID propagation

• Add a hidden CTID target entry for named vertices and edges read by
  MATCH, but only when the statement contains SET, REMOVE, or DELETE.
• Carry the hint through simple WITH n and WITH n AS alias projections.
• Stop propagation at aggregation, DISTINCT, computed expressions, and
  parser-generated variables, where a hidden value could change row-shaping
  semantics or become associated with the wrong entity.
• Store the hidden column position in update/delete plan metadata and include
  it in ExtensibleNode copy/out/read support.

PostgreSQL's native tid is packed into a scalar agtype value for transport
through AGE target lists:

BlockNumber (32 bits) << 16 | OffsetNumber (16 bits)

Read-only MATCH/RETURN statements do not receive hidden target entries.

Shared entity lookup

find_entity_tuple() centralizes tuple lookup for the writable executors.

1. If a hidden CTID is available, fetch that row version directly.
2. Accept it only when it is visible to the current snapshot and contains
   the expected graphid.
3. Otherwise, use a valid non-partial B-tree index whose first key is the
   entity id.
4. Fall back to a sequential scan only when no suitable id index exists.

This remains correct if a user drops a default vertex id index. It also
allows a user-created edge id index to serve as the fallback. The default
edge start_id and end_id indexes cannot resolve an edge by graphid.

A statement-local cache retains the tuple fetch slot, opened id index
relation, and negative index-search result. SET and DELETE own that cache
for their custom-scan lifetime.

CTID is always a hint rather than identity. A previous writable clause may
move the tuple and leave another alias with a stale CTID; the validation and
graphid fallback prevent an update or delete from targeting the wrong row.
After an update, the new CTID is propagated for later writes in the same
result row.

Same-row alias consistency

The original apply_update_list() built replacement properties from the
agtype value stored at each update item's tuple position before locating the
current heap tuple. Different aliases were treated as independent positions
even when they contained the same graphid, and the existing synchronization
pass updated entities inside paths but not top-level aliases.

For example:

MATCH (n:T), (m:T)
WHERE id(n) = id(m)
SET n.a = 1
SET m.b = n.a + 1
RETURN n.a, m.a, m.b

The query could return 1, NULL, 2 while the stored entity contained
a = NULL, b = 2.

The updated executor locates the current heap tuple before building the first
update for an entity position, refreshes stale base properties from that
tuple, synchronizes top-level aliases and paths with the same graphid, and
reuses the tuple for the physical update. REMOVE uses the same path and
receives the same consistency fix.

DELETE does not reconstruct properties, but it uses the validated lookup
helper so a stale alias CTID after an earlier SET cannot cause the wrong
tuple to be deleted.

The correctness fix and the lookup optimization are included together
because they share the same apply_update_list() control flow: the executor
must locate the current tuple before rebuilding properties, synchronize
same-entity references, and then reuse that tuple for the physical update.
Splitting them would require implementing and immediately replacing an
intermediate lookup path. If a correctness-only backport is requested, it
can be adapted to use the existing graphid fallback first.

MERGE fallback cache

MERGE uses a separate lateral/eager custom-scan transform that does not
currently carry a reliable entity-to-CTID column position into its SET
metadata, so it does not receive a hidden CTID fast path in this change. When
a plan contains ON MATCH SET or ON CREATE SET, it creates the shared
graphid fallback cache for the statement lifetime and releases it when the
custom scan ends.

Suggested review order

1. src/backend/executor/cypher_utils.c: shared CTID/index/SeqScan lookup.
2. src/backend/parser/cypher_clause.c: hidden CTID generation and WITH
   propagation.
3. src/backend/executor/cypher_set.c: current-property refresh and
   alias/path synchronization.
4. src/backend/executor/cypher_delete.c and
   src/backend/executor/cypher_merge.c: lookup/cache integration.
5. ExtensibleNode definitions and copy/out/read serialization.
6. Regression SQL and expected output.

Performance

Environment and method

• Baseline AGE:
  801417404978823bd8732452c3f7959017584785
• Patched AGE:
  1bfdc51b32f67303132cedde7820748e0b46f224
• PostgreSQL 18.4, built with --disable-debug --disable-cassert and
  CFLAGS=-O2 -g
• Ubuntu 22.04 and GCC 11.4.0 on both hosts
• Intel Xeon 6982P-C, 8 cores / 16 threads, 29 GiB RAM
• AMD EPYC 9T25, 4 cores / 8 threads, 15 GiB RAM
• Local Unix-domain socket
• shared_buffers=1GB, fsync=off, synchronous_commit=off,
  full_page_writes=off, autovacuum=off, jit=off

Each host ran the same complete benchmark matrix from 1,000 through
1,000,000 rows. PostgreSQL and its child backends were pinned to vCPU 2 to
avoid migration between virtual CPUs. Only age.so changed between variants.
Data reset, VACUUM ANALYZE, server restart, and shared-library replacement
were outside the timed interval.

The read-only control executes 20 prepared queries in one backend and reports
the per-query time, avoiding client startup and cold-backend costs from
dominating this short control. At 1,000,000 rows, controls use 15 measured
AB/BA-interleaved runs, DELETE uses five, and the longer writable and edge
workloads use three. Both variants receive an unreported warm-up. Negative
change means the patch is faster.

Workload	Rows	Intel baseline	Intel patched	Change	AMD baseline	AMD patched	Change
Read-only MATCH/count	1,000,000	173.207 ms	173.665 ms	+0.26%	119.926 ms	117.678 ms	-1.87%
SQL insert control	1,000,000	1.512 s	1.511 s	-0.08%	1.411 s	1.411 s	+0.07%
Cypher CREATE control	1,000,000	1.769 s	1.781 s	+0.68%	1.579 s	1.589 s	+0.63%
SET	1,000,000 vertices	1,146.793 s	1,083.368 s	-5.53%	772.290 s	740.688 s	-4.09%
REMOVE	1,000,000 vertices	1,105.770 s	1,087.867 s	-1.62%	754.795 s	757.081 s	+0.30%
DELETE	1,000,000 vertices	5.668 s	4.654 s	-17.88%	4.651 s	3.870 s	-16.79%
MERGE ON MATCH SET	1,000,000 operations	1,085.185 s	1,071.211 s	-1.29%	755.175 s	737.758 s	-2.31%
Edge SET, no edge id index	1,000,000	8,958.601 s	1,098.437 s	-87.74%	9,974.254 s	756.924 s	-92.41%
DISTINCT edge SET, edge id index	1,000,000	1,154.458 s	1,109.373 s	-3.91%	795.588 s	756.580 s	-4.90%

The same matrix was also run at 1,000, 10,000, and 100,000 rows. DELETE
improves from 5–8% at 1,000 vertices to approximately 17–18% at 1,000,000 on
both CPUs. Edge SET without an id index improves from 31–35% at 1,000
edges to 88–93% at 1,000,000 edges. These are the two effects the new lookup
path is intended to address.

SET and REMOVE still spend most of their time rebuilding properties and
updating PostgreSQL tuples, so their smaller gains vary by processor.
MERGE, which uses only the fallback cache, and indexed DISTINCT edge
SET, where the hidden hint cannot cross DISTINCT, do not show a
consistent direct-CTID effect.

Read-only statements do not receive hidden CTID target entries. In the final
1,000,000-row cross-CPU run, read-only elapsed time changed by +0.26% on
Intel, with CV below 0.5% for both variants, and by -1.87% on AMD. An Intel
perf stat comparison also showed unchanged retired instructions and only
+0.36% cycles. No size-dependent read-only regression was observed from
1,000 through 1,000,000 rows.

A supplemental 10,000,000-row Intel run remained neutral for the control
workloads: read-only +0.44%, SQL insert -0.30%, and Cypher CREATE +0.45%.
DELETE improved by 19.30%, from 57.332 s to 46.265 s, across five measured
runs with CV below 0.5%.

Benchmark reproduction

A self-contained reviewer script builds the baseline and patched commits,
creates an isolated PostgreSQL 18 cluster, runs the workloads in AB/BA order,
and writes raw and summarized CSV files. It neither installs AGE into the
PostgreSQL prefix nor modifies an existing cluster.

repro_pr_benchmark.sh

From an AGE checkout on the pull request branch:

chmod +x repro_pr_benchmark.sh
SERVER_CPU=2 READ_ONLY_BATCH_COUNT=20 CONTROL_ROWS=1000000 \
CONTROL_TRIALS=15 WRITE_ROWS=1000000 WRITE_TRIALS=3 \
WRITE_SCENARIOS="set remove merge_on_match_set" \
EDGE_SIZES=1000000 EDGE_TRIALS=3 \
DELETE_ROWS=1000000 DELETE_TRIALS=5 ./repro_pr_benchmark.sh \
  --age-repo "$PWD" \
  --pg-config /path/to/postgresql-18/bin/pg_config \
  --full

This command reproduces the 1,000,000-row timing matrix above. It is
intentionally long: a single no-index edge baseline sample takes several
hours on the tested hosts. Reviewers can first validate their environment
with:

SERVER_CPU=2 READ_ONLY_BATCH_COUNT=20 ./repro_pr_benchmark.sh \
  --age-repo "$PWD" \
  --pg-config /path/to/postgresql-18/bin/pg_config \
  --quick

The main output is all-summary.csv; per-group raw CSV, build logs,
PostgreSQL logs, and an environment manifest are retained alongside it.
Choose a SERVER_CPU available on the test host.

Standalone edge-label benchmark

The following self-contained script creates an edge label with only the
default start_id and end_id indexes, loads the requested number of edges,
prints the actual indexes, runs EXPLAIN ANALYZE, and verifies the updated
row count:

repro_edge_set_no_id_index.sql

# Quick check: 10,000 edges
psql -X -d <database> -f repro_edge_set_no_id_index.sql

# Same scale as the largest reported result; this can take several hours
psql -X -d <database> -v edge_count=1000000 \
  -f repro_edge_set_no_id_index.sql

CPU profile

The 2,000,000-vertex DELETE workload was sampled at 199 Hz with
cycles:u and DWARF call graphs:

Profile	Samples	Sample duration	Sampled cycles
Baseline	2,217	11.158 s	37.43 billion
Patched	1,822	9.144 s	29.70 billion

In the baseline, stacks below process_delete_list() containing
index_getnext_slot account for 16.56% of sampled cycles:

process_delete_list
  index_getnext_slot
    index_getnext_tid
      btgettuple
        _bt_first
          _bt_search

That stack disappears from the patched profile. Its replacement accounts for
3.09% of patched sampled cycles:

process_delete_list
  find_entity_tuple
    try_ctid_fetch_tuple
      table_tuple_fetch_row_version
        heapam_fetch_row_version
          heap_fetch

Baseline flame graph:

Patched flame graph:

Five-run perf stat medians for the 500,000-vertex DELETE workload:

Event	Baseline	Patched	Change
CPU cycles	8,784,745,592	7,027,241,137	-20.01%
Instructions	28,468,929,329	22,572,493,361	-20.71%
Branches	5,228,809,347	4,153,536,763	-20.56%
Branch misses	5,534,090	3,547,249	-35.90%
Cache references	14,515,881	13,605,987	-6.27%
Cache misses	3,389,227	2,924,470	-13.71%

The elapsed-time improvement is accompanied by comparable reductions in
instructions and branches, supporting a CPU-work reduction rather than an
I/O-only effect.

Tests

Built and tested against PostgreSQL 18.4:

• clean release build completed without warnings;
• targeted writable/VLE regression suites passed under debug/cassert:
  cypher_set, cypher_remove, cypher_delete, cypher_with,
  cypher_vle, and cypher_merge;
• targeted writable suites and the complete security/RLS suite passed with
  AddressSanitizer and MEMORY_CONTEXT_CHECKING;
• the current PR commit passed the full debug/cassert AGE regression suite:
  43/43.

New regression coverage includes:

• valid and stale CTID lookup, including fallback after an earlier SET;
• vertex and edge aliases that share a graphid across chained writes;
• dependent right-hand-side expressions and persisted-state verification;
• alias/path synchronization for SET and REMOVE;
• simple WITH references and renames;
• aggregation, DISTINCT, and computed-expression propagation boundaries;
• writable WITH * and RETURN * without leaking hidden CTID columns;
• stale CTID fallback for REMOVE, DELETE, and DETACH DELETE;
• fallback through a user-created edge id B-tree index;
• sequential-scan fallback after dropping a vertex label's id index;
• forced generic prepared-plan reuse across label-table DDL;
• direct TID-wrapper packing, including block/offset boundary values.

Related issues

Closes #2484
Closes #2485