From cde2eb9dc2915edf535859d031ee3c507fd72fe2 Mon Sep 17 00:00:00 2001 From: Pedro Diogo Date: Wed, 29 Jul 2026 11:34:32 +0100 Subject: [PATCH] feat(vendor): add The Graph (decentralized indexing, audit read path) --- CHANGELOG.md | 2 + .../pattern-reproducible-audit-extraction.md | 121 ++++++++++++++++++ vendors/the-graph.md | 87 +++++++++++++ 3 files changed, 210 insertions(+) create mode 100644 patterns/pattern-reproducible-audit-extraction.md create mode 100644 vendors/the-graph.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8395e75..d6be8eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ All notable changes to the EthSystems Map are documented here. - docs: add Espalier preprint reference to [ZK Proof Systems](patterns/pattern-zk-proof-systems.md), [Post-Quantum Threats](domains/post-quantum.md), and [Safe Proof Delegation](patterns/pattern-safe-proof-delegation.md) `## See also` sections - feat(jurisdiction): [EU / EUDR (Deforestation Regulation)](jurisdictions/eu-EUDR.md) -- Article 9 plot-level geolocation and the DDS reference-number model, ahead of the 30 December 2026 application date ([#181](https://github.com/ethsystems/map/pull/181)) +- feat(vendor): add [The Graph](vendors/the-graph.md), decentralized indexing and query infrastructure as the audit read path ([#186](https://github.com/ethsystems/map/pull/186)) +- feat(pattern): add [Reproducible Audit Extraction](patterns/pattern-reproducible-audit-extraction.md), the verifier's data-acquisition leg the disclosure patterns presuppose ([#186](https://github.com/ethsystems/map/pull/186)) ## [0.4.0] - 2026-07-02 diff --git a/patterns/pattern-reproducible-audit-extraction.md b/patterns/pattern-reproducible-audit-extraction.md new file mode 100644 index 0000000..cf25639 --- /dev/null +++ b/patterns/pattern-reproducible-audit-extraction.md @@ -0,0 +1,121 @@ +--- +title: "Pattern: Reproducible Audit Extraction" +status: draft +maturity: testnet +type: standard +layer: offchain +last_reviewed: 2026-07-16 + +works-best-when: + - An auditor, regulator, or counterparty must verify that disclosed or reported figures reflect the complete set of on-chain emissions, not a curated subset. + - Disclosure patterns (viewing keys, predicate proofs, encrypted audit logs) are in place and the open question is what the disclosed material is checked against. + - The verifying party can run, or commission, an independent re-execution of the data pipeline. +avoid-when: + - The facts to verify are off-chain (bank balances, custodial holdings); no extraction of chain data can establish them. + - The verifier will never re-execute; without at least sampled re-runs the pattern collapses to trusting the pipeline operator. + +context: both +context_differentiation: + i2i: "An institution's auditor or a supervisory authority re-runs the published pipeline over public inputs and compares output hashes, replacing reliance on the reporting institution's (or its data vendor's) pipeline with reliance on canonical chain data. Bilateral disputes reduce to a hash mismatch that either party can demonstrate." + i2u: "End users and the public can verify institutional claims anchored on-chain (supply figures, registry roots, log anchors) without permission from the institution, provided the recipe is published. The binding constraint is resources: re-execution favors parties who can run infrastructure, so public-interest verification in practice depends on independent third parties publishing their re-runs." + +crops_profile: + cr: high + o: yes + p: none + s: medium + +crops_context: + cr: "Verification is permissionless: inputs are public chain data and the recipe is published, so no operator can prevent an independent party from re-deriving the result. Drops to `medium` when the canonical input view itself depends on a small set of archive-data providers for deep history." + o: "Open by construction: the pattern does not work with a closed recipe, since the verifier must re-run it. Several implementations under permissive and copyleft licenses exist. Deployments that keep transform code proprietary are not instances of this pattern." + p: "None. The pattern operates on public emissions (events, commitments, nullifiers, anchors, attestation logs) and adds no confidentiality. Privacy of the underlying business data comes from the shielding and disclosure patterns it composes with; this card covers the verification leg those patterns presuppose." + s: "Rests on determinism engineering (pinned toolchains, content-addressed inputs, no wall-clock or float nondeterminism) and on the verifier obtaining a canonical view of the chain. No consensus, committee, or hardware trust is introduced; the residual risk is nondeterminism bugs and verifiers who never actually re-run." + +post_quantum: + risk: low + vector: "The pattern itself relies on hash functions (content addressing, Merkle anchors), which retain security against a CRQC up to Grover-type reductions. Signature and commitment exposure is inherited from the host chain and from the composed disclosure patterns, not added here." + mitigation: "Use hash constructions with adequate post-quantum margins for content addressing and anchors (SHA-2/SHA-3, Poseidon per its stated margins). See [Post-Quantum Threats](../domains/post-quantum.md)." + +standards: [] + +related_patterns: + composes_with: [pattern-l2-encrypted-offchain-audit, pattern-regulatory-disclosure-keys-proofs, pattern-compliance-monitoring, pattern-commit-and-prove] + see_also: [pattern-verifiable-attestation, pattern-private-information-retrieval] + +open_source_implementations: + - url: https://github.com/streamingfast/firehose-core + description: "Firehose: deterministic, content-addressed flat-file extraction of chain history (The Graph / StreamingFast, Apache 2.0)" + language: "Go" + - url: https://github.com/streamingfast/substreams + description: "Substreams: deterministic WASM transform modules over Firehose files (Apache 2.0)" + language: "Rust" + - url: https://github.com/TrueBlocks/trueblocks-core + description: "TrueBlocks: local, reproducible index of Ethereum address appearances (GPL-3.0)" + language: "Go" + - url: https://github.com/paradigmxyz/cryo + description: "cryo: deterministic extraction of chain data to Parquet/CSV datasets (Apache 2.0)" + language: "Rust" + - url: https://github.com/graphprotocol/graph-node + description: "graph-node: deterministic indexing of extracted data into a queryable entity store, with proofs of indexing over the result (Apache 2.0)" + language: "Rust" + - url: https://github.com/subsquid/squid-sdk + description: "Subsquid: independent indexing framework serving queryable indexes over extracted chain data (Apache 2.0)" + language: "TypeScript" +--- + +## Intent + +Give auditors, regulators, and counterparties a read path over public chain state they can independently re-execute to byte-identical output. Disclosure patterns govern who sees plaintext; this pattern establishes what the complete emitted record was, so a scoped disclosure can be checked against it. + +## Components + +- Deterministic extraction: canonical blocks, receipts, and logs written to content-addressed files traceable to block hashes. +- Deterministic transform: versioned code (pinned dependencies, no nondeterministic ordering, no wall-clock or floating-point dependence) deriving the report-shaped output (a supply figure, a registry root). +- Re-execution manifest: input content hashes, transform code version and hash, output hashes, and the block range covered. +- Completeness scope: the contract addresses, event signatures, and block range that bound "the complete set" for an engagement. +- Optional on-chain anchor: a hash or Merkle root of the output posted on-chain as a fixed point for later verification. +- Serving layer: a queryable index over the extracted record, so a relying party can ask completeness-shaped questions (every in-scope entry across a range, in order, with gaps and conflicts surfaced) without re-running the pipeline. It is audit infrastructure rather than convenience only if it **reports what the record contains**, including malformed and conflicting entries, since dropping them silently is indistinguishable from their absence, and if the serving party is **identifiable and answerable** for a wrong answer. Serving never replaces re-execution as the trust mechanism. + +## Protocol + +1. [operator] Extract the in-scope chain data into content-addressed files anchored to block hashes. +2. [operator] Run the versioned transform; produce outputs plus the re-execution manifest. +3. [operator] Deliver outputs and manifest to the relying party; optionally anchor the output hash. +4. [auditor] Obtain a canonical chain view independently (own node, or light-client-verified headers) and re-run extraction and transform from the manifest. +5. [auditor] Compare output hashes; byte-identical output verifies the pipeline, a mismatch localizes the dispute to specific inputs or code versions. +6. [regulator] Check that material disclosed under a viewing key or proof reconciles against the re-derived record, both directions: every anchored commitment accounted for, none added, none omitted. + +## Guarantees & threat model + +Guarantees: + +- Faithfulness: any party can re-derive the figures from canonical chain data; the operator cannot alter inputs or logic without breaking hash equality. +- Completeness within scope: the re-derived record contains every in-scope emission, so a disclosure checked against it cannot omit anchored records. + +Threat model: + +- Completeness is relative to on-chain emissions. Facts never emitted (an unrecorded liability, an off-chain balance) are invisible to any extraction; closing that gap needs legal attestation, not this pattern. +- Nondeterminism silently breaks verification; determinism must be asserted in CI, not assumed. +- The verifier's chain view is a dependency: fed a wrong canonical view, it verifies the wrong record. +- Scope-definition gaming: an operator who controls the scope (addresses, event set) can exclude inconvenient contracts, so the scope must be fixed by the relying party or disclosed for challenge. +- A verifier who never re-runs receives no guarantee; sampled re-execution is the minimum honest posture. +- A serving layer that is not itself checkable becomes a new trusted party: where practical access is a served index rather than the verifier's own re-run, the guarantee degrades to trusting the server unless served output is periodically reconciled against re-derived output. + +## Trade-offs + +- The verifier does the work, and the cost depends on what the transform reads. Event-derived values are re-derivable from any node retaining historical receipts; values derived from contract-call results or execution traces need the EVM re-executed against archival state or a trace-capable node, a materially heavier requirement. Deep history is a third constraint: as history expiry advances, old ranges depend on independent archives, not on any node's retention policy. Proof-carrying alternatives (SNARK-verified SQL over committed tables) shift cost from verifier to prover but cover a narrower class of transforms and add cryptographic assumptions. +- Determinism engineering is real maintenance overhead across toolchain upgrades. +- Monthly or quarterly attestation cadences absorb re-run latency; real-time supervisory queries do not. + +## Example + +- A bond issuer settles on an L2, anchoring hourly Merkle roots of its encrypted trade log on-chain ([Low-cost L2 + Off-chain Encrypted Audit Log](pattern-l2-encrypted-offchain-audit.md)). +- At period close, the auditor receives the manifest, re-extracts the anchor events from an independent node, and re-derives the root chain byte-identically. +- A scoped viewing key on specific trades then confirms the disclosed records reconcile against the complete anchored set: nothing omitted, nothing inserted. +- A supervisor repeats the re-run a year later from the on-chain anchors alone, without the issuer's cooperation. + +## See also + +- [EIP-4444 (history expiry: why independent archives of chain history matter for deep re-runs)](https://eips.ethereum.org/EIPS/eip-4444) +- [Firehose documentation](https://firehose.streamingfast.io/) +- [TrueBlocks documentation](https://trueblocks.io/) diff --git a/vendors/the-graph.md b/vendors/the-graph.md new file mode 100644 index 0000000..3dc650b --- /dev/null +++ b/vendors/the-graph.md @@ -0,0 +1,87 @@ +--- +title: "Vendor: The Graph" +status: ready +website: https://thegraph.com +category: infrastructure +ethereum_aligned: true +last_reviewed: 2026-08-05 +maturity: production +--- + +# The Graph – Firehose, Substreams, Subgraphs (decentralized indexing and query infrastructure) + +## What it is + +An open source indexing stack. 76 networks are supported for subgraph indexing; on 23 of them (18 mainnets, including Ethereum) subgraphs are served by a decentralized network of independent, GRT-staked indexers. The Graph indexes public chain state and is not a privacy technology. Its role in privacy architectures is the read and audit path: deterministic, re-executable extraction and serving of the public artifacts confidential systems emit (commitments, nullifiers, anchored roots, attestation logs), so auditors and regulators can check disclosures against an independently recomputable record of on-chain emissions. + +## Fits with patterns + +- [Pattern: Reproducible Audit Extraction](../patterns/pattern-reproducible-audit-extraction.md) +- [Pattern: Compliance Monitoring](../patterns/pattern-compliance-monitoring.md) +- [Pattern: Low-cost L2 + Off-chain Encrypted Audit Log](../patterns/pattern-l2-encrypted-offchain-audit.md) +- [Pattern: Selective Disclosure (Viewing Keys + Zero-Knowledge Proofs)](../patterns/pattern-regulatory-disclosure-keys-proofs.md) +- [Pattern: Verifiable Attestation](../patterns/pattern-verifiable-attestation.md) +- [Pattern: Private Information Retrieval](../patterns/pattern-private-information-retrieval.md) + +## Not a substitute for + +- Privacy layers (shielded pools, privacy L2s, FHE or TEE execution); it adds no confidentiality. +- Reserve or solvency attestation; off-chain assets cannot be proven from chain data. +- In-zone operator privacy on permissioned ledgers with no public state to index. +- Price-feed oracles; it serves chain-derived facts, never off-chain observations. +- KYT screening and sanctions intelligence. + +## Architecture + +- **Subgraphs** are the indexing data service: mappings turn contract events into a typed entity store queried over GraphQL. On the decentralized network since 2020: independent indexers serve each deployment with proofs of indexing and signed response attestations. +- **Substreams** is the streaming transform engine: deterministic, composable Rust modules over raw block data with high-throughput parallel backfill, feeding SQL, file, and custom sinks. In production since 2023, run self-hosted or bought from a hosted operator. It is not yet a data service on the decentralized network, so a Substreams result carries no indexer attestation and no dispute path. Determinism is its trust mechanism: anyone can re-run it and compare. The network supplies a different one, attributability, where an indexer stakes collateral against the answer it signs. Subgraphs on the network have both. +- **Firehose** is the extraction layer beneath Substreams, also usable by other indexing systems: full chain history captured as deterministic, content-addressed flat files surviving execution-layer history pruning (EIP-4444); open source tooling recomputes receipt and transaction roots from extracted data and proves pre-Merge block inclusion against Ethereum's canonical header accumulator. +- **Decentralized network:** publishing is permissionless; indexers stake GRT and serve queries for fees. Paid responses carry EIP-712 signed attestations binding request and response hashes to the indexer's staked allocation; conflicting attestations ground an on-chain dispute settled by arbitration with slashing. Attestations are signatures, not validity proofs: responses become non-repudiable and slashable. +- **Gateways:** optional open source routing; anyone can operate one; clients can query indexers directly or pin one to cross-check. +- **Hosted delivery** is the common enterprise path: a single operator runs Firehose, Substreams, or subgraph APIs under contract. Open source core, but deployment, billing and API layers are proprietary; one operator can exclude a customer and attestations become contractual terms. The self-hosting exit stays open but is operationally heavy. + +## Privacy domains + +- **Stored data: none.** Indexed data is public chain state, served as-is. +- **Query privacy: absent.** The serving operator sees what a client queries; the leak described in the [Private Read use case](../use-cases/private-read.md). +- **Encrypted payload anchoring.** Client-side-encrypted records anchored on-chain are indexable as ciphertext, establishing existence, ordering, and completeness without plaintext access; keys stay with the owner. + +## Enterprise demand and use cases + +- **Audit and disclosure read path:** an institution keeps records private but publishes a tamper-evident fingerprint of each on-chain. Its auditor, shown a sample, must establish the sample is complete rather than curated. An indexed, independently re-derivable record of every published fingerprint lets the auditor check both directions: each disclosed record matches a published entry, and no published entry is missing from the disclosure. +- **Compliance monitoring substrate:** indexed views and evidence exports feeding rule engines and case management. +- **Market and risk monitoring** for institutions and supervisors needing a non-conflicted source. +- **Application backends:** the incumbent, high-volume use; regulated deployments typically buy through hosting operators. + +## Technical details + +- Deterministic end to end: content-addressed inputs and pinned code versions yield byte-identical re-runs. +- Reorg-aware; resumable cursors; historical queries at past blocks. +- Serving surfaces: GraphQL APIs, gRPC streams, SQL and file sinks. +- Licenses: Apache 2.0 (firehose-core, substreams, graph-node), MIT (indexer components), GPL-2.0 (protocol contracts). + +## Strengths + +- **Neutrality:** many independent operators; no affiliated exchange, analytics, or trading business. +- **Re-performability:** auditors re-run the open pipeline and compare hashes instead of trusting served output. +- **Accountable responses:** network responses are signed and disputable with staked collateral; conventional data API answers are deniable. +- **Coverage and maturity:** live since 2020; per-network status in the [networks registry](https://networks-registry.thegraph.com/). +- **Exit paths:** open licenses, public underlying data, self-hosting. + +## Risks and open questions + +- Query results are attested, not proof-carrying: extraction-level verification exists, but validity proofs for derived results are research. +- Completeness ceiling: an index attests on-chain emissions, never what went unrecorded nor off-chain reality; indexer output is not evidence of reserves or solvency. +- Read privacy: query patterns leak to the serving operator; multi-server private information retrieval over independent indexers is research. +- Serving plaintext is not on offer: The Graph serves ciphertext and the client decrypts, which works today and is an application concern. Indexers would need viewing keys only for general-purpose plaintext queries. Restricted predicates need no keys and largely work at the application layer today, since a deterministically encrypted field can be matched for equality as bytes; but such schemes buy selectivity by leaking structure (equality patterns, ordering), with a real inference-attack literature. General computation over ciphertext (FHE) is far from query-serving latency; enclave-based serving is research. +- Accountability is economic and procedural, not cryptographic: disputes are settled by a governance-appointed arbitrator, and protocol contracts are upgradeable under council governance. +- The decentralized network offers no SLA or SOC-type report; those come from hosting operators, which reintroduce single-operator trust. + +## Links + +- [https://thegraph.com](https://thegraph.com) +- [https://github.com/graphprotocol/graph-node](https://github.com/graphprotocol/graph-node) +- [https://github.com/streamingfast/firehose-core](https://github.com/streamingfast/firehose-core) +- [https://github.com/streamingfast/substreams](https://github.com/streamingfast/substreams) +- [https://github.com/graphprotocol/veemon](https://github.com/graphprotocol/veemon) +