docs: cross-file coherence pass — endpoint costs, crawl intervals, retention, DVM format

Audited every doc (README, IMPACT-STATEMENT, DEPLOY.md, INTEGRATION.md,
methodology.html, index.html, mcp-server.json, .env.example) against the
source of truth (src/config/*.ts, src/routes/*.ts, src/nostr/dvm.ts, live
prod) and fixed the remaining inconsistencies.

README.md
- `/api/agent/{hash}/attestations` listed as free → corrected to **1 sat**.
  Source of truth: `src/routes/attestation.ts:20` wires the endpoint
  through the `apertureGateAuth` middleware, and live prod responds 402
  without an L402 token. INTEGRATION.md already had this right.

public/index.html
- `/api/agent/{hash}/attestations` added to the "Detailed Queries (1 sat)"
  section so the landing page matches README's endpoint table.

IMPACT-STATEMENT.md
- Business-model table: `/api/agent/:hash/attestations` added to the 1-sat
  row alongside the other detailed queries.

public/methodology.html
- "Unique Position" list: added **Brainstorm** to the other NIP-85
  implementations (it was in IMPACT-STATEMENT/README but missing here).
  Rewrote the paragraph to highlight the live-verified interop on the
  shared canonical `rank` tag.
- DVM section: replaced the vague "parameter target" description with the
  actual request format — `["j", "trust-check"]` + `["i", "<ln_pubkey>",
  "text"]` — matching `src/nostr/dvm.ts` constants and the README. Added
  the kind 31990 handler-info publish note.

DEPLOY.md
- Section 7 crawl-intervals table: `CRAWL_INTERVAL_PROBE_MS` 3,600,000 →
  **1,800,000** (30 min) and `PROBE_MAX_PER_SECOND` 10 → **15**. Both
  defaults now match `src/config.ts` and `.env.example`.
- Section 8 snapshot-retention: rewritten from scratch. The old text
  described a 3-tier (<7d / 7-30d / >30d) policy embedded in `runCrawl()`,
  none of which is accurate. The real policy is a flat 45-day cutoff on
  `score_snapshots` (with separate 14-day cutoffs for probe/channel/fee
  snapshots) applied by a dedicated 24-hour cron inside the crawler
  process, using 50k-row chunks to keep the SQLite WAL bounded. Source:
  `src/config/retention.ts`.
- Section 1 Aperture config snippet: replaced the `YOUR_VOLTAGE_NODE.
  voltageapp.io` placeholder host with `localhost:10009`, and
  `tlscertpath` / `macaroonpath` now point directly at `<LND_DATA_DIR>/...`
  so Aperture picks up an LND cert regeneration via a simple `systemctl
  restart aperture`.
- Section 2 nginx config: rewritten to match the real prod routing. The
  old regex `^/api/(agent|agents|decide|profile)` would have captured
  `/api/agents/top` and sent it through Aperture (making it 1-sat), but
  that endpoint is free in README + landing page + prod. New regex
  `^/api/agent/[a-f0-9]+` only matches paths with a hex hash after
  `/api/agent/`, so `/api/agents/top` correctly falls through to Express.
  Routing-logic bullets rewritten to match endpoint-by-endpoint.
- Section 7 post-crawl note: removed the "top 50 agents" claim (bulk
  scoring touches every eligible agent, not a top-N subset).

src/crawler/lndGraphCrawler.ts
- Top-of-file comment: stale `~17,000 nodes` → `~14,000 active Lightning
  nodes after UTXO validation`.

Tests: 464 / 34 files green.

Author: proofoftrust21

DEPLOY.md

@@ -40,9 +40,13 @@ autocert: false
 
 authenticator:
   lnd:
-    host: "YOUR_VOLTAGE_NODE.voltageapp.io:10009"
-    macaroonpath: "/etc/aperture/admin.macaroon"
-    tlscertpath: "/etc/aperture/tls.cert"
+    # gRPC host of the LND node Aperture uses to mint L402 invoices.
+    # Prefer "localhost:10009" when LND runs on the same host — this avoids
+    # copying macaroons/certs around and picks up LND cert regenerations
+    # automatically (Aperture re-reads `tlspath` on restart).
+    host: "localhost:10009"
+    macaroonpath: "<LND_DATA_DIR>/data/chain/bitcoin/mainnet/admin.macaroon"
+    tlscertpath: "<LND_DATA_DIR>/tls.cert"
 
 servicesettings:
   - name: "satrank"
@@ -60,8 +64,8 @@ dbdir: "/var/lib/aperture"
 - `price: 1` — 1 satoshi per query
 - `duration: 31536000` — L402 token valid for 1 year (365 days in seconds)
 - `pathregexp` — only agent/agents endpoints require payment; health/stats/version are free
-- Copy your LND admin macaroon to `/etc/aperture/admin.macaroon`
-- Copy your LND TLS cert to `/etc/aperture/tls.cert`
+- Replace `<LND_DATA_DIR>` with the absolute path to your LND data directory (where `tls.cert` and `data/chain/bitcoin/mainnet/admin.macaroon` live). Pointing Aperture at LND's live files instead of a copy means a cert regeneration on the LND side is picked up by `systemctl restart aperture` without any file juggling.
+- `host: "localhost:10009"` assumes LND is on the same host and listens on the loopback interface (recommended — see the `restlisten=127.0.0.1:10009` convention in `lnd.conf`). If LND is remote, replace with `hostname:port` and add that IP to LND's `tlsextraip`.
 
 ### Systemd: /etc/systemd/system/aperture.service
 
@@ -104,54 +108,81 @@ sudo systemctl enable --now aperture
 ### /etc/nginx/sites-available/satrank.dev
 
 ```nginx
+server {
+    listen 80;
+    server_name satrank.dev;
+    return 301 https://$host$request_uri;
+}
+
 server {
     listen 443 ssl http2;
     server_name satrank.dev;
 
     ssl_certificate /etc/letsencrypt/live/satrank.dev/fullchain.pem;
     ssl_certificate_key /etc/letsencrypt/live/satrank.dev/privkey.pem;
 
-    # L402-gated endpoints — proxy through Aperture
-    # Matches: /api/agent/*, /api/agents/*, /api/decide, /api/profile/*
-    location ~ ^/api/(agent|agents|decide|profile) {
-        proxy_pass http://127.0.0.1:8443;
+    # L402-gated endpoints — proxy through Aperture.
+    # `/api/agent/{hash}` and `/api/agent/{hash}/{verdict,history,attestations}`
+    # all match because the regex requires a 64-hex-char id after `/agent/`.
+    # `/api/agents/top`, `/api/agents/movers`, `/api/agents/search` do NOT match
+    # (no hex id after `/agents/`) — they fall through to Express and are free.
+    location ~ ^/api/agent/[a-f0-9]+ {
+        proxy_pass https://127.0.0.1:8443;
+        proxy_ssl_verify off;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
     }
 
-    # Free endpoint — report goes direct to Express (API key auth, no L402)
-    location = /api/report {
-        proxy_pass http://127.0.0.1:3000;
+    # Explicit paid routes (exact match on /api/decide, prefix on /api/profile/).
+    location = /api/decide {
+        proxy_pass https://127.0.0.1:8443;
+        proxy_ssl_verify off;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
     }
 
-    # Free endpoints — direct to Express (health, stats, attestations, etc.)
-    location /api/ {
+    location ~ ^/api/profile/ {
+        proxy_pass https://127.0.0.1:8443;
+        proxy_ssl_verify off;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+
+    # Free endpoint — report goes direct to Express (API key auth, no L402).
+    location = /api/report {
         proxy_pass http://127.0.0.1:3000;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
     }
 
-    # Static assets (landing page, favicon)
+    # NIP-05 (.well-known/nostr.json) — public JSON, needs CORS.
+    location /.well-known/nostr.json {
+        proxy_pass http://127.0.0.1:3000;
+        proxy_set_header Host $host;
+        add_header Access-Control-Allow-Origin * always;
+    }
+
+    # Catch-all — every non-paid `/api/*` (health, stats, agents/top,
+    # agents/movers, agents/search, ping, verdicts, attestations, docs,
+    # openapi.json), plus static assets (landing page, methodology, icons).
+    # `/api/verdicts` is L402-gated at the Express level via `apertureGateAuth`,
+    # so reaching Express direct returns 402 for external callers.
     location / {
         proxy_pass http://127.0.0.1:3000;
         proxy_set_header Host $host;
         proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
     }
 }
-
-server {
-    listen 80;
-    server_name satrank.dev;
-    return 301 https://$host$request_uri;
-}
 ```
 
 ```bash
@@ -160,11 +191,15 @@ sudo certbot --nginx -d satrank.dev
 sudo nginx -t && sudo systemctl reload nginx
 ```
 
-**Routing logic:**
-- `/api/agent/*`, `/api/agents/*`, `/api/decide`, `/api/profile/*` → Aperture (L402, 1 sat)
-- `/api/report` → Express direct (API key auth, free)
-- `/api/health`, `/api/stats`, `/api/attestations`, `/api/docs` → Express direct (free)
-- `/`, `/app.js`, `/favicon.png` → Express static (free)
+**Routing logic (matches README + landing page cost tables):**
+- `/api/agent/{hash}`, `/api/agent/{hash}/verdict`, `/api/agent/{hash}/history`, `/api/agent/{hash}/attestations` → nginx → Aperture → Express (L402, 1 sat)
+- `/api/decide` → nginx → Aperture → Express (L402, 1 sat)
+- `/api/profile/{id}` → nginx → Aperture → Express (L402, 1 sat)
+- `/api/verdicts` → nginx → Express direct, gated at Express by `apertureGateAuth` middleware (L402, 1 sat/batch — the middleware returns 402 for non-loopback callers)
+- `/api/report`, `/api/attestations` → nginx → Express direct (free, X-API-Key required)
+- `/api/health`, `/api/stats`, `/api/ping/{pubkey}`, `/api/agents/top`, `/api/agents/movers`, `/api/agents/search`, `/api/docs`, `/api/openapi.json` → nginx → Express direct (free, no auth)
+- `/`, static assets, `/methodology.html` → nginx → Express static (free)
+- `/.well-known/nostr.json` → nginx → Express direct (free, CORS enabled for NIP-05)
 
 ## 3. SatRank (Docker)
 
@@ -338,8 +373,8 @@ Each data source runs on its own timer in `--cron` mode. At startup, a full craw
 | `CRAWL_INTERVAL_OBSERVER_MS` | `300000` (5 min) | Observer Protocol transactions |
 | `CRAWL_INTERVAL_LND_GRAPH_MS` | `3600000` (1 hour) | LND full graph (~14k active nodes on mainnet) |
 | `CRAWL_INTERVAL_LNPLUS_MS` | `86400000` (24 hours) | LN+ community ratings |
-| `CRAWL_INTERVAL_PROBE_MS` | `3600000` (1 hour) | Route probe (reachability check) |
-| `PROBE_MAX_PER_SECOND` | `10` | Max probes per second (rate limiter) |
+| `CRAWL_INTERVAL_PROBE_MS` | `1800000` (30 min) | Route probe (reachability check) |
+| `PROBE_MAX_PER_SECOND` | `15` | Max probes per second (rate limiter) |
 | `PROBE_AMOUNT_SATS` | `1000` | Amount in sats to test routes with |
 
 Override in `.env.production`:
@@ -356,16 +391,32 @@ CRAWL_INTERVAL_LND_GRAPH_MS=21600000    # 6 hours
 CRAWL_INTERVAL_LNPLUS_MS=86400000       # 24 hours
 ```
 
-After each Observer and LND crawl, scores are pre-computed for the top 50 agents and old snapshots are purged.
+After each LND crawl, the scoring pipeline runs in two passes: first any unscored
+agents that accumulated since the last cycle (bulk scoring, unscored), then all
+previously scored agents are rescored with fresh data (bulk rescore). On a normal
+cycle this touches every eligible agent in the index, not a top-N subset. Logs
+show exact counts (`scored X/Y errors=0`) for each pass.
 
 ## 8. Snapshot retention
 
-The `score_snapshots` table grows with each score computation (~50 agents every 5 minutes = ~14,400 rows/day).
-The crawler automatically purges old snapshots after each crawl run:
-
-- **< 7 days**: all snapshots retained
-- **7–30 days**: 1 snapshot per agent per day (deduplication via `ROW_NUMBER()`)
-- **> 30 days**: deleted
-
-This runs inside `runCrawl()` at the end of each cycle (cron or single run).
-No additional cron job is needed — the purge is embedded in the crawler process.
+The `score_snapshots` table grows with every score computation — on the production
+mainnet instance each cycle writes ~7,000+ rows, and the retention cron keeps the
+last 45 days.
+
+The retention policy is defined in `src/config/retention.ts` and applied by a
+dedicated cron inside the crawler process (not embedded in each crawl):
+
+- **`RETENTION_POLICIES`** (flat cutoff per table):
+  - `probe_results`       — 14 days (regularity uses the last 7, kept 2× for margin)
+  - `score_snapshots`     — **45 days** (delta windows up to 30d, kept 1.5× for margin)
+  - `channel_snapshots`   — 14 days
+  - `fee_snapshots`       — 14 days
+- **`RETENTION_CHUNK_SIZE`** — deletes run in chunks of 50,000 rows per
+  transaction so the SQLite WAL never balloons (previous multi-million-row
+  monolithic `DELETE` grew the WAL past 1 GB and stalled — that's why chunking).
+- **`RETENTION_INTERVAL_MS`** — the retention cron runs every **24 hours**,
+  independent from the crawler's data ingestion cycle. At startup it also runs
+  once immediately.
+
+The retention cron is started from `src/crawler/run.ts` and logs each table's
+`{deleted, durationMs}` after every sweep.

IMPACT-STATEMENT.md

@@ -207,7 +207,7 @@ The freemium architecture is limpid and already profitable at the margin:
 | **NIP-85 scores (kind 30382 on 3 relays)** | **free** | Distribution, adoption, composability with other NIP-85 providers |
 | **NIP-05, NIP-90 DVM, `/api/ping`, `/api/agents/top`, `/api/stats`, `/api/health`** | **free** | Discovery, live reachability, public statistics |
 | **`/api/decide` (personalized GO/NO-GO + pathfinding)** | **1 sat via L402** | Primary revenue — the personalized oracle call |
-| **`/api/profile`, `/api/agent/:hash`, `/api/agent/:hash/verdict`, `/api/agent/:hash/history`, `/api/verdicts`** | **1 sat via L402** | Detailed queries — secondary revenue |
+| **`/api/profile`, `/api/agent/:hash`, `/api/agent/:hash/verdict`, `/api/agent/:hash/history`, `/api/agent/:hash/attestations`, `/api/verdicts`** | **1 sat via L402** | Detailed queries — secondary revenue |
 | **`/api/report`, `/api/attestations`** | **free** (API key for identity) | Closes the feedback loop — reports improve `P_empirical` in future decide responses |
 
 Why it works:

README.md

@@ -156,7 +156,7 @@ curl https://satrank.dev/api/agent/<hash>/verdict
 | GET | `/api/agent/:hash` | Detailed score + evidence | 1 sat |
 | GET | `/api/agent/:hash/verdict` | SAFE/RISKY/UNKNOWN + flags + risk profile | 1 sat |
 | GET | `/api/agent/:hash/history` | Score snapshots with deltas | 1 sat |
-| GET | `/api/agent/:hash/attestations` | Received attestations | free |
+| GET | `/api/agent/:hash/attestations` | Received attestations (list) | 1 sat |
 | GET | `/api/agents/top` | Leaderboard by score | free |
 | GET | `/api/agents/movers` | Top 7-day movers | free |
 | GET | `/api/agents/search?alias=…` | Search by alias | free |

public/index.html

@@ -260,6 +260,11 @@ <h2>Detailed Queries <span class="api-tag paid-tag">1 sat</span></h2>
         <span class="endpoint-path">/api/agent/:hash/history</span>
         <span class="endpoint-desc">Score history</span>
       </li>
+      <li>
+        <span class="method method-get">GET</span>
+        <span class="endpoint-path">/api/agent/:hash/attestations</span>
+        <span class="endpoint-desc">Received attestations (list)</span>
+      </li>
       <li>
         <span class="method method-post">POST</span>
         <span class="endpoint-path">/api/verdicts</span>

public/methodology.html

@@ -970,20 +970,25 @@ <h3>Why Nostr?</h3>
   <h3>DVM Trust-Check (NIP-90)</h3>
   <p>
     SatRank also operates a Data Vending Machine (NIP-90) for real-time trust queries.
-    Send a kind <code>5900</code> event with parameter <code>target</code> set to any
-    Lightning node pubkey, and receive a kind <code>6900</code> response with the trust
-    score, verdict, and reachability. Unknown nodes trigger an on-demand <code>QueryRoutes</code>
-    probe so the answer reflects the live graph, not just the cached snapshot.
+    Publish a kind <code>5900</code> job request with <code>["j", "trust-check"]</code>
+    and <code>["i", "&lt;ln_pubkey&gt;", "text"]</code> tags, and SatRank responds with
+    a signed kind <code>6900</code> event containing the trust score, verdict, and
+    reachability. Unknown nodes trigger an on-demand <code>QueryRoutes</code> probe so
+    the answer reflects the live graph, not just the cached snapshot. The DVM is
+    discoverable via a kind <code>31990</code> handler-info event published to the
+    three canonical relays on every startup. Free, no payment required.
   </p>
 
   <h3>Unique Position</h3>
   <p>
-    Every other NIP-85 implementation (wot-scoring, nostr-wot-sdk, nostr-wot-oracle, Vertex)
-    operates on the Nostr social graph &mdash; scoring based on follows, mutes, and zaps.
-    SatRank is the only provider that bridges the Lightning payment graph into NIP-85. That
-    means a wallet or agent can query <strong>both</strong> social trust and payment
-    reliability from the same Nostr relay infrastructure, without running two parallel
-    pipelines.
+    Every other NIP-85 implementation (Brainstorm, wot-scoring, nostr-wot-sdk,
+    nostr-wot-oracle, Vertex) operates on the Nostr social graph &mdash; scoring based on
+    follows, mutes, and zaps. SatRank is the only provider that bridges the Lightning
+    payment graph into NIP-85. Because Brainstorm and SatRank both use the canonical
+    <code>rank</code> tag on kind <code>30382</code> events, a wallet or agent can list
+    both in its kind <code>10040</code> and receive social trust <em>and</em> payment
+    reliability from the same Nostr relay connection &mdash; verified live on
+    <code>relay.damus.io</code> and <code>nos.lol</code>, where both providers publish.
   </p>
 
   <h3 id="declare-provider">Declaring SatRank as a Trusted Provider (kind 10040)</h3>

src/crawler/lndGraphCrawler.ts

@@ -1,5 +1,6 @@
-// Indexes Lightning Network nodes from our LND node's graph view
-// Primary source — replaces mempool.space for full graph coverage (~17,000 nodes)
+// Indexes Lightning Network nodes from our LND node's graph view.
+// Primary source — replaces mempool.space for full graph coverage.
+// Mainnet today: ~14,000 active Lightning nodes after UTXO validation.
 import type { AgentRepository } from '../repositories/agentRepository';
 import type { ChannelSnapshotRepository } from '../repositories/channelSnapshotRepository';
 import type { FeeSnapshotRepository } from '../repositories/feeSnapshotRepository';