perf: smart-wait — waitForCapture, wait({ selector }), daemon backoff

- waitForCapture(): polls window.__opencli_xhr instead of DOM-stable; fixes INTERCEPT adapters returning empty after smart-wait refactor
- wait({ selector }): MutationObserver-based wait; resolves instantly on element insertion
- CDPPage.wait(N): smart DOM-stable wait (matches Page.wait behavior)
- Daemon cold-start: exponential backoff [50..3000ms]
- README: simplified to 50-line overview

Author: jackwener

README.md

@@ -0,0 +1,170 @@
+# Performance: Smart Wait & INTERCEPT Fix
+
+**Date**: 2026-03-28
+**Status**: Approved
+
+## Problem
+
+Three distinct performance/correctness issues:
+
+1. **INTERCEPT strategy semantic bug**: After `installInterceptor()` + `goto()`, adapters call `wait(N)` — which now uses `waitForDomStableJs` and returns early when the DOM settles. But DOM-settle != network capture. The API response may arrive *after* DOM is stable, causing `getInterceptedRequests()` to return an empty array.
+
+2. **Blind `wait(N)` in adapters**: ~30 high-traffic adapters (Twitter family, Medium, Substack, etc.) call `wait(5)` waiting for React/Vue to hydrate. These should wait for a specific DOM element to appear, not a fixed cap.
+
+3. **Daemon cold-start polling**: Fixed 300ms poll loop means ~600ms before first successful `isExtensionConnected()` check, even though the daemon is typically ready in 500–800ms.
+
+## Design
+
+### Layer 1 — `waitForCapture()` (correctness fix + perf)
+
+Add `waitForCapture(timeout?: number): Promise<void>` to `IPage`.
+
+Polls `window.__opencli_xhr.length > 0` every 100ms inside the browser tab. Resolves as soon as ≥1 capture arrives; rejects after `timeout` seconds.
+
+```typescript
+// dom-helpers.ts
+export function waitForCaptureJs(maxMs: number): string {
+  return `
+    new Promise((resolve, reject) => {
+      const deadline = Date.now() + ${maxMs};
+      const check = () => {
+        if ((window.__opencli_xhr || []).length > 0) return resolve('captured');
+        if (Date.now() > deadline) return reject(new Error('No capture within ${maxMs / 1000}s'));
+        setTimeout(check, 100);
+      };
+      check();
+    })
+  `;
+}
+```
+
+`page.ts` and `cdp.ts` implement `waitForCapture()` by calling `waitForCaptureJs`.
+
+**All INTERCEPT adapters** replace `wait(N)` → `waitForCapture(N+2)` (slightly longer timeout as safety margin).
+
+`stepIntercept` in `pipeline/steps/intercept.ts` replaces its internal `wait(timeout)` with `waitForCapture(timeout)`.
+
+**Expected gain**: 36kr hot/search: 6s → ~1–2s. Twitter search/followers: 5–8s → ~1–3s.
+
+### Layer 2 — `wait({ selector })` (semantic precision)
+
+Extend `WaitOptions` with `selector?: string`.
+
+Add `waitForSelectorJs(selector, timeoutMs)` to `dom-helpers.ts` — polls `document.querySelector(selector)` every 100ms, resolves on first match, rejects on timeout.
+
+```typescript
+// types.ts
+export interface WaitOptions {
+  text?: string;
+  selector?: string;   // NEW
+  time?: number;
+  timeout?: number;
+}
+```
+
+```typescript
+// dom-helpers.ts
+export function waitForSelectorJs(selector: string, timeoutMs: number): string {
+  return `
+    new Promise((resolve, reject) => {
+      const deadline = Date.now() + ${timeoutMs};
+      const check = () => {
+        if (document.querySelector(${JSON.stringify(selector)})) return resolve('found');
+        if (Date.now() > deadline) return reject(new Error('Selector not found: ' + ${JSON.stringify(selector)}));
+        setTimeout(check, 100);
+      };
+      check();
+    })
+  `;
+}
+```
+
+`page.ts` and `cdp.ts` handle `selector` branch in `wait()`.
+
+**High-impact adapter changes**:
+
+| Adapter | Old | New |
+|---------|-----|-----|
+| `twitter/*` (15 adapters) | `wait(5)` | `wait({ selector: '[data-testid="primaryColumn"]', timeout: 6 })` |
+| `twitter/reply.ts` | `wait(5)` | `wait({ selector: '[data-testid="tweetTextarea_0"]', timeout: 8 })` |
+| `medium/utils.ts` | `wait(5)` + inline 3s setTimeout | `wait({ selector: 'article', timeout: 8 })` + remove inline sleep |
+| `substack/utils.ts` | `wait(5)` × 2 | `wait({ selector: 'article', timeout: 8 })` |
+| `bloomberg/news.ts` | `wait(5)` | `wait({ selector: 'article', timeout: 6 })` |
+| `sinablog/utils.ts` | `wait(5)` | `wait({ selector: 'article, .article', timeout: 6 })` |
+| `producthunt` (already covered by layer 1) | — | — |
+
+**Expected gain**: Twitter commands: 5s → ~0.5–2s. Medium: 8s → ~1–3s.
+
+### Layer 3 — Daemon exponential backoff (cold-start)
+
+Replace fixed 300ms poll in `_ensureDaemon()` (`browser/mcp.ts`) with exponential backoff:
+
+```typescript
+// before
+while (Date.now() < deadline) {
+  await new Promise(resolve => setTimeout(resolve, 300));
+  if (await isExtensionConnected()) return;
+}
+
+// after
+const backoffs = [50, 100, 200, 400, 800, 1500, 3000];
+let i = 0;
+while (Date.now() < deadline) {
+  await new Promise(resolve => setTimeout(resolve, backoffs[Math.min(i++, backoffs.length - 1)]));
+  if (await isExtensionConnected()) return;
+}
+```
+
+**Expected gain**: First cold-start check succeeds at ~150ms instead of ~600ms.
+
+## Files Changed
+
+### New / Modified (framework)
+- `src/types.ts` — `WaitOptions.selector`, `IPage.waitForCapture()`
+- `src/browser/dom-helpers.ts` — `waitForCaptureJs()`, `waitForSelectorJs()`
+- `src/browser/page.ts` — `waitForCapture()`, `wait()` selector branch
+- `src/browser/cdp.ts` — `waitForCapture()`, `wait()` selector branch
+- `src/browser/mcp.ts` — exponential backoff in `_ensureDaemon()`
+- `src/pipeline/steps/intercept.ts` — use `waitForCapture()`
+
+### Modified (adapters — Layer 1, INTERCEPT)
+- `src/clis/36kr/hot.ts`
+- `src/clis/36kr/search.ts`
+- `src/clis/twitter/search.ts`
+- `src/clis/twitter/followers.ts`
+- `src/clis/twitter/following.ts`
+- `src/clis/producthunt/hot.ts`
+- `src/clis/producthunt/browse.ts`
+
+### Modified (adapters — Layer 2, selector)
+- `src/clis/twitter/reply.ts`
+- `src/clis/twitter/follow.ts`
+- `src/clis/twitter/unfollow.ts`
+- `src/clis/twitter/like.ts`
+- `src/clis/twitter/bookmark.ts`
+- `src/clis/twitter/unbookmark.ts`
+- `src/clis/twitter/block.ts`
+- `src/clis/twitter/unblock.ts`
+- `src/clis/twitter/hide-reply.ts`
+- `src/clis/twitter/notifications.ts`
+- `src/clis/twitter/profile.ts`
+- `src/clis/twitter/thread.ts`
+- `src/clis/twitter/timeline.ts`
+- `src/clis/twitter/delete.ts`
+- `src/clis/twitter/reply-dm.ts`
+- `src/clis/medium/utils.ts`
+- `src/clis/substack/utils.ts`
+- `src/clis/bloomberg/news.ts`
+- `src/clis/sinablog/utils.ts`
+
+## Delivery Order
+
+1. Layer 1 (`waitForCapture`) — correctness fix, highest ROI
+2. Layer 3 (backoff) — 3-line change, zero risk
+3. Layer 2 (`wait({ selector })`) — largest adapter surface, can be done per-site
+
+## Testing
+
+- Unit tests: `waitForCaptureJs`, `waitForSelectorJs` exported and tested in `dom-helpers.test.ts` (if exists) or new test file
+- Adapter tests: existing tests must continue to pass (mock `page.wait` / `page.waitForCapture`)
+- Run: `npx vitest run --project unit --project adapter`

docs/superpowers/plans/2026-03-28-perf-smart-wait.md

@@ -25,6 +25,8 @@ import {
   autoScrollJs,
   networkRequestsJs,
   waitForDomStableJs,
+  waitForCaptureJs,
+  waitForSelectorJs,
 } from './dom-helpers.js';
 import { isRecord, saveBase64ToFile } from '../utils.js';
 
@@ -247,6 +249,15 @@ class CDPPage implements IPage {
 
   async wait(options: number | WaitOptions): Promise<void> {
     if (typeof options === 'number') {
+      if (options >= 1) {
+        try {
+          const maxMs = options * 1000;
+          await this.evaluate(waitForDomStableJs(maxMs, Math.min(500, maxMs)));
+          return;
+        } catch {
+          // Fallback: fixed sleep
+        }
+      }
       await new Promise((resolve) => setTimeout(resolve, options * 1000));
       return;
     }
@@ -255,6 +266,11 @@ class CDPPage implements IPage {
       await new Promise((resolve) => setTimeout(resolve, waitTime * 1000));
       return;
     }
+    if (options.selector) {
+      const timeout = (options.timeout ?? 10) * 1000;
+      await this.evaluate(waitForSelectorJs(options.selector, timeout));
+      return;
+    }
     if (options.text) {
       const timeout = (options.timeout ?? 30) * 1000;
       await this.evaluate(waitForTextJs(options.text, timeout));
@@ -326,6 +342,11 @@ class CDPPage implements IPage {
     const result = await this.evaluate(generateReadInterceptedJs('__opencli_xhr'));
     return Array.isArray(result) ? result : [];
   }
+
+  async waitForCapture(timeout: number = 10): Promise<void> {
+    const maxMs = timeout * 1000;
+    await this.evaluate(waitForCaptureJs(maxMs));
+  }
 }
 
 function isCookie(value: unknown): value is BrowserCookie {

docs/superpowers/specs/2026-03-28-perf-smart-wait-design.md

@@ -0,0 +1,100 @@
+import { describe, it, expect } from 'vitest';
+import { waitForCaptureJs, waitForSelectorJs } from './dom-helpers.js';
+
+describe('waitForCaptureJs', () => {
+  it('returns a non-empty string', () => {
+    const code = waitForCaptureJs(1000);
+    expect(typeof code).toBe('string');
+    expect(code.length).toBeGreaterThan(0);
+    expect(code).toContain('__opencli_xhr');
+    expect(code).toContain('resolve');
+    expect(code).toContain('reject');
+  });
+
+  it('resolves "captured" when __opencli_xhr is populated before deadline', async () => {
+    const g = globalThis as any;
+    g.__opencli_xhr = [];
+    g.window = g; // stub window for Node eval
+    const code = waitForCaptureJs(1000);
+    const promise = eval(code) as Promise<string>;
+    g.__opencli_xhr.push({ data: 'test' });
+    await expect(promise).resolves.toBe('captured');
+    delete g.__opencli_xhr;
+    delete g.window;
+  });
+
+  it('rejects when __opencli_xhr stays empty past deadline', async () => {
+    const g = globalThis as any;
+    g.__opencli_xhr = [];
+    g.window = g;
+    const code = waitForCaptureJs(50); // 50ms timeout
+    const promise = eval(code) as Promise<string>;
+    await expect(promise).rejects.toThrow('No network capture within 0.05s');
+    delete g.__opencli_xhr;
+    delete g.window;
+  });
+
+  it('resolves immediately when __opencli_xhr already has data', async () => {
+    const g = globalThis as any;
+    g.__opencli_xhr = [{ data: 'already here' }];
+    g.window = g;
+    const code = waitForCaptureJs(1000);
+    await expect(eval(code) as Promise<string>).resolves.toBe('captured');
+    delete g.__opencli_xhr;
+    delete g.window;
+  });
+});
+
+describe('waitForSelectorJs', () => {
+  it('returns a non-empty string', () => {
+    const code = waitForSelectorJs('#app', 1000);
+    expect(typeof code).toBe('string');
+    expect(code).toContain('#app');
+    expect(code).toContain('querySelector');
+    expect(code).toContain('MutationObserver');
+  });
+
+  it('resolves "found" immediately when selector already present', async () => {
+    const g = globalThis as any;
+    const fakeEl = { tagName: 'DIV' };
+    g.document = { querySelector: (_: string) => fakeEl };
+    const code = waitForSelectorJs('[data-testid="primaryColumn"]', 1000);
+    await expect(eval(code) as Promise<string>).resolves.toBe('found');
+    delete g.document;
+  });
+
+  it('resolves "found" when selector appears after DOM mutation', async () => {
+    const g = globalThis as any;
+    let mutationCallback!: () => void;
+    g.MutationObserver = class {
+      constructor(cb: () => void) { mutationCallback = cb; }
+      observe() {}
+      disconnect() {}
+    };
+    let calls = 0;
+    g.document = {
+      querySelector: (_: string) => (calls++ > 0 ? { tagName: 'DIV' } : null),
+      body: {},
+    };
+    const code = waitForSelectorJs('#app', 1000);
+    const promise = eval(code) as Promise<string>;
+    mutationCallback(); // simulate DOM mutation
+    await expect(promise).resolves.toBe('found');
+    delete g.document;
+    delete g.MutationObserver;
+  });
+
+  it('rejects when selector never appears within timeout', async () => {
+    const g = globalThis as any;
+    g.MutationObserver = class {
+      constructor(_cb: () => void) {}
+      observe() {}
+      disconnect() {}
+    };
+    g.document = { querySelector: (_: string) => null, body: {} };
+    const code = waitForSelectorJs('#missing', 50);
+    await expect(eval(code) as Promise<string>).rejects.toThrow('Selector not found: #missing');
+    delete g.document;
+    delete g.MutationObserver;
+  });
+});

src/browser/cdp.ts

@@ -179,3 +179,47 @@ export function waitForDomStableJs(maxMs: number, quietMs: number): string {
     })
   `;
 }
+
+/**
+ * Generate JS to wait until window.__opencli_xhr has ≥1 captured response.
+ * Polls every 100ms. Resolves 'captured' on success; rejects after maxMs.
+ * Used after installInterceptor() + goto() instead of a fixed sleep.
+ */
+export function waitForCaptureJs(maxMs: number): string {
+  return `
+    new Promise((resolve, reject) => {
+      const deadline = Date.now() + ${maxMs};
+      const check = () => {
+        if ((window.__opencli_xhr || []).length > 0) return resolve('captured');
+        if (Date.now() > deadline) return reject(new Error('No network capture within ${maxMs / 1000}s'));
+        setTimeout(check, 100);
+      };
+      check();
+    })
+  `;
+}
+
+/**
+ * Generate JS to wait until document.querySelector(selector) returns a match.
+ * Uses MutationObserver for near-instant resolution; falls back to reject after timeoutMs.
+ */
+export function waitForSelectorJs(selector: string, timeoutMs: number): string {
+  return `
+    new Promise((resolve, reject) => {
+      const sel = ${JSON.stringify(selector)};
+      if (document.querySelector(sel)) return resolve('found');
+      const cap = setTimeout(() => {
+        obs.disconnect();
+        reject(new Error('Selector not found: ' + sel));
+      }, ${timeoutMs});
+      const obs = new MutationObserver(() => {
+        if (document.querySelector(sel)) {
+          clearTimeout(cap);
+          obs.disconnect();
+          resolve('found');
+        }
+      });
+      obs.observe(document.body || document.documentElement, { childList: true, subtree: true });
+    })
+  `;
+}

src/browser/dom-helpers.test.ts

@@ -96,10 +96,11 @@ export class BrowserBridge implements IBrowserFactory {
     });
     this._daemonProc.unref();
 
-    // Wait for daemon to be ready AND extension to connect
+    // Wait for daemon to be ready AND extension to connect (exponential backoff)
+    const backoffs = [50, 100, 200, 400, 800, 1500, 3000];
     const deadline = Date.now() + timeoutMs;
-    while (Date.now() < deadline) {
-      await new Promise(resolve => setTimeout(resolve, 300));
+    for (let i = 0; Date.now() < deadline; i++) {
+      await new Promise(resolve => setTimeout(resolve, backoffs[Math.min(i, backoffs.length - 1)]));
       if (await isExtensionConnected()) return;
     }