Add deterministic token counting route (#20)

[kiki830621]

Refs #20

Summary

• add macdoc convert --to tokens for exact offline GPT-4o and provider-reported Claude input-token counts
• add reusable TokenCounter package with typed models/results/errors and public injectable Anthropic transport
• bundle and SHA-256 verify the o200k_base vocabulary; no runtime download/cache/HOME writes
• require per-invocation --allow-network and a non-empty ANTHROPIC_API_KEY before sending the complete admitted file to Anthropic
• provide deterministic single-number/TSV output with all-or-nothing stdout/file presentation

Boundary hardening

• input uses O_NOFOLLOW + fstat + same-descriptor capped read; maximum 1,000,000 valid UTF-8 bytes
• URLSession data delegate retains at most 64 KiB plus one detection byte and cancels on first overflow callback; no independently producing unbounded bridge
• public injected transport errors are normalized/redacted and injected response Data is independently size bounded
• cancellation and redirect retain typed outcomes through provider/service/CLI layers
• all requested providers must succeed before presentation; cancellation before presentation writes nothing

Verification

• TokenCounter package: 30 tests, 0 failures (including 5 official Python tiktoken vectors)
• TokenCountCommandTests: 22 tests, 0 failures
• full root suite with verified prerequisite PR Restore clean-clone WordToMD dependency #165 temporarily overlaid: 53 XCTest (0 failures, 4 fixture/environment skips) + 26 Swift Testing (0 failures)
• spectra validate macdoc-token-counting --strict
• root and nested swift package describe --type json
• vocabulary SHA-256 446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d
• git diff --check origin/main...HEAD
• five independent reviewer roles: requirements, concurrency/delegate, privacy, regression, public API — PASS

No live Anthropic request was made. Test credentials and source markers are synthetic and verified not to escape errors.

Depends on PR #165 for clean-clone root dependency resolution; that prerequisite is not included in this branch.

[kiki830621]

Verify: #20 — Round 3 PASS

Candidate

• Branch: idd/20-token-counting
• Head: b6c86e1
• PR: Add deterministic token counting route (#20) #170
• Verified tag: idd-20-verified
• PR closingIssuesReferences: empty

Round 2 blocker closure

1. Receive-boundary claim: resolved by replacing production AsyncBytes with direct URLSessionDataDelegate handling. Package-owned storage is strictly capped at 64 KiB plus one overflow byte; a 2,000,000-byte immediate callback retained exactly 65,537 bytes and cancelled. Spectra now truthfully states that Foundation controls callback allocation.
2. Injected-error disclosure: resolved by normalizing every non-package injected error to a redacted package-owned error and independently bounding injected response Data before decode.

Additional Round 3 findings resolved

• Production loader and response-body cancellation had been mapped to .providerFailure(0); direct RED now proves CancellationError is preserved.
• Production URLSession redirects ended as URLError.cancelled; a URLProtocol→delegate RED now proves .redirectRejected survives the exactly-once completion race.
• Non-cooperative provider cancellation now fails before presentation and writes no stdout, diagnostic, or output file.

Test evidence

• packages/token-counter-swift: 30 tests, 0 failures.
  • Public API 4/4, service 6/6, Anthropic 15/15, OpenAI 4/4 plus import contract.
  • Five official Python tiktoken reference vectors: empty, ASCII,台灣正體中文, mixed script, emoji.
• TokenCountCommandTests: 22 tests, 0 failures.
• Full root suite with verified prerequisite bug: clean clone baseline regressed through ignored word-to-md-swift path dependency #164 temporarily overlaid after a clean build: 53 XCTest, 0 failures, 4 fixture/environment skips; 26 Swift Testing, 0 failures. Overlay aborted and is not in feat: add token counting (OpenAI tiktoken + Claude API) #20.
• Root and nested swift package describe --type json: pass.
• spectra validate macdoc-token-counting --strict: valid.
• Bundled vocabulary SHA-256: 446a9538cb6c348e3516120d7c08b09f57c36495e2acfffe59a5bf8b0cfb1a2d.
• git diff --check origin/main...HEAD: pass.
• Secret/private-path scan excluding the audited vocabulary: pass.
• No live Anthropic request was made.

Requirement coverage

• Exact model allow-list, consent/key preflight, fixed Anthropic request, typed status/timeout/redirect/size errors, offline GPT resource integrity, descriptor-safe input, deterministic output, atomic failure/cancellation presentation, and user-facing privacy/platform docs all satisfy the final Spectra contract.

Independent review

• Requirements/spec review: PASS after production cancellation correction.
• Delegate/concurrency devil's advocate: PASS after production redirect correction.
• Privacy/error/output review: PASS.
• Regression/SwiftPM/resource review: PASS.
• Public API/provider semantics review: PASS.
• Root verifier: PASS.

The available reviewer harness was Codex-family only, so this run does not claim true cross-model diversity.

Residue / prerequisite

• Anthropic values remain provider-reported estimates and external availability is not guaranteed.
• PR Add deterministic token counting route (#20) #170 depends on clean-clone prerequisite PR Restore clean-clone WordToMD dependency #165; Restore clean-clone WordToMD dependency #165 is not included in this branch.

Verdict: PASS — verified-gated, ready for human review; not merged and issue remains open.