docs: design for image input — paste a screenshot, ask about it - #89
Open
ndemianc wants to merge 2 commits into
Open
docs: design for image input — paste a screenshot, ask about it#89ndemianc wants to merge 2 commits into
ndemianc wants to merge 2 commits into
Conversation
The interaction being copied — ⌘V a screenshot into the composer, no upload step — is
the easy part. This documents the four things in our code that break quietly when the
first image goes through, and the cost model that decides how bytes should be shaped
before they leave the webview.
The four, each verified against develop rather than recalled:
1. conversation carries content as a STRING (extension.js:1752). There is no shape an
image can take. This is the structural change; everything else follows.
2. translate.js SILENTLY DROPS any block it doesn't recognise. On OpenAI-compatible
providers an attached image would vanish between composer and wire, and the model
would answer confidently about text it never saw — no error, no log line. A user
would reasonably conclude the model hallucinates. Worth fixing on its own merits,
before images, which is why it's slice I1.
3. `vision: true` is ALREADY in the catalog, per model, and nothing reads it. There's
a supportsToolsForModel and no supportsVisionForModel. Half the gate exists.
4. estimateMsgTokens is JSON.stringify(m).length / 4 — sound for text, catastrophic
for base64: a 1MB screenshot books ~333,000 phantom tokens, more than most context
windows. The same estimate drives findCompactionCut, so pasting one screenshot
would evict real conversation history. This is the bug that would have shipped as
"long conversations forget things after I paste an image".
ON THE NUMBERS. I checked the vision API rather than trusting my prior, and the prior
was wrong: cost is not w×h/750, it is ⌈w/28⌉ × ⌈h/28⌉ visual tokens over 28px patches,
with a per-tier cap (2576px/4784 tokens on 4.7+, 1568/1568 below). I reimplemented the
resize rule and reproduced the documented figures exactly — 1092² → 1521, 1000² → 1296,
1920×1080 → 2691 — so the cost table in §1 is arithmetic, not estimate.
That verification changed a decision. Token cost is ALREADY capped server-side, so
client-side downscaling is not a defence against a token blowup — it is a deliberate
fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire. The
doc says so rather than implying downscaling is load-bearing for cost safety.
Also recorded: writing §D3 I ran a 1160×480 capture through `sips -Z 1568` and it GREW,
40KB → 89KB, because the tool scaled it up to meet the cap. Never upscale — the rule is
min(1, cap/longEdge), and a factor of 1 means pass the original bytes through untouched,
which also avoids stacking compression artifacts on screenshots of text.
Seven slices, I1–I7, each independently shippable with bypass-verifiable exit criteria.
Deferred with reasons: Files API upload (Anthropic-only, wins on repeat turns), PDFs,
coordinates, client-side OCR.
Building a live cost calculator from the doc's numbers caught two things the prose had glossed. The resize rule is not an iterative shrink. My first implementation stepped the scale down by 1% until the patch grid fit, which gets the TOKEN COUNT right every time but misreports the sent dimensions — 1447x814 where the docs say 1456x819. Replaced with a binary search for the largest scale whose grid fits the cap, which is what the rule actually is. Checked against every worked example in the vision docs: token count matches on all twelve, dimensions on eleven. The twelfth is one standard-tier row a single pixel wide of the reference (1270 vs 1269, same 1564 tokens) — a rounding convention I could not derive from six data points, and the doc now says so rather than claiming the rule was reproduced "exactly". Two rows in the cost table were computed with the stepping version and are corrected: the macOS retina grab is 2380x1546 (not 2377x1544) and the 12 MP photo is 2212x1659 at 4740 tokens (not 2193x1645 / 4661). Both were mine, neither came from the docs. None of this moves a decision — the 1568 default and the "server already caps the cost" argument rest on the 4K row, which was right. But a plan whose own arithmetic disagrees with its live calculator is worse than one with no calculator.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a design document outlining how LevelCode’s AI chat should support image input (e.g., pasting screenshots), focusing on the current codebase constraints, a verified vision token cost model, and a staged implementation plan (I1–I7) to avoid silent failures and incorrect token accounting.
Changes:
- Introduces
docs/IMAGES.mddescribing the end-to-end image pipeline (webview normalization → host storage → provider request shaping). - Documents four current “quiet breakages” (string-only
content, silent block drops, unused vision gating, and naive token estimation) and their implications. - Specifies concrete design decisions (block widening, image-first ordering, normalization rules, content-addressed storage, and capability gating) plus slice-by-slice exit criteria.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| This is the most dangerous thing in the list. On any OpenAI-compatible provider — which is most of them through the gateway — an attached image would **vanish between the composer and the wire**, and the model would answer confidently about text it never saw. No error, no warning, no log line. A user would reasonably conclude the model is hallucinating. | ||
|
|
||
| **The vision capability is already modelled, and nothing reads it.** `providers/catalog.js` carries `vision: true` per model and has done since the multi-provider work: |
| } | ||
| ``` | ||
|
|
||
| Sound for text. For a base64 image it charges roughly **one third of the byte count as tokens** — a 1 MB screenshot books ~333,000 phantom tokens, which is larger than most context windows. `findCompactionCut` would fire on the first screenshot and evict real conversation history to make room for an image that actually costs ~4,800. This is not a rounding error; it is the meter reading the wrong quantity entirely. |
| | High-resolution | Claude 4.7 and later | 2576 px | 4784 | | ||
| | Standard | everything else | 1568 px | 1568 | | ||
|
|
||
| Images above either limit are **downscaled server-side, preserving aspect ratio**. I reimplemented the rule and checked it against every worked example in the documentation: the **token count matches on all twelve** (1092² → 1521, 1000² → 1296, 1920×1080 → 2691, 3840×2160 → 2576×1449 at 4784), and the sent dimensions match on eleven — one standard-tier row lands a single pixel off (1270 vs 1269 wide, same 1564 tokens), a rounding convention I could not derive from six data points. Cost is exact; geometry is exact to a pixel: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design only — no feature code. Seven slices, I1–I7.
The target is what Cursor and the Claude Code console already do: screenshot,
⌘Vinto the composer, ask why it looks wrong. That part is easy. This documents what breaks in our code when the first image goes through, and the cost model that decides how the bytes should be shaped before they leave the webview.Four things that break quietly
Verified against
develop, not recalled. Three of the four fail silently, which is the reason to write them down before writing feature code.extension.js:1752contentis a string —blocks.join('\n\n') + textproviders/translate.js:99agentMemory.js:40JSON.stringify(m).length / 4findCompactionCut, so one paste evicts real historyproviders/catalog.jsvision: trueis already there, nothing reads itsupportsToolsForModeland no vision equivalentThe third would have shipped as a mystery report: "long conversations forget things after I paste a screenshot."
The numbers changed a decision
I checked the vision API rather than trusting my prior, and the prior was wrong — cost is not
w × h / 750, it is⌈w/28⌉ × ⌈h/28⌉over 28px patches, capped per tier (2576px/4784 tokens on 4.7+, 1568/1568 below).That changed the design. Token cost is already capped server-side. Sending a 12 MB PNG does not buy more than 4784 tokens of fidelity — it buys latency. So client-side downscaling is not a defence against a token blowup, as I would otherwise have written it; it is a deliberate fidelity-for-cost trade (4784 → 1792 on a 4K grab) and a defence against the wire.
I reimplemented the resize rule and checked it against every worked example in the docs: token count matches on all twelve, dimensions on eleven — one standard-tier row lands a single pixel off, and the doc says so rather than claiming exactness.
Slices
I1is worth merging regardless of images — the silent block drop is a live bug today.supportsVisionForModelEvery exit criterion is written to be bypass-verifiable.
Deferred, with reasons
Files API upload (Anthropic-direct only; wins on repeat turns, not the first), image output (Claude does not generate images), PDF blocks, coordinates, client-side OCR.
Also
A readable version of this plan, with a live visual-token calculator built from the verified formula: https://claude.ai/code/artifact/a31690f7-79f1-4f6e-9902-3c32fd7f91de