feat(vlm): add vision feature caching for LFM2-VL - #1135
Open
MrFelt0385 wants to merge 1 commit into
Open
Conversation
LFM2-VL currently skips the opportunistic vision feature cache
("skip the opportunistic vision cache for this first integration").
This change implements it:
- lfm2_vl.rs: add get_input_embeddings_with_cache, mirroring the
Qwen2.5-VL pattern. The concatenated post-connector features are
keyed by SHA-256 of the request image bytes + soft-token budget.
On HIT the vision tower + connector are skipped entirely; on MISS
the features are computed, evaluated, deep-copied and stored.
- vlm_runtime.rs: wire the Lfm2Vl runtime to the single-feature
vision cache (ModelVisionCaches.single), deriving the cache key
from the concatenated pixel tensor like the Qwen path.
Measured on a Mac mini M1 with LFM2.5-VL-3B-MLX-4bit:
- First request (MISS): 1158 ms
- Second request, same image (HIT): 547 ms (2.12x)
- Third request, same image, different text: 550 ms (2.10x)
Output is bit-identical on HIT (deterministic vision features).
This is the main win for multi-turn conversations and continuous
screen analysis, where the same images are revisited repeatedly.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(vlm): add vision feature caching for LFM2-VL
Summary
LFM2-VL currently skips the opportunistic vision feature cache (the code
says
// LFM2-VL runs every image in one preparation pass; skip the opportunistic vision cache for this first integration.). This PR implementsit, giving a 2.12× speedup on repeated images — the main win for
multi-turn conversations and continuous screen analysis.
Changes
src/vision/lfm2_vl.rs— addget_input_embeddings_with_cache,mirroring the existing Qwen2.5-VL pattern:
(
image_hash_from_bytes_with_soft_tokenson the server side)tower + connector entirely
eval+ deep-copy(decouple from thedeferred graph), store snapshot in the cache
src/multimodal/vlm_runtime.rs— wire theLfm2Vlruntime to thesingle-feature cache (
ModelVisionCaches::single), deriving the key fromthe concatenated pixel tensor (same approach as the Qwen path)
Benchmarks
Measured on a Mac mini M1,
LFM2.5-VL-3B-MLX-4bit, same image acrossrequests (
zone_hud.png, 120×70):Finesse preserved: HIT responses are byte-identical to MISS responses
(vision features are deterministic; only the tower output is cached).
Notes
which is folded into the key by the soft-token budget — two resolutions of
the same image cannot collide.
clear_allon unload),same as the existing Qwen/Gemma/Granite caches.