Skip to content

feat(vlm): add vision feature caching for LFM2-VL - #1135

Open
MrFelt0385 wants to merge 1 commit into
lablup:mainfrom
MrFelt0385:feat/lfm2-vl-vision-cache
Open

feat(vlm): add vision feature caching for LFM2-VL#1135
MrFelt0385 wants to merge 1 commit into
lablup:mainfrom
MrFelt0385:feat/lfm2-vl-vision-cache

Conversation

@MrFelt0385

Copy link
Copy Markdown

feat(vlm): add vision feature caching for LFM2-VL

Summary

LFM2-VL currently skips the opportunistic vision feature cache (the code
says // LFM2-VL runs every image in one preparation pass; skip the opportunistic vision cache for this first integration.). This PR implements
it, giving a 2.12× speedup on repeated images — the main win for
multi-turn conversations and continuous screen analysis.

Changes

  • src/vision/lfm2_vl.rs — add get_input_embeddings_with_cache,
    mirroring the existing Qwen2.5-VL pattern:
    • Cache key: SHA-256 of the request image bytes + soft-token budget
      (image_hash_from_bytes_with_soft_tokens on the server side)
    • HIT: reuse the concatenated post-connector features, skip the vision
      tower + connector entirely
    • MISS: compute features, eval + deep-copy (decouple from the
      deferred graph), store snapshot in the cache
  • src/multimodal/vlm_runtime.rs — wire the Lfm2Vl runtime to the
    single-feature cache (ModelVisionCaches::single), deriving the key from
    the concatenated pixel tensor (same approach as the Qwen path)

Benchmarks

Measured on a Mac mini M1, LFM2.5-VL-3B-MLX-4bit, same image across
requests (zone_hud.png, 120×70):

Request Time Notes
1st (MISS) 1158 ms vision tower encodes the image
2nd (HIT, same image) 547 ms 2.12× faster
3rd (HIT, same image, different text) 550 ms 2.10× faster

Finesse preserved: HIT responses are byte-identical to MISS responses
(vision features are deterministic; only the tower output is cached).

Notes

  • The connector output shape depends on the image's native patch count,
    which is folded into the key by the soft-token budget — two resolutions of
    the same image cannot collide.
  • Cache lifetime is tied to the model registry entry (clear_all on unload),
    same as the existing Qwen/Gemma/Granite caches.

LFM2-VL currently skips the opportunistic vision feature cache
("skip the opportunistic vision cache for this first integration").
This change implements it:

- lfm2_vl.rs: add get_input_embeddings_with_cache, mirroring the
  Qwen2.5-VL pattern. The concatenated post-connector features are
  keyed by SHA-256 of the request image bytes + soft-token budget.
  On HIT the vision tower + connector are skipped entirely; on MISS
  the features are computed, evaluated, deep-copied and stored.
- vlm_runtime.rs: wire the Lfm2Vl runtime to the single-feature
  vision cache (ModelVisionCaches.single), deriving the cache key
  from the concatenated pixel tensor like the Qwen path.

Measured on a Mac mini M1 with LFM2.5-VL-3B-MLX-4bit:
- First request (MISS): 1158 ms
- Second request, same image (HIT): 547 ms (2.12x)
- Third request, same image, different text: 550 ms (2.10x)
Output is bit-identical on HIT (deterministic vision features).

This is the main win for multi-turn conversations and continuous
screen analysis, where the same images are revisited repeatedly.
@cla-assistant

cla-assistant Bot commented Aug 13, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant