issue/477 - feat: support partial cache hits for multimodal visual embeddings

[ma-hang]

Summary

• Support partial cache hits for multimodal visual embeddings

Motivation

• Enable partial cache hits for multimodal inputs such as image patches and video tokens by encoding multimodal feature identity into the block hash, so cache lookup can rely on hash equality without special-casing whether a block contains image or video content.

Type of Change

• feat — new feature / new model
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

MiniCPM-V-2_6 - image

Enxin_VideoNSA - image

Enxin_VideoNSA - video

Benchmark / Performance Impact

Notes for Reviewers

CI / ChatOps

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from main — the branch is rebased cleanly on top of the current main.
• No fixup! / squash! / wip commits remain.
• Existing PR/branch/commit that followed the legacy issue format.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• No raw new/delete; RAII / smart pointers / existing allocators are used.
• Changed files are formatted by scripts/format.py.
• No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
• Passed single request test (examples/test_infer.py), or specify the reason for skipping.
• Passed offline performance test (examples/bench.py), or specify the reason for skipping.
• Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
• Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for skipping.

Build, CI, and Tooling

• The project builds cleanly from a fresh directory on at least one affected platform.
• CI has been triggered manually (Actions → CI on this branch), or /retest was requested.

Documentation

• README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• Any user-visible breaking change is called out explicitly under "Motivation" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

[wooway777]

这不是推理流程内一下子多了一堆to cpu

[wooway777]

我测了一版 VideoNSA 五组性能，issue/477 分支相对 main 在无 partial cache hit 的普通多模态场景有明显回退，尤其 batch=16：

case	input/output	main InfiniLM ms	issue/477 ms	PR影响
4 / 2k-3k	2048/64	3566.55	3722.46	+4.4%
4 / 8k-10k	8192/64	4668.35	4744.19	+1.6%
16 / 200-400	256/64	4220.89	5118.33	+21.3%
16 / 2k-3k	2048/64	4937.21	7258.08	+47.0%
16 / 8k-10k	8192/64	6490.86	6828.52	+5.2%

主要看起来有两类开销：

1. BlockManager.compute_hash 从原来的二进制 hash 变成了每个 block 构造 dict + token ids 转 Python int + json.dumps。这个函数在 prefill block 计算、allocate slots、decode 追加 block hash 时都会走，即使没有 cache hit 也会触发，容易解释 batch=16 场景的回退。建议保留原来的 binary hash 方式，只把 mm extra keys 以稳定的二进制/tuple 增量方式 append 进 hash，不要走 JSON 序列化。

2. VideoNSA 的 no-hit 路径现在也会做 partial-cache 相关的数据重组：processor 侧每个请求/media part 都会重新 slice/cast/contiguous pixel tensor、构造 bound tensor、infinicore.from_torch；C++ forward 里还会对 grid/bounds/input_offsets 做多次 to(cpu)。这些在没有 partial hit 时都是额外开销。建议保留一个 fast path：当 compute_start == 0 或没有 partial visual hit 时，继续走原来的整图/整视频 batch 逻辑，只有真正 partial 命中时才启用新的 image_embed_bound/分段逻辑。

另外还有一个设计点需要确认：当前 partial visual embedding 只是切分 placeholder/KV 的替换范围，但 visual_->forward 仍然对传入的完整 pixel payload 跑视觉编码，然后再按 image_embed_bound slice。也就是说 partial cache hit 并没有节省视觉 encoder 的计算，只节省部分语言侧 prefill/KV。这个行为如果是预期的，建议在 PR 描述里说明；如果目标包含视觉 encoder 复用，还需要进一步设计。

结论：功能测试能跑通，但普通无 partial hit 场景有明显性能回退，建议先修 hash hot path 和 no-hit fast path 后再合。