Skip to content

fix(internvl): pass max patch count to vllm - #9850

Open
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/internvl-vllm-max-num
Open

fix(internvl): pass max patch count to vllm#9850
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/internvl-vllm-max-num

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fixes #7631.

Background

InternVL templates read the MAX_NUM environment variable into
template.max_num. The Transformers backend uses this value during
local image preprocessing, but the vLLM branch only emitted the image
placeholder and did not forward the patch limit.

As a result, vLLM used the model configuration's default
max_dynamic_patch, potentially producing prompts that exceeded the
model context even when users set MAX_NUM.

vLLM's InternVL processor accepts max_dynamic_patch through its
multimodal processor kwargs, and VllmEngine already forwards
inputs.mm_processor_kwargs with each request.

Changes

  • Forward template.max_num as max_dynamic_patch for InternVL
    templates in vLLM mode.
  • Use setdefault so an explicit per-request max_dynamic_patch
    remains authoritative.
  • Add CPU tests for default propagation, explicit override precedence,
    and unchanged Transformers behavior.

Verification

  • PYTHONPATH=. .venv/bin/python -m unittest tests.general.test_internvl_template
  • PYTHONPATH=. .venv/bin/python tests/run.py --pattern test_internvl_template.py
  • PYTHONPATH=. .venv/bin/python -m unittest tests.general.test_internvl_template tests.general.test_data_preprocess.TestProviderMessagesPreprocess tests.general.test_data_preprocess.TestRejectedMessagesPreprocess
  • .venv/bin/pre-commit run --all-files
  • git diff --check

Impact

The change only adds a vLLM multimodal processor kwarg for InternVL
templates. Transformers preprocessing is unchanged. Existing explicit
per-request values are preserved.

Experiment results

Before the fix, the vLLM regression test observed an empty
mm_processor_kwargs dictionary despite max_num=6. After the fix it
contains {"max_dynamic_patch": 6}; all three template tests and ten
related tests pass.

Forward the InternVL MAX_NUM setting as max_dynamic_patch through vLLM multimodal processor kwargs while preserving explicit per-request overrides.

Fixes modelscope#7631

Test: python -m unittest tests.general.test_internvl_template
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The max_num environment variable does not work when using vllm as the backend

1 participant