Skip to content

fix(infer): align adapter names with batch size - #9847

Open
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/batch-adapter-names
Open

fix(infer): align adapter names with batch size#9847
RerankerGuo wants to merge 1 commit into
modelscope:mainfrom
RerankerGuo:fix/batch-adapter-names

Conversation

@RerankerGuo

Copy link
Copy Markdown
Contributor

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fixes #5285.

Background

TransformersEngine accepted one AdapterRequest for a batch, but
_get_adapter_names() always returned a one-element list. PEFT requires
one adapter name per input, so batches larger than one failed with:

ValueError: Length of `adapter_names` should be the same as the number
of inputs, but got 1 and 10 respectively.

The same mismatch affected base-model inference after an adapter had
already been loaded because the __base__ marker was also returned only
once.

Changes

  • Pass the encoded batch size from streaming, forward, and generation
    inference paths to _get_adapter_names().
  • Repeat the selected adapter name for every sample in the batch.
  • Repeat the __base__ marker when running the base model after loading
    an adapter.
  • Add CPU regression tests for both adapter and base-model batches.

Verification

  • PYTHONPATH=. .venv/bin/python -m unittest tests.infer.test_transformers_engine_adapters
  • PYTHONPATH=. .venv/bin/python tests/run.py --pattern test_transformers_engine_adapters.py
  • .venv/bin/pre-commit run --all-files
  • git diff --check

Impact

The change is limited to per-batch adapter-name construction in
TransformersEngine. Single-sample inference is unchanged, and the
existing API still applies one AdapterRequest to the whole batch.

Experiment results

Before the fix, both regression cases failed because
_get_adapter_names() could not produce batch-aligned names. After the
fix, the selected adapter and __base__ paths return lists whose lengths
match the batch size; both tests pass.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Replicate the selected LoRA adapter name, or the base adapter marker, for every sample in a TransformersEngine batch.

Fixes modelscope#5285

Test: python -m unittest tests.infer.test_transformers_engine_adapters
@RerankerGuo
RerankerGuo force-pushed the fix/batch-adapter-names branch from 9b3a925 to 77fc950 Compare August 4, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LoRA batch推理,报错:Length of adapter_names should be the same as the number of inputs, but got 1 and 10 respectively

1 participant