Maximum token limit per request causes failure with large `system_prompt_len`

[kfirtoledo]

What happened:
Running inference-perf with a large system_prompt_len (25000 tokens) causes the tool to fail with the following error:

File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 388, in readuntil
    raise ValueError("Chunk too big")

How to reproduce it (as minimally and precisely as possible):
Use a shared_prefix workload with a large system prompt:

type: shared_prefix
  shared_prefix:
    num_groups: 20              
    num_prompts_per_group: 2      
    system_prompt_len: 25000   

[effi-ofer]

Hey @kfirtoledo I just ran inference-perf/shared_prefix_syntehetic with system prompt len set to 25000 and it finished successfully.
Here are my settings:

load:
  type: constant
  stages:
  - rate: 1
    duration: 20
  # - rate: 3
  #   duration: 50
  # - rate: 5
  #   duration: 50
  # - rate: 7
  #   duration: 50
api:
  type: completion
  streaming: true
server:
  type: vllm
  model_name: REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_MODEL
  base_url: REPLACE_ENV_LLMDBENCH_HARNESS_STACK_ENDPOINT_URL
  ignore_eos: true
tokenizer:
  pretrained_model_name_or_path: REPLACE_ENV_LLMDBENCH_DEPLOY_CURRENT_TOKENIZER
data:
  type: shared_prefix
  shared_prefix:
    num_groups: 32                # Number of distinct shared prefixes
    num_prompts_per_group: 32     # Number of unique questions per shared prefix
    system_prompt_len: 25000 # 2048        Length of the shared prefix (in tokens)
    question_len: 256             # Length of the unique question part (in tokens)
    output_len: 256               # Target length for the model's generated output (in tokens)
report:
  request_lifecycle:
    summary: true
    per_stage: true
    per_request: true
storage:
  local_storage:
    path: /workspace

Which workload are you running? What is your profile config file?

[effi-ofer]

Increasing the system_prompt_len to 40,000 results the llmdbench-inference-perf-launcher pod failing here:

Stage 0 progress:  35%|███▌      | 0.35/1.0 [01:19<02:05, 192.93s/it]2025-11-20 10:39:18,459 - inference_perf.client.modelserver.openai_client - ERROR - error occured during request processing:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 355, in _wait
    await waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/inference_perf/client/modelserver/openai_client.py", line 92, in process_request
    response_info = await data.process_response(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/inference_perf/apis/completion.py", line 51, in process_response
    async for chunk_bytes in response.content:
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 52, in __anext__
    rv = await self.read_func()
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 360, in readline
    return await self.readuntil()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 394, in readuntil
    await self._wait("readuntil")
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 354, in _wait
    with self._timer:
         ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/helpers.py", line 685, in __exit__
    raise asyncio.TimeoutError from exc_val
TimeoutError
Stage 0 progress:  40%|████      | 0.4/1.0 [05:10<16:02, 1604.07s/it]2025-11-20 10:39:20,457 - inference_perf.client.modelserver.openai_client - ERROR - error occured during request processing:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/streams.py", line 355, in _wait
    await waiter
asyncio.exceptions.CancelledError

[SachinVarghese]

/assign @jjk-g