Multi Lora Benchmarking Support

[rlakhtakia]

Traffic splitting across multiple adapters will be useful for benchmarking a multi lora scenario. This is an important benchmarking area to cover.

[changminbark]

Can I get some pointers to where I can learn about Multi LoRA serving and how benchmarking this would look like? I would like to tackle this issue. Thanks!

[changminbark]

@rlakhtakia After reading about Multi LoRA, I see that it is a feature that is supported in llm serving runtimes like vLLM. How would the traffic be split across multiple adapters? Isn't that something you set in runtimes like vLLM rather than in inference-perf? Is there any way to specify which LoRA adapter to use when sending requests to an inference endpoint?

@Bslabe123 @jjk-g I will continue to investigate this, but it would be great if I could get a few pointers. I am still interested in solving this issue. Thank you!

[changminbark]

After doing a bit more research on vLLM's LoRA inference serving. I see that we can serve LoRA adapters on startup. I am thinking of creating more fields in the config.yaml file where users can specify the names of these LoRA adapters, which must be served on vLLM serving startup. We can also verify this using the curl localhost:8000/v1/models | jq . command.

As for the traffic splitting, we would need to modify loadgen to produce load intended for specific LoRA adapters. This will then be processed by the client and forwarded to the proper model (LoRA adapter) endpoints.

Note: I am not sure if this is also the case for sglang and tgi.

Source: https://docs.vllm.ai/en/stable/features/lora/#serving-lora-adapters

[jjk-g]

@changminbark usually the lora is specified via the model field in the request.

I believe it would be sufficient to add the ability to configure inference-perf to assign a percent of traffic per lora

something like

loadgen:
   traffic_split:
      # List of loras and % split, must add to 100%
      - name: lora-name-1
         split: 0.50
      - name: lora-name-2
         split: 0.50

It is important that reported results are available per traffic split.

There was a previous tool that supported multi-lora GoogleCloudPlatform/ai-on-gke@a3401f2