About the xy\yx lora.

[Teiboku]

Hello, thank you for your exciting work.

I noticed that both xy_lora and yx_lora are LoRA modules applied to the same target modules. However, in your paper, you mentioned that they can control the information flow from x to y or y to x. Could you please clarify how this differs from simply training a single LoRA on the same module?

[lixirui142]

Hi, thanks for your question.

We have x and y attend to each other in our joint cross-attention module. x and y each need a set of attention weights. So we apply two sets of LoRA to the same joint cross-attention module. To distinguish them, we use a LoRA mask to make sure each LoRA adapter only works for part of the inputs, e.g. xy_lora for x and yx_lora for y. That's why it differs from training a single LoRA.