index.json

[{"summary":"\u003cp\u003e\u003ca href=\"https://arxiv.org/abs/2501.05067\" class=\"btn external\" target=\"_blank\"\u003ePAPER\u003c/a\u003e\n\u003ca href=\"https://github.com/Jiaxing-star/LLaVA-Octopus\" class=\"btn external\" target=\"_blank\"\u003eCODE\u003c/a\u003e\n\u003ca href=\"https://github.com/Jiaxing-star/LLaVA-Octopus\" class=\"btn external\" target=\"_blank\"\u003eCheckpoints\u003c/a\u003e\n\u003ca href=\"https://github.com/Jiaxing-star/LLaVA-Octopus\" class=\"btn external\" target=\"_blank\"\u003eDemo\u003c/a\u003e\u003c/p\u003e\n\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eWe present \u003cstrong\u003eLLaVA-Octopus\u003c/strong\u003e, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the \u003cstrong\u003ecomplementary strengths of each projector\u003c/strong\u003e. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model\u0026rsquo;s performance in multimodal tasks. LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as \u003cstrong\u003emultimodal understanding, visual question answering, and video understanding\u003c/strong\u003e, highlighting its broad application potential.\u003c/p\u003e","title":"LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding"}]