InternVideo-NeXt clip_projector training procedure: eval only? from scratch?

[MichaelRamamonjisoa]

Dear authors,

Thanks for your interesting paper!
I see you mention in your paper in section B.2 of the appendix that you train an attention pooling head for evaluation.

At the same time, I see that the model you released on HuggingFace has a clip_projector module (AttentionPoolingBlock) attached to it, and in the forward of the model you set projected=True (HF code here)

What I would like to know is:

1. on what data was this release clip_projector trained?
2. if I want to evaluate your model on action recognition, should I:
   a. train the attentive probe from scratch on the target dataset
   b. finetune the attentive probe on the target dataset (if so, from which weights?)
3. which setup did you use in your paper, 2.a. or 2.b., or none of them?

You released some action recognition code for InternVideo2 and for linear probing / attention probing there is this --open_clip_projector parameter which controls whether you're finetuning the head, or not. But this doesn't say on which data the head was trained before the finetuning.

You mention in the paper in section 4.2 (Video Classification)

 We test the model in an ‘Attentive Probing’ setting
where the encoders are frozen and a single-layer attention
pooling head is trained. Such Frozen Encoder settings can
test representation’s quality in an unbiased way. Our methods achieve the best results with only public data and less
computation cost on these foundation tasks.

What would make sense to me is that you're not using the Internvideo2 evaluation script anymore and you're training the attentive probe from scratch (2.a. mentioned above). But if so, I'd like to know on which evaluation experiment the weights of the release clip_projector were obtained.

Thanks in advance for your clarifications!

[Revliter]

Thanks again for your thoughtful questions and for digging into the details so carefully.

1. About the released clip_projector
The clip_projector included in the released checkpoint was trained during stage 1 pretraining, where it was optimized to align with the attentive pooled tokens produced by SigLIP2. This module was not used during stage 2, so using it directly for inference may introduce a small gap. However, in downstream action recognition tasks, we found that keeping this pretrained projector and finetuning it on the target dataset works very well, and the results reported in our paper are based on this setup.

2. About evaluation setup (2.a vs 2.b)
We do not train the attentive probe from scratch for evaluation (i.e., not 2.a). Instead, we use setup 2.b: we start from the released pretrained clip_projector weights and finetune them on the downstream dataset.

3. About --open_clip_projector
Most of the current codebase is kept compatible with InternVideo2. The --open_clip_projector flag simply controls whether the released clip_projector is finetuned. Importantly, this projector has not been finetuned on any downstream dataset before release; it is exactly the one obtained during pretraining.

We will also release the full training code upon the paper’s official publication, which should make the training and evaluation pipeline even clearer.

We hope this clarifies things, and we appreciate your careful reading and questions!