Skip to content

Visual Encoders (ViT, DeiT) Support Rollout#1540

Closed
david-wei-01001 wants to merge 101 commits into
TransformerLensOrg:mainfrom
david-wei-01001:visual_encoder
Closed

Visual Encoders (ViT, DeiT) Support Rollout#1540
david-wei-01001 wants to merge 101 commits into
TransformerLensOrg:mainfrom
david-wei-01001:visual_encoder

Conversation

@david-wei-01001

@david-wei-01001 david-wei-01001 commented Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@david-wei-01001
Create HookedVisualEncoder.py
3d1ce8e
@david-wei-01001
Update HookedVisualEncoder.py
0da6a7c
@david-wei-01001
Update __init__.py
7fbfd93
@david-wei-01001
Update HookedAudioEncoder.py
26e2250
@david-wei-01001
Update HookedVisualEncoder.py
99adc0c
@david-wei-01001
Update HookedVisualEncoder.py
e4510c3
@david-wei-01001
Update HookedVisualEncoder.py
7a9b4ed
@david-wei-01001
Update HookedVisualEncoder.py
c03042c
@david-wei-01001
Create ClassifierHead.py
69d4b30
@david-wei-01001
Update __init__.py
9379d0e
@david-wei-01001
Update ClassifierHead.py
b59bf7a
@david-wei-01001
Rename ClassifierHead.py to classifier_head.py
31d3871
@david-wei-01001
Update HookedVisualEncoder.py
b1f5f14
@david-wei-01001
Update HookedVisualEncoder.py
321c22a
@david-wei-01001
Update HookedVisualEncoder.py
ec73db7
@david-wei-01001
Create prenorm_block.py
aeb1b4c
@david-wei-01001
Update __init__.py
c61656d
@david-wei-01001
Update __init__.py
985cb0b
@david-wei-01001
Update HookedVisualEncoder.py
c82dfcb
@david-wei-01001
Update HookedVisualEncoder.py
d0075d3
@david-wei-01001
Delete transformer_lens/components/prenorm_block.py
a41c472
@david-wei-01001
Update __init__.py
ef2a752
@david-wei-01001
Update HookedVisualEncoder.py
900b96c
@david-wei-01001
Update HookedAudioEncoder.py
f3ff4ff
@david-wei-01001
Update HookedVisualEncoder.py
1dac0be
@david-wei-01001
Update HookedVisualEncoder.py
cd71f0b
@david-wei-01001
Update supported_models.py
61d7f79
@david-wei-01001
Update HookedVisualEncoder.py
2c80f0f
@david-wei-01001
Update loading_from_pretrained.py
dd1e9cd
@david-wei-01001
Update HookedVisualEncoder.py
96e5f99
david-wei-01001 and others added 24 commits May 10, 2026 14:13
@david-wei-01001
Update supported_models.py
ee8d7db
@david-wei-01001
Update supported_models.py
7e0e469
@david-wei-01001
Update loading_from_pretrained.py
baaf558
@david-wei-01001
Update HookedVisualEncoder.py
5e62c7f
@david-wei-01001
Update HookedVisualEncoder.py
42e8f41
@david-wei-01001
Update supported_models.py
98d572f
@david-wei-01001
Update HookedVisualEncoder.py
684927a
Jiankun Wei
done
df810d3
@david-wei-01001
Create test_deit_hooks.py
9cfac51
@david-wei-01001
Update HookedVisualEncoder.py
b469099
@david-wei-01001
Create test_vit_model_compare.py
ad115fc
@david-wei-01001
Update HookedVisualEncoder.py
d699108
@david-wei-01001
Update HookedVisualEncoder.py
77b35c2
@david-wei-01001
Create test_vit_model_hooks.py
8603eee
@david-wei-01001
Update HookedVisualEncoder.py
09957bc
@david-wei-01001
Update supported_models.py
ad48409
@david-wei-01001
Update supported_models.py
c6814e3
@david-wei-01001
Update loading_from_pretrained.py
143d84a
@david-wei-01001
Update vit.py
c1c9318
@david-wei-01001
Update vit.py
3c0b511
@david-wei-01001
Update vit.py
b9de77b
@david-wei-01001
Update vit.py
d26f5ca
@david-wei-01001
Update loading_from_pretrained.py
de58cc1
@david-wei-01001
Merge branch 'main' into visual_encoder
237fb22
@david-wei-01001 david-wei-01001 changed the title TransformerLens Support Visual Encoders Visual Encoders (ViT, DeiT) Support Rollout Jul 24, 2026
@jlarson4

jlarson4 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Hi @david-wei-01001! We are planning to deprecate the Hooked systems in the near future, would you be willing to migrate this system to TransformerBridge?

@david-wei-01001

david-wei-01001 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Hi @david-wei-01001! We are planning to deprecate the Hooked systems in the near future, would you be willing to migrate this system to TransformerBridge?

Of course! But I am not too familiar to TransformerBridge, could you point me to where it is?

@david-wei-01001

david-wei-01001 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Ah I see, yes of course I can!

@jlarson4

jlarson4 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Thank you! Looking forward to this, I was reading it over, looks very interesting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@david-wei-01001 @jlarson4