Skip to content

[Bug Report] Fix dense component initialization for Pythia hook shapes #1326

@jlarson4

Description

@jlarson4

Summary

test_transformer_bridge_hook_shapes is parameterized over multiple models, but unconditionally skips Pythia at the top of the test body due to a known initialization issue with the dense component.

Affected test

  • tests/integration/test_hook_shape_compatibility.py:127 has inline pytest.skip("Pythia architecture needs dense component initialization fix")

What the test verifies

That all hooks fire with the expected tensor shapes (matching HookedTransformer's contract) on bridge models.

Acceptance criteria

  • The Pythia branch passes the same shape assertions as the other parameterized models.
  • The inline skip is removed.

Where to start

  • Pythia uses JointQKVPositionEmbeddingsAttentionBridge (rotary + joint QKV; same family as GPT-NeoX). Look at transformer_lens/model_bridge/supported_architectures/pythia.py and the dense o component construction in the adapter's _get_blocks().
  • Run the test locally with the skip removed to see the actual failure mode first.

Metadata

Metadata

Assignees

Labels

TransformerBridgeBug specific to the new TransformerBridge systembugSomething isn't workingcomplexity-moderateModerately complicated issues for people who have intermediate experience with the codegood first issueGood for newcomershelp wantedExtra attention is needed

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions