Skip to content

Let a transformer block initialize its residual projections - #83

Merged
jessegrabowski merged 2 commits into
pymc-devs:mainfrom
jessegrabowski:transformer-initializers
Aug 11, 2026
Merged

Let a transformer block initialize its residual projections#83
jessegrabowski merged 2 commits into
pymc-devs:mainfrom
jessegrabowski:transformer-initializers

Conversation

@jessegrabowski

Copy link
Copy Markdown
Member

Nothing could reach the projections inside a transformer block, so GPT-style init — scale the two that write back into the residual stream, leave their siblings alone — was inexpressible. One keyword per level routes it now: out_proj_initializer on attention, residual_initializer on the block, which feeds both that and the feed-forward's second layer.

Biases stay unreachable on purpose: pinned to zero by declaration, which is what the models this serves want, and bias=False covers the ones that drop them. A one-off gets the per-parameter override coming next.

@codecov-commenter

codecov-commenter commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.47%. Comparing base (f0b7651) to head (0b231c4).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #83   +/-   ##
=======================================
  Coverage   95.46%   95.47%           
=======================================
  Files          45       45           
  Lines        1809     1811    +2     
=======================================
+ Hits         1727     1729    +2     
  Misses         82       82           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jessegrabowski
jessegrabowski merged commit 620fbd1 into pymc-devs:main Aug 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants