Skip to content

Add Starcoder2 architecture adapter#1533

Open
SanjidMzi wants to merge 1 commit into
TransformerLensOrg:devfrom
SanjidMzi:feat/starcoder2-adapter
Open

Add Starcoder2 architecture adapter#1533
SanjidMzi wants to merge 1 commit into
TransformerLensOrg:devfrom
SanjidMzi:feat/starcoder2-adapter

Conversation

@SanjidMzi

Copy link
Copy Markdown

Adds a TransformerBridge adapter for Starcoder2 (Starcoder2ForCausalLM), which had no support (StarCoder1 / gpt_bigcode is already supported).

Starcoder2 is a Llama-shaped decoder (sequential pre-norm, GQA, rotary) that differs from Llama in three ways it shares with GPTBigCode: LayerNorm+bias instead of RMSNorm, a non-gated GELU MLP (c_fc/c_proj), and biases on all projections.

  • Adapter + four-place registration (factory, __init__, model registry, report).
  • Verified numerically: bridge logits match HuggingFace on tiny-random-Starcoder2ForCausalLM (max abs diff ~1e-7, 100% argmax agreement).
  • Unit tests (config, component mapping, GQA weight/bias conversions) + an integration test asserting the forward pass matches HuggingFace.

make format, uv run mypy ., and unit + integration + registry tests pass locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant