Skip to content

Consistency checks for weight initialization #438

@le1nux

Description

@le1nux

The model weights are currently initialised by matching regexes against the FQNs.
For each regex, we define a routine specifying how the weights are supposed to be initialised.

This can be prone to errors if FQNs change, as is the case for torch.compile wich introduces a _orig_mod. to FQNs.

Since side-effects like these can easily go unnoticed we should introduce some consistency checks:

  • Each FQN must match with exactly one regex.
  • If a FQN is not supposed to be initialized via in a initialisation approach (e.g., some norms), the regex should call a "no-op" instead of an init routine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions