Skip to content

[tx] Implement context parallelism #1056

@pcmoritz

Description

@pcmoritz

There was recently an interesting blog post by nvidia about this: https://developer.nvidia.com/blog/accelerating-long-context-model-training-in-jax-and-xla/

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions