Skip to content

Add Echo-Memory optional loader and Wan 1.3B example - #1592

Open
WayneJin0918 wants to merge 1 commit into
modelscope:mainfrom
WayneJin0918:echo-memory-example
Open

Add Echo-Memory optional loader and Wan 1.3B example#1592
WayneJin0918 wants to merge 1 commit into
modelscope:mainfrom
WayneJin0918:echo-memory-example

Conversation

@WayneJin0918

@WayneJin0918 WayneJin0918 commented Aug 13, 2026

Copy link
Copy Markdown

Summary

  • Add an optional Echo-Memory loader that overlays the released context_k1 DiT fine-tune onto official Wan-AI/Wan2.1-T2V-1.3B.
  • Matching keys use original DiffSynth / Wan names, so no conversion is required. Extra action-MLP / SSM / spatial slots are skipped.
  • Example + low-VRAM example, plus a short EN/中文 note in the Wan docs.

This is a community fine-tune overlay, not a new backbone. The multi-chunk camera-action / revisit protocol stays in Echo-Memory (arXiv:2606.09803).

Test plan

  • python examples/wanvideo/model_inference/Echo-Memory.py with Wan 2.1 1.3B and Hugging Face access to Echo-Team/Echo-Memory
  • Confirm the loader prints ~825 overlaid DiT keys and skipped action/SSM extras

Made with Cursor

Overlay the released context_k1 DiT fine-tune onto official Wan 2.1 1.3B without changing the core pipeline. Extra action/SSM keys stay in the Echo-Memory research stack.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Artiprocher

Copy link
Copy Markdown
Collaborator

@WayneJin0918 Thanks for your contribution! We'll merge this PR after some code cleanup.

Also, for future development, feel free to check out our skills guide. It can help you write code that aligns with the framework's best practices more quickly: https://modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills

@WayneJin0918

Copy link
Copy Markdown
Author

Thanks @Artiprocher — appreciated. We'll leave the cleanup to you unless you want a specific change from our side.

If it helps, the loader is intentionally a thin overlay (pipe.dit.load_state_dict(..., strict=False)), not a new backbone. Extra action / SSM keys are skipped on purpose.

Skills guide noted for later rows.

@Artiprocher

Copy link
Copy Markdown
Collaborator

@WayneJin0918 As this open-source project is built upon the ModelScope platform, we would kindly ask you to upload your model to ModelScope first. Once that's done, we'll be more than happy to provide full support for it. Thank you for your understanding!

@Artiprocher

Copy link
Copy Markdown
Collaborator

@WayneJin0918 Additionally, we noticed that the model you provided doesn't include the complete set of weights. Would you mind packaging it into a full weight file? This would allow us to integrate the model in line with the framework's best practices and ensure proper support for training. We really appreciate your help!

@WayneJin0918

Copy link
Copy Markdown
Author

Thanks @Artiprocher — we'll do both.

  1. ModelScope: we'll upload a DiffSynth-facing context_k1 repo (same Apache-2.0 row as Echo-Team/Echo-Memory) and post the model_id here.

  2. Full weights: the current HF file is a research overlay (epoch-0.safetensors) — matching Wan DiT keys plus extra action/SSM slots, loaded with strict=False on official Wan 2.1 1.3B. We'll publish a complete DiT as diffusion_pytorch_model.safetensors (official Wan 1.3B DiT keys with the Echo-Memory overlay already merged; extras dropped) so it loads like a normal Wan transformer. T5 / VAE stay the official Wan-AI/Wan2.1-T2V-1.3B files.

If you also want the research extras (action_mlp / SSM) in that file for training, we can add a second full checkpoint.

Happy to point #1612 at the ModelScope id once it's up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants