[BUG] Wan2.2 5B training demo failed in DiffSynth2.0

[shad0wta9]

I'm encountering a training regression after upgrading to DIffSynth2.0. When running the official training script example/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh, the resulting model generates severely distorted outputs, particularly in the first few frames of the generated video. See example output:

video_Wan2.2-TI2V-5B.mp4

However, when I downgrade back to the codebase to v1.19 (and use the corresponding training script from that release), training succeeds and produces expected results—no such artifacts appear. I have compared the corresponding codes but I have no idea about what makes the difference. I think it should a bug in 2.0. Can anyone help?
testing env: torch==2.5.1+cu12.4 torchvision==0.20.1

[Sun-Happy-YKX]

Hi @shad0wta9, we’re seeing the same regression after upgrading to DiffSynth 2.0 when running example/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh: the trained model produces heavily distorted results, especially in the first few frames. With v1.19 + the matching script everything looks normal.

Have you already found the root cause or a workaround/fix (e.g., specific commit, config change, training args, scheduler/vae setting, precision setting, etc.)? If yes, could you share what you changed?

[shad0wta9]

Hi, @Sun-Happy-YKX.
No. The only thing I can confirm is that this bug occurs during model training. I've compared most of the training code and found nothing unusual. I believe the bug was introduced by some low-level optimizations in this major update.

[Sun-Happy-YKX]

Thanks, @shad0wta9

[kellyiss]

I am confused about the logic here:
In the WanVideoUnit_ImageEmbedderFused pipeline, the first frame latent is set

DiffSynth-Studio/diffsynth/pipelines/wan_video.py

Line 496 in de0aa94

latents[:, :, 0: 1] = z

But in the loss function, the latents are reset to noise:

DiffSynth-Studio/diffsynth/diffusion/loss.py

Line 13 in de0aa94

inputs["latents"] = pipe.scheduler.add_noise(inputs["input_latents"], noise, timestep)

In the model_fn_wan_video function, I cannot locate where the original first frame latent (set in the pipeline) is restored or reintegrated into the computation

Am I missing something? Or is it a bug?

[shad0wta9]

@kellyiss Yes, you are right. The only difference between 1.19 and 2.0 is the noise used in training.
In 1.19, they used inputs["noise"] instead of a newly created noise variable.

DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py

Line 119 in afd101f

inputs["latents"] = self.scheduler.add_noise(inputs["input_latents"], inputs["noise"], timestep)

inputs["noise"] was synchronized with inputs["latents"] in model training:

DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py

Lines 629 to 630 in afd101f

	if pipe.scheduler.training:
	return {"latents": noise, "input_latents": input_latents}

When inserting the first frame, the first-frame noise is replaced by the gt latent, therefore the first frame of inputs["input_latents"] and inputs["noise"] will be the same, and add_noise function won't affect the first-frame latent.

DiffSynth-Studio/diffsynth/pipelines/wan_video_new.py

Lines 766 to 767 in afd101f

	latents[:, :, 0: 1] = z
	return {"latents": latents, "fuse_vae_embedding_in_latents": True, "first_frame_latents": z}

I'm not sure whether this bug could be resolved by simply replacing noise back to inputs["noise"]. @Artiprocher please check it.