RAE support

[bytetriper]

Can we have support for Representation Autoencoders in diffusers? Since RAE already uses hf load for encoders, it will be much simpler if we can directly from_pretrained the whole autoencoder model.

Link: https://github.com/bytetriper/RAE

Paper: https://arxiv.org/abs/2510.11690

[sayakpaul]

There will definitely be a lot of interest in the RAE area. Since the pre-trained checkpoints are based on ImageNet, I think we can support RAEs.

To clarify, would it mean we support RAEs (pre-trained encoders, connector, the decoder) along with the generation pipeline?

Cc: @kashif if you are interested in this.

[bytetriper]

I think the autoencoder part would be more important! It doesn't necessarily need to include connector, but only the encoder-decoder arch itself. We have more large-scale trained RAE coming so it would super beneficial. The generation pipeline could be optional since arch and data is relatively confined to ImageNet, which is a bit small-scale nowadays.

[kashif]

happy to do this if no one else is assigned