[CVPR 2026 Highlight] MatLat: Material Latent Space for PBR Texture Generation

Kyeongmin Yeo, Yunhong Min, Jaihoon Kim, Minhyuk Sung

KAIST

Paper | Project Page

MatLat learns a material latent space for physically based rendering (PBR) texture generation. The release contains the two main training and inference components:

• MatVAE: encodes and decodes PBR texture maps in the material latent space.
• MatDiff: generates MatLat latents from text and rendered geometry conditioning.

News

• [2026.06] Code released.
• [2026.04] MatLat has been selected as a Highlight at CVPR 2026.
• [2024.12] Preprint released on arXiv.

Qualitative Results

For more videos and comparisons, visit the project page.

Installation

conda create -n matlat python=3.10
conda activate matlat

# Choose the PyTorch command that matches your CUDA/runtime.
# Example for CUDA 12.8:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128

pip install -r requirements.txt
pip install git+https://github.com/NVlabs/nvdiffrast.git

Blender is required only when rendering datasets with the Blender backend. The code downloads SD3.5 and LPIPS components when they are not already cached locally.

Checkpoints

Place model files under the following release paths:

checkpoints/
  matvae_control_ema.pt
  matvae_v.ours.pt          # optional: no-control MatVAE variants
  RealESRGAN_x2plus.pth
  big-lama.pt
vae_sd35/
precomputed_tensors/
  null_prompt_embeds.pt
  null_pooled_prompt_embeds.pt

The RealESRGAN and LaMa checkpoints are used by inference post-processing:

mkdir -p checkpoints
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth \
  -O checkpoints/RealESRGAN_x2plus.pth
wget https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt \
  -O checkpoints/big-lama.pt

Create the null prompt tensors with:

python tools/calculate_prompt_embeds.py \
  --tag null \
  --prompts "" \
  --out-dir precomputed_tensors

Dataset And Caches

The training code expects RGBMR WebDataset shards and MatDiff cache subdatasets under:

${DATA_ROOT}/
  train/main/
  train/prompt_embeds/
  train/latents/
  val_matdiff/main/
  val_matdiff/prompt_embeds/
  val_matdiff/latents/
  val_matvae/main/

Create the main shards and caches as follows. The renderer takes a mesh list and a prompt CSV:

glbs.txt
path/to/object_000.glb
path/to/object_001.glb

prompts.csv
object_000,a red painted metal with scratches
object_001,"a rough ceramic material, blue and white"

Render the training and validation splits:

export DATA_ROOT=/path/to/rgbmr_mv

python data/generate_dataset.py \
  --backend blender \
  --blender-renderer eevee \
  --out-dir "$DATA_ROOT" \
  --part-dir "$DATA_ROOT/train/main" \
  --glb-list path/to/glbs.txt \
  --prompt-path path/to/prompts.csv

python data/generate_dataset.py \
  --backend blender \
  --blender-renderer eevee \
  --out-dir "$DATA_ROOT" \
  --part-dir "$DATA_ROOT/val_matvae/main" \
  --glb-list path/to/val_glbs.txt \
  --prompt-path path/to/val_prompts.csv

python data/generate_dataset.py \
  --backend blender \
  --blender-renderer eevee \
  --out-dir "$DATA_ROOT" \
  --part-dir "$DATA_ROOT/val_matdiff/main" \
  --glb-list path/to/val_glbs.txt \
  --prompt-path path/to/val_prompts.csv

The default training configs expect validation shards to exist. MatVAE uses val_matvae/main; MatDiff uses val_matdiff/main plus the cache subdatasets below. MatDiff validation can be disabled with train.validation_steps=0.

Build MatDiff cache subdatasets:

python data/precompute_text_embeddings.py \
  --dataset-root "$DATA_ROOT/train/main" \
  --out-dir "$DATA_ROOT/train/prompt_embeds"

python data/precompute_latents.py \
  --dataset-root "$DATA_ROOT/train/main" \
  --out-dir "$DATA_ROOT/train/latents" \
  --matvae-config configs/matvae/config_with_control.json \
  --matvae-ckpt checkpoints/matvae_control_ema.pt

Repeat the same cache commands for val_matdiff/main using val_matdiff/prompt_embeds and val_matdiff/latents as outputs.

Training

MatVAE:

export DATA_ROOT=/path/to/rgbmr_mv
OUTPUT_DIR=outputs/matvae_release \
scripts/train_matvae_8gpu.sh trainer.max_steps=200001

MatDiff:

export DATA_ROOT=/path/to/rgbmr_mv
OUTPUT_DIR=outputs/matdiff_release \
scripts/train_matdiff_8gpu.sh train.max_train_steps=40001

Inference

Run mesh-to-texture generation with a trained MatDiff checkpoint:

python inference.py \
  --mesh assets/motorcycle_cafe_racer/motorcycle_cafe_racer.obj \
  --prompt "a matte olive green cafe racer motorcycle with brushed steel parts" \
  --weights path/to/matdiff_weights \
  --out outputs/demo

Acknowledgements

The inference texture post-processing path builds utilities from MV-Adapter.

Citation

@inproceedings{yeo2026matlat,
  title     = {MatLat: Material Latent Space for PBR Texture Generation},
  author    = {Yeo, Kyeongmin and Min, Yunhong and Kim, Jaihoon and Sung, Minhyuk},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
}