Skip to content

README.md

Latest commit

 

History

History
176 lines (145 loc) · 6.17 KB

README.md

File metadata and controls

176 lines (145 loc) · 6.17 KB

CrossScore: Towards Multi-View Image Evaluation and Scoring

Project Page | arXiv | Data Processing Script

Zirui Wang, Wenjing Bian, Victor Adrian Prisacariu.

Active Vision Lab (AVL), University of Oxford.

Table of Content

Installation (pip)

Step 1: Install PyTorch with your preferred CUDA version (see pytorch.org):

# Example: PyTorch with CUDA 12.1
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

# Example: PyTorch with CUDA 11.8
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Example: CPU only
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

Step 2: Install CrossScore:

pip install crossscore

Or install from source:

git clone https://github.com/ActiveVisionLab/CrossScore.git
cd CrossScore
pip install -e .

Why install PyTorch separately? PyTorch distributions are coupled with specific CUDA versions. By letting you install PyTorch first, we avoid version conflicts with your system's CUDA setup. CrossScore works with PyTorch 2.0+ and any CUDA version it supports.

Quick Start

Python API

import crossscore

# Score query images against reference images
# Model checkpoint is auto-downloaded on first use (~129MB)
results = crossscore.score(
    query_dir="path/to/query/images",
    reference_dir="path/to/reference/images",
)

# Per-image mean scores
print(results["scores"])  # [0.82, 0.91, 0.76, ...]

# Score map tensors (pixel-level quality maps)
for score_map in results["score_maps"]:
    print(score_map.shape)  # (batch_size, H, W)

# Colorized score map PNGs are written to results["out_dir"]

Command Line

crossscore --query-dir path/to/queries --reference-dir path/to/references

# With options
crossscore --query-dir renders/ --reference-dir gt/ --metric-type mae --batch-size 4

# Force CPU mode
crossscore --query-dir renders/ --reference-dir gt/ --cpu

Environment Variables

  • CROSSSCORE_CKPT_PATH: Use a specific local checkpoint instead of auto-downloading
  • CROSSSCORE_CACHE_DIR: Custom cache directory (default: ~/.cache/crossscore)

Environment (conda, legacy)

We also provide a environment.yaml file to set up a conda environment:

git clone https://github.com/ActiveVisionLab/CrossScore.git
cd CrossScore
conda env create -f environment.yaml
conda activate CrossScore

Data

TLDR: download this file (~3GB), put it in datadir:

mkdir datadir
cd datadir
wget https://www.robots.ox.ac.uk/~ryan/CrossScore/MFR_subset_demo.tar.gz
tar -xzvf MFR_subset_demo.tar.gz
rm MFR_subset_demo.tar.gz
cd ..

To demonstrate a minimum working example for training and inferencing steps shown below, we provide a small pre-processed subset. The is a subset of Map-Free Relocalisation (MFR) and is pre-processed using 3D Gaussian Splatting (3DGS). This small demo dataset is available at this link (~3GB). This is the file in TLDR. We only use this demo subset to present the expected dataloading structure.

In our actual training, our model is trained using MFR that pre-processed by three NVS methods: 3DGS, TensoRF, and NeRFacto. Due to the preprocessed file size (~2TB), it is challenging to directly share this pre-processed data. One work around is to release a data pre-processing script for MFR. We aim to release the pre-processing script in Dec 2024. We have released the data pre-processing script at this repository.

Training

We train our model with two NVIDIA A5000 (24GB) GPUs for about two days. However, the model should perform reasonably well after 12 hours of training. It is also possible to train with a single GPU.

python task/train.py trainer.devices='[0,1]'  # 2 GPUs
# python task/train.py trainer.devices='[0]'  # 1 GPU

Inferencing

We provide an example command to predict CrossScore for NVS rendered images by referencing real captured images.

git lfs install && git lfs pull  # get our ckpt using git LFS
bash predict.sh

After running the script, our CrossScore score maps should be written to predict dir. The output should be similar to our demo video on our project page.

Todo

  • Create a HuggingFace demo page.
  • Release ECCV quantitative results related scripts.
  • Release data processing scripts
  • Release PyPI package.

Acknowledgement

This research is supported by an ARIA research gift grant from Meta Reality Lab. We gratefully thank Shangzhe Wu, Tengda Han, Zihang Lai for insightful discussions, and Michael Hobley for proofreading.

Citation

@inproceedings{wang2024crossscore,
  title={CrossScore: Towards Multi-View Image Evaluation and Scoring},
  author={Zirui Wang and Wenjing Bian and Victor Adrian Prisacariu},
  booktitle={ECCV},
  year={2024}
}