🤗 Layout Evaluation Metrics by Huggingface Evaluate

A collection of metrics to evaluate layout generation that can be easily used in 🤗 huggingface evaluate.

Generative Model Evaluation Metrics

📊 Metric	🤗 Space	📝 Paper
	creative-graphic-design/layout-generative-model-scores	Heusel+ NeurIPS'171, Naeem+ ICML'202

Basic Layout Quality Metrics

📊 Metric	🤗 Space	📝 Paper
	creative-graphic-design/layout-maximum-iou	Kikuchi+ ACMMM'213
	creative-graphic-design/layout-average-iou	Arroyo+ CVPR'214, Kong+ ECCV'225
	creative-graphic-design/layout-alignment	Lee+ ECCV'206, Li+ TVCG'217, Kikuchi+ ACMMM'213
	creative-graphic-design/layout-overlap	Li+ ICLR'198, Li+ TVCG'217, Kikuchi+ ACMMM'213

Content-Aware Layout Metrics (PosterLayout)

📊 Metric	🤗 Space	📝 Paper
	creative-graphic-design/layout-validity	Hsu+ CVPR'239
	creative-graphic-design/layout-occlusion	Hsu+ CVPR'239
	creative-graphic-design/layout-overlay	Hsu+ CVPR'239
	creative-graphic-design/layout-underlay-effectiveness	Hsu+ CVPR'239
	creative-graphic-design/layout-unreadability	Hsu+ CVPR'239
	creative-graphic-design/layout-utility	Hsu+ CVPR'239
	creative-graphic-design/layout-non-alignment	Hsu+ CVPR'239, Li+ TVCG'217

Usage

• Install evaluate library

pip install evaluate

• Load the layout metric and then compute the score

import evaluate
import numpy as np

# Load the evaluation metric named "creative-graphic-design/layout-alignment"
alignment_score = evaluate.load("creative-graphic-design/layout-alignment")

# `batch_bbox` is a tensor representing (batch_size, max_num_elements, coordinates)
# and `batch_mask` is a boolean tensor representing (batch_size, max_num_elements).
batch_bbox = np.random.rand(512, 25, 4)
# Note that padded fields will be set to `False`
batch_mask = np.full((512, 25), fill_value=True)

# Add the batch of bboxes and masks to the metric
alignment_score.add_batch(batch_bbox=batch_bbox, batch_mask=batch_mask)
# Perform the computation of the evaluation metric
alignment_score.compute()

Reference

Footnotes

1. Heusel, Martin, et al. "Gans trained by a two time-scale update rule converge to a local nash equilibrium." Advances in neural information processing systems 30 (2017). ↩

2. Naeem, Muhammad Ferjad, et al. "Reliable fidelity and diversity metrics for generative models." International Conference on Machine Learning. PMLR, 2020. ↩

3. Kikuchi, Kotaro, et al. "Constrained graphic layout generation via latent optimization." Proceedings of the 29th ACM International Conference on Multimedia. 2021. ↩ ↩² ↩³

4. Arroyo, Diego Martin, Janis Postels, and Federico Tombari. "Variational transformer networks for layout generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021. ↩

5. Kong, Xiang, et al. "BLT: bidirectional layout transformer for controllable layout generation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. ↩

6. Lee, Hsin-Ying, et al. "Neural design network: Graphic layout generation with constraints." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020. ↩

7. Li, Jianan, et al. "Attribute-conditioned layout gan for automatic graphic design." IEEE Transactions on Visualization and Computer Graphics 27.10 (2020): 4039-4048. ↩ ↩² ↩³

8. Li, Jianan, et al. "LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators." International Conference on Learning Representations. 2019. ↩

9. Hsu, Hsiao Yuan, et al. "Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷