BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models

A pioneering framework that effectively reduces gender bias in neutral cases while maintaining gender faithfulness in explicit ones, thus providing a promising direction toward achieving selective fairness in VLMs.

🏴 Overview

Vision-Language Models (VLMs) often inherit social biases (e.g., gender stereotypes) from large-scale training data. Existing debiasing methods typically adopt a difference-unaware strategy, enforcing uniform treatment across groups—this may remove bias but also destroy legitimate semantic distinctions.

BioPro (Bias Orthogonal Projection) introduces a difference-aware fairness paradigm for VLMs:

• ✅ Remove bias in neutral contexts
• ✅ Preserve semantics in explicit contexts
• ✅ Maintain overall generation quality

Our method is training-free and operates directly in the representation space.

Core Idea

BioPro performs selective debiasing via orthogonal projection:

1. Bias Subspace Construction

   • Use counterfactual pairs (e.g., male vs female)
   • Extract bias directions via SVD

2. Orthogonal Projection

   • Remove bias components:

     h' = (I - UU^T)h
     

   • Preserve semantic components

3. Selective Debiasing

   • Apply projection only to neutral samples
   • Avoid over-debiasing explicit samples

4. Calibration (for Generation)

   • Balance gender distribution in generated images

Key Features

• Difference-Aware Fairness: Distinguishes neutral vs explicit contexts
• Training-Free: No fine-tuning required
• Selective Debiasing: Avoids semantic distortion
• Closed-Form Solution: Efficient projection computation
• Generalizable: Supports both discrete and continuous bias (e.g., brightness)
• Plug-and-Play: Works with existing VLMs

---

📊 Main Results

🖼️ Image Captioning

We evaluate BioPro on LLaVA-1.5 and LLaVA-NeXT using bias and semantic metrics.

Key Results

Method	BRₙ ↓	BRₑ	CBR ↓	METEOR ↑	CLIP-S ↑
LLaVA-1.5
Base	36.22	80.27	36.22	0.317	0.316
Prompt-1	21.83	61.83	28.58	0.325	0.316
Prompt-2	16.34	54.92	30.16	0.324	0.315
LIBRA	64.13	90.19	64.89	0.339	0.309
SFID	35.46	79.72	35.46	0.317	0.316
BioPro	23.01	68.74	25.74	0.315	0.315
w/o Selection	20.29	61.92	27.36	0.312	0.314

Method	BRₙ ↓	BRₑ	CBR ↓	METEOR ↑	CLIP-S ↑
LLaVA-NeXT
Base	15.87	72.55	15.87	0.237	0.330
Prompt-1	8.32	48.89	25.08	0.236	0.329
Prompt-2	7.08	38.69	34.59	0.235	0.330
LIBRA	68.97	93.42	72.06	0.263	0.314
SFID	16.34	73.38	16.36	0.238	0.330
BioPro	12.27	64.06	14.92	0.238	0.329
w/o Selection	11.33	55.52	20.45	0.236	0.328

Takeaways

• 🏆 Best overall fairness (CBR) on both models
• ⚖️ Effectively reduces bias on neutral samples (BRₙ ↓)
• 🎯 Preserves explicit gender faithfulness (BRₑ close to base)
• 🧠 Maintains semantic quality (METEOR / CLIP-S unchanged)
• 🔍 Selection module is crucial: removing it hurts balance (CBR ↑)

---

🎨 Text-to-Image Generation

We evaluate BioPro on FLUX.1-dev and FLUX.1-schnell.

Key Results

Method	Skew ↓	MR ↓	CLIP-S ↑	MUSIQ ↑
FLUX.1-dev
Base	93.2	0	0.291	75.42
BendVLM	88.6	0	0.286	74.38
SFID	93.8	0	0.289	75.56
Prompt-Projection	92.3	0	0.289	75.58
ForcePrompt	87.2	0	0.294	75.81
FairImagen	72.5	0	0.291	75.41
BioPro	67.8	0.2%	0.288	75.84
w/o Calibration	92.1	0	0.290	75.88

Method	Skew ↓	MR ↓	CLIP-S ↑	MUSIQ ↑
FLUX.1-schnell
Base	98.5	0	0.291	76.85
BendVLM	93.4	0.1%	0.285	73.53
SFID	99.0	0	0.291	77.09
Prompt-Projection	98.7	0	0.290	77.03
ForcePrompt	89.1	0	0.295	77.28
FairImagen	64.6	0	0.291	76.84
BioPro	60.4	0.2%	0.290	76.69
w/o Calibration	97.1	0	0.291	77.02

Takeaways

• 🏆 Best debiasing performance (lowest Skew) across both models
• ⚖️ Achieves balanced gender distribution in neutral prompts
• 🎯 Maintains explicit faithfulness (very low MR ≤ 0.2%)
• 🧠 Keeps semantic alignment (CLIP-S ≈ baseline)
• 🎨 Preserves image quality (no drop in MUSIQ)
• 🔧 Calibration is essential: removing it collapses debiasing

---

🔥 Overall Summary

• BioPro achieves state-of-the-art fairness in both:
  • Image Captioning
  • Text-to-Image Generation
• Provides consistent gains without retraining
• Demonstrates a strong balance between:
  • Fairness
  • Faithfulness
  • Generation quality

---

⚡️ Quickstart Guide

1. Debiasing for image captioning

cd image_captioning

python debias_vlm/debias_only_P-perp.py
python calculate_bias_debiased.py

2. Debiasing for text-to-image generation

Update the paths in sh/run.sh:

cd image_generation_k=2
python calculate_projection/compute_projections.py
python flux_debiased_k=2_calibration.py

---

❓ FAQ

Q: Does BioPro require retraining?

A: No. BioPro is training-free and works via representation-level projection.

---

Q: Will debiasing hurt performance?

A: No. It preserves semantic quality:

• Captioning: METEOR / CLIP unchanged
• Generation: CLIP-S / MUSIQ unchanged

---

Q: What is difference-aware fairness?

A: BioPro debiases only neutral cases while preserving explicit attributes, avoiding over-debiasing.

---

Q: Why is calibration needed for generation?

A: Because generated images must have gender, calibration ensures balanced outputs for neutral prompts.

---

Q: Can BioPro generalize to other biases?

A: Yes. It supports both discrete (e.g., gender) and continuous (e.g., brightness) biases.

---

📄 License

This project is licensed under the MIT License.