-- **Develop a Controlled Data Pipeline**
- Create a reproducible, configurable pipeline for generating **text, image, and video** with precise control over **demographic** and **contextual** variables.
+- **Explainable, accountable AI**
+ Develop tools and benchmarks for **interpretability**, **fairness**, and **transparency** in generative and multimodal AI, aligned with AIXpert’s vision.
-- **Enable Fairness-Aware Benchmarking**
- Provide tools to build matched **baseline vs. fairness-aware** datasets for bias diagnosis and mitigation experiments.
+- **Trust, risk, and security in agentic AI**
+ Advance **TRiSM** (Trust, Risk, and Security Management) and transparency frameworks for safe, explainable agentic and multi-agent systems.
-- **Support Multi-Domain Risk Analysis**
- Generate multimodal data for **hiring, healthcare, legal, education**, and more, covering risks like **bias, toxicity, misinformation**.
+- **Multimodal and real-world evaluation**
+ Create benchmarks and datasets for **audio-video understanding**, **vision-language** assessment, and **fairness** across domains and demographics.
-- **Integrate Agentic AI for Automation**
- Orchestrate generation and QC with **CrewAI** and **custom LLM agents** (prompts, assets, annotations, validation).
+- **Define and assess AI trustworthiness**
+ Establish **measurable criteria** and evaluation suites for reliability, factuality, and ethical alignment of AI systems.
-- **Advance Interpretability & Explainability**
- Combine **zero-shot LLM explainers** and fairness metrics to produce **interpretable** assessments and visualizations.
+- **Real-world impact**
+ Validate approaches across **healthcare, employment, education**, and high-stakes domains through pilots, benchmarks, and open releases.
-- **Foster Open Research & Collaboration**
- Share configs, tools, and docs openly to enable **reproducible research** and **transparent governance**.
+- **Open, reproducible research**
+ Share **code**, **datasets**, and **documentation** openly to support reproducible research, benchmarking, and transparent governance.
---
-## Pipeline
-
-
-
----
-
## Recent updates
-- :material-rocket-launch: **Released data generation pipeline** (multimodal, configurable, agent-orchestrated).
-- :material-robot: **Single-agent pipeline** prototype for rapid dataset bootstrapping.
-- :material-file-document: NeurIPS 2025 LLM-eval Workshop paper: [_Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment_](https://arxiv.org/abs/2509.19659)
-- :material-file-document: Preprint: [_TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems_](https://arxiv.org/abs/2506.04133)
-- :material-file-document-edit-outline: TechRxiv article: [_Responsible Agentic Reasoning and AI Agents—A Critical Survey_](https://www.techrxiv.org/users/574774/articles/1329333-responsible-agentic-reasoning-and-ai-agents-a-critical-survey?mode=edit)
-- :material-post-outline: Poster: **Single-Agent TRiSM** (NeurIPS LAW)
-
----
+- :material-newspaper: **AIXpert news** — Our work was highlighted on the [AIXpert project website](https://aixpert-project.eu/2026/01/28/advancing-trustworthy-explainable-and-responsible-ai-at-neurips-2025/): *Advancing Trustworthy, Explainable, and Responsible AI at NeurIPS 2025* (Bias in the Picture, HumaniBench, Carbon Literacy, and more).
+- :material-play-circle: **SONIC-O1** — Paper: [_A Real-World Benchmark for Evaluating MLLMs on Audio-Video Understanding_](https://arxiv.org/abs/2601.21666) (arXiv).
+- :material-database: **SONIC-O1** — Dataset on [Hugging Face](https://huggingface.co/datasets/vector-institute/sonic-o1) (231 videos, ~60h, 4,958 QAs, 13 domains, demographic metadata).
+- :material-github: **SONIC-O1** — [Code](https://github.com/VectorInstitute/sonic-o1) and evaluation pipeline (summarization, MCQ, temporal localization).
+- :material-medal: **SONIC-O1** — [Leaderboard](https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard) for model comparisons and fairness analysis.
-
-
-> Have feedback or want to contribute? See the [:material-account-group: Team](team.md) page and open an issue or pull request.
+[:material-arrow-right: **View full list**](updates.md){ .md-button .md-button--primary }
---
-## License
-
-This code in this repo is released under the **MIT License**.
+> Have feedback or want to contribute? See the [:material-account-group: Team](team.md) page and open an issue or pull request.
diff --git a/docs/papers.md b/docs/papers.md
new file mode 100644
index 0000000..3061943
--- /dev/null
+++ b/docs/papers.md
@@ -0,0 +1,63 @@
+# Papers
+
+Selected publications and preprints from the AIXpert project. Each entry links to arXiv (or equivalent) where available.
+
+---
+
+## AIXpert project papers
+
+### SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
+
+**Paper** ·
-
---
diff --git a/docs/updates.md b/docs/updates.md
new file mode 100644
index 0000000..3cb1818
--- /dev/null
+++ b/docs/updates.md
@@ -0,0 +1,21 @@
+# Updates
+
+Full list of recent papers, releases, and news.
+
+---
+
+- :material-newspaper: **AIXpert news** — Our work was highlighted on the [AIXpert project website](https://aixpert-project.eu/2026/01/28/advancing-trustworthy-explainable-and-responsible-ai-at-neurips-2025/): *Advancing Trustworthy, Explainable, and Responsible AI at NeurIPS 2025* (Bias in the Picture, HumaniBench, Carbon Literacy, and more).
+- :material-play-circle: **SONIC-O1** — Paper: [_A Real-World Benchmark for Evaluating MLLMs on Audio-Video Understanding_](https://arxiv.org/abs/2601.21666) (arXiv).
+- :material-database: **SONIC-O1** — Dataset on [Hugging Face](https://huggingface.co/datasets/vector-institute/sonic-o1) (231 videos, ~60h, 4,958 QAs, 13 domains, demographic metadata).
+- :material-github: **SONIC-O1** — [Code](https://github.com/VectorInstitute/sonic-o1) and evaluation pipeline (summarization, MCQ, temporal localization).
+- :material-medal: **SONIC-O1** — [Leaderboard](https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard) for model comparisons and fairness analysis.
+- :material-file-document: **F-DPO** — Paper: [_Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning_](https://arxiv.org/abs/2601.03027) (arXiv).
+- :material-database: **F-DPO** — Factuality-aware preference [dataset](https://huggingface.co/datasets/vector-institute/Factuality_Alignment) on Hugging Face.
+- :material-github: **F-DPO** — [Code](https://github.com/VectorInstitute/Factual-Preference-Alignment) and [project page](https://vectorinstitute.github.io/Factual-Preference-Alignment/) (factuality-aware DPO, no reward model).
+- :material-book-open-variant: **Survey (forthcoming)** — _Transparency in Agentic AI: A Survey of Interpretability, Explainability, and Governance_. Surveys traditional methods, aligns them to agent layers, and proposes a five-axis framework for organizing agentic transparency.
+- :material-rocket-launch: **Released data generation pipeline** (multimodal, configurable, agent-orchestrated).
+- :material-robot: **Single-agent pipeline** prototype for rapid dataset bootstrapping.
+- :material-file-document: **Paper:** [_Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment_](https://arxiv.org/abs/2509.19659) (NeurIPS 2025 LLM-eval Workshop)
+- :material-file-document: **Paper:** [_TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems_](https://arxiv.org/abs/2506.04133)
+- :material-file-document-edit-outline: **Paper:** [_Responsible Agentic Reasoning and AI Agents—A Critical Survey_](https://www.techrxiv.org/users/574774/articles/1329333-responsible-agentic-reasoning-and-ai-agents-a-critical-survey?mode=edit) (TechRxiv)
+- :material-post-outline: **Paper:** **Single-Agent TRiSM** (Poster, NeurIPS LAW)
diff --git a/docs/user_guide.md b/docs/user_guide.md
index 84202cc..643e7a4 100644
--- a/docs/user_guide.md
+++ b/docs/user_guide.md
@@ -1,203 +1,70 @@
# User Guide
-Transparent tools and standardized benchmarks for **fair**, **explainable**, and **accountable** generative AI. This guide introduces modules, setup, and usage patterns for fairness-aware data generation and analysis.
-
----
-
-## Table of Contents
-
-- [Getting Started](#getting-started)
-- [Core Concepts](#core-concepts)
-- [Contributing and Documentation](#contributing-and-documentation)
+This guide points you to **each project’s repo and docs**. Installation and quick start are in the linked READMEs and project pages.
---
## Getting Started
-**Four Essential Questions**
+- **What is this?** — A research framework and toolkit for generating, evaluating, and mitigating bias in multimodal AI systems.
+- **Why fairness-aware data?** — Controlled datasets isolate bias-inducing factors for targeted experiments on fairness and representation.
+- **Why agentic AI?** — We use LLM agents (e.g. CrewAI) to scale prompt, image, and metadata generation.
+- **Who is it for?** — Researchers and practitioners studying or benchmarking bias in generative models.
-| Question | Answer |
-|-----------|---------|
-| **What is the project about?** | A research framework and toolkit for generating, evaluating, and mitigating bias in multimodal AI systems. |
-| **Why Fairness-Aware Synthetic Data?** | Controlled datasets isolate bias-inducing factors, allowing targeted experiments on fairness, explainability, and representation. |
-| **Why Agentic AI?** | We use autonomous LLM agents (via **CrewAI**) to scale prompt, image, and metadata generation. |
-| **Who is project for?** | Researchers, data scientists, and fairness practitioners studying or benchmarking bias in generative models. |
+Each project has its **own repository or module**; use the sections below for summaries and links.
---
-### Installation
-
-#### From source (recommended)
-
-```bash
-git clone https://github.com/VectorInstitute/vector-aixpert.git
-cd vector-aixpert
-uv sync
-```
-
-#### Optional groups
+## SONIC-O1
-```bash
-# Development extras
-uv sync --dev
-# Documentation build
-uv sync --no-group docs
-```
+Real-world benchmark for evaluating multimodal LLMs on **audio-video understanding**: short to long-form videos across 13 conversational domains (job interviews, medical, legal, etc.), with three tasks—summarization, multiple-choice QA, and temporal localization—and demographic metadata for fairness analysis.
-#### Verify installation
-
-```bash
-pytest -q
-mkdocs serve
-```
-
----
-
-### Quick Starts
-
-Minimal commands to explore modules.
-
-```bash
-# 1. Set up environment
-uv sync
-source .venv/bin/activate
-
-# 2. Run controlled image generation
-cd src/aixpert/controlled_images/
-uv run python src/main.py \
- --config configs/img_gen_config.yaml
-
-# 3. Generate synthetic data (text)
-cd src/aixpert/data_generation/synthetic_data_generation/nlp/
-uv run main.py \
- --config config.yaml \
- --stage all
-
-# 4. Generate synthetic data (images)
-cd src/aixpert/data_generation/synthetic_data_generation/images/
-uv run main.py all_stages \
- --config_file ../../config.yaml \
- --prompt_yaml prompt_paths.yaml \
- --domain hiring \
- --risk security_risks
-
-# 5. Compute fairness metrics
-cd src/aixpert/toxicity_fairness_analysis/
-
-uv run python scripts/download_data.py \
- --dataset jigsaw \
- --out data/jigsaw.parquet \
- --sample 50000
-
-uv run python scripts/llm_zero_shot_explain.py \
- --in data/jigsaw.parquet \
- --text_col comment_text \
- --task toxicity \
- --out outputs/zs_preds.parquet \
- --model distilgpt2 \
- --max_rows 1000 \
- --ig_rows 25 \
- --ig_steps 32 \
- --save_heatmaps \
- --force_float32 \
- --label_col target
- --id_cols male female black white muslim jewish
-```
-
-Each module provides a focused README with configuration details and output examples.
+**Links:** [GitHub](https://github.com/VectorInstitute/sonic-o1) · [Project page](https://vectorinstitute.github.io/sonic-o1/) · [Dataset](https://huggingface.co/datasets/vector-institute/sonic-o1) · [Leaderboard](https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard)
---
-### Standard Usage
+## F-DPO (Factual Preference Alignment)
-Typical workflow for fairness-aware data generation:
+Factuality-aware Direct Preference Optimization (F-DPO): extends DPO with binary factuality labels and a factuality-aware margin to reduce LLM hallucinations without an auxiliary reward model. Single-stage and compute-efficient.
-1. **Generate controlled data**
- Create matched datasets (e.g., gender, occupation, or ethnicity pairs).
-2. **Run agentic generation pipeline**
- Use CrewAI agents for multimodal prompt, image, and metadata generation.
-3. **Perform fairness analysis**
- Compute bias metrics such as Statistical Parity or Equal Opportunity.
-4. **Visualize or export results**
- Generate structured outputs or Hugging Face datasets for benchmarking.
+**Links:** [GitHub](https://github.com/VectorInstitute/Factual-Preference-Alignment) · [Project page](https://vectorinstitute.github.io/Factual-Preference-Alignment/) · [Dataset](https://huggingface.co/datasets/vector-institute/Factuality_Alignment)
---
-## Core Concepts
+## Modules in AIXpert
+
+These modules live in the main [AIXpert](https://github.com/VectorInstitute/AIXpert) repository. Clone once, run `uv sync`, then use the READMEs below for setup and commands.
### Controlled Images
-Generates baseline vs fairness-aware image sets for occupations or social groups.
-Supports configurable attributes, matched prompts, and consistent random seeds for reproducibility.
+Baseline vs fairness-aware image sets for occupations or social groups; configurable attributes, matched prompts, and reproducible seeds.
-### Synthetic Data Generation
+**Links:** [Module README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/controlled_images/README.md)
-Multi-modal data synthesis modules under:
+### Synthetic Data Generation
-* `synthetic_data_generation/images` — image + VQA pairs
-* `synthetic_data_generation/nlp` — textual scenes and MCQs
-* `synthetic_data_generation/videos` — Veo/Gemini video generation
+Multi-modal synthesis: image + VQA pairs, textual scenes and MCQs, and video generation (Veo/Gemini). Driven by LLM-designed prompts and metadata templates.
-Each generator is driven by LLM-designed prompts and metadata templates.
+**Links:** [Images README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/images/README.md) · [NLP README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/nlp/README.md)
### Agent Pipeline (CrewAI)
-Implements single-agent orchestration to chain prompt → image → metadata generation.
-Enables autonomous large-scale data creation using structured JSON task definitions.
+Single-agent orchestration for prompt → image → metadata generation and large-scale data creation with structured JSON task definitions.
+**Links:** [Module README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/agent_pipeline/README.md)
### Fairness & Explainability
-Evaluates generated data and model outputs via:
-
-* **Statistical metrics** — Statistical Parity, Equal Opportunity
-* **Zero-shot explainers** — integrated gradients, concept attributions
-* **Visualization tools** — disparity plots, attribution maps
-
-### Module Quick Start (one-liners + deep links)
-
-Each core module has its own README with commands, configurations, and sample outputs.
-
-* **Controlled Images** — Generate matched baseline vs fairness-aware images across professions.
- ➜ [`View Module README`](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/controlled_images/README.md)
-
-* **Agent Pipeline (CrewAI)** — Single-agent orchestration for prompt, image, and metadata generation.
- ➜ [`View Module README`](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/agent_pipeline/README.md)
-
-* **Synthetic Data · Images** — Domain- and risk-specific image prompts and VQA pairs.
- ➜ [`View Module README`](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/images/README.md)
-
-* **Synthetic Data · NLP** — Scene descriptions and MCQ generation for text pipelines.
- ➜ [`View Module README`](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/nlp/README.md)
+Statistical metrics (e.g. Statistical Parity, Equal Opportunity), zero-shot explainers (integrated gradients, concept attributions), and visualization (disparity plots, attribution maps). Lives under `toxicity_fairness_analysis` in the AIXpert repo.
+**Links:** [AIXpert](https://github.com/VectorInstitute/AIXpert)
---
## Contributing and Documentation
-See [CONTRIBUTING.md](https://github.com/VectorInstitute/vector-aixpert/blob/main/CONTRIBUTING.md) for:
+- [CONTRIBUTING.md](https://github.com/VectorInstitute/AIXpert/blob/main/CONTRIBUTING.md): coding standards (PEP8, Google docstrings), pre-commit (`ruff`, `mypy`, `typos`, `nbQA`), branching, and tests.
-* Coding standards and style guide (PEP8 + Google docstrings)
-* Pre-commit setup (`ruff`, `mypy`, `typos`, `nbQA`)
-* Branching and PR workflow
-* Test coverage requirements
+**Docs locally:** `uv sync --no-group docs` then `mkdocs serve` → [http://127.0.0.1:8000](http://127.0.0.1:8000).
-### Docs build
-
-```bash
-uv sync --no-group docs
-mkdocs serve
-```
-
-The site will be live at [http://127.0.0.1:8000](http://127.0.0.1:8000).
-
-### Testing and Standards
-
-```bash
-pytest -v
-pre-commit run --all-files
-```
-
-Continuous integration runs these via GitHub Actions (`code_checks.yml`, `unit_tests.yml`, `integration_tests.yml`).
-
----
+**CI:** GitHub Actions (`code_checks.yml`, `unit_tests.yml`, `integration_tests.yml`).
diff --git a/mkdocs.yml b/mkdocs.yml
index 30b8485..ecdf7f0 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -5,8 +5,6 @@ extra:
social:
- icon: fontawesome/brands/discord
link: 404.html
- - icon: fontawesome/brands/github
- link: https://github.com/VectorInstitute/vector-aixpert
markdown_extensions:
- attr_list
- admonition
@@ -39,8 +37,9 @@ markdown_extensions:
nav:
- Home: index.md
- User Guide: user_guide.md
+ - Updates: updates.md
# - API Reference: api.md
- - Technical Report: assets/aixpert-technical-report.pdf
+ - Papers: papers.md
- Team: team.md
- About: about.md
plugins:
@@ -59,9 +58,7 @@ plugins:
show_root_toc_entry: false
show_symbol_type_heading: true
show_symbol_type_toc: true
-repo_url: https://github.com/VectorInstitute/vector-aixpert
-repo_name: VectorInstitute/vector-aixpert
-site_name: vector-aixpert
+site_name: AIXpert
theme:
custom_dir: docs/overrides
favicon: assets/favicon-48x48.svg
From 00c4e3a84902d4f54aaea9950dd40d8e3a2e25a0 Mon Sep 17 00:00:00 2001
From: aravind-3105
Date: Wed, 4 Feb 2026 14:16:41 -0500
Subject: [PATCH 2/2] Update repository links in user guide to point to
vector-aixpert
---
docs/user_guide.md | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/user_guide.md b/docs/user_guide.md
index 643e7a4..3db65f8 100644
--- a/docs/user_guide.md
+++ b/docs/user_guide.md
@@ -33,37 +33,37 @@ Factuality-aware Direct Preference Optimization (F-DPO): extends DPO with binary
## Modules in AIXpert
-These modules live in the main [AIXpert](https://github.com/VectorInstitute/AIXpert) repository. Clone once, run `uv sync`, then use the READMEs below for setup and commands.
+These modules live in the main [AIXpert](https://github.com/VectorInstitute/vector-aixpert) repository. Clone once, run `uv sync`, then use the READMEs below for setup and commands.
### Controlled Images
Baseline vs fairness-aware image sets for occupations or social groups; configurable attributes, matched prompts, and reproducible seeds.
-**Links:** [Module README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/controlled_images/README.md)
+**Links:** [Module README](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/controlled_images/README.md)
### Synthetic Data Generation
Multi-modal synthesis: image + VQA pairs, textual scenes and MCQs, and video generation (Veo/Gemini). Driven by LLM-designed prompts and metadata templates.
-**Links:** [Images README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/images/README.md) · [NLP README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/nlp/README.md)
+**Links:** [Images README](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/images/README.md) · [NLP README](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/synthetic_data_generation/nlp/README.md)
### Agent Pipeline (CrewAI)
Single-agent orchestration for prompt → image → metadata generation and large-scale data creation with structured JSON task definitions.
-**Links:** [Module README](https://github.com/VectorInstitute/AIXpert/blob/main/src/aixpert/data_generation/agent_pipeline/README.md)
+**Links:** [Module README](https://github.com/VectorInstitute/vector-aixpert/blob/main/src/aixpert/data_generation/agent_pipeline/README.md)
### Fairness & Explainability
Statistical metrics (e.g. Statistical Parity, Equal Opportunity), zero-shot explainers (integrated gradients, concept attributions), and visualization (disparity plots, attribution maps). Lives under `toxicity_fairness_analysis` in the AIXpert repo.
-**Links:** [AIXpert](https://github.com/VectorInstitute/AIXpert)
+**Links:** [AIXpert](https://github.com/VectorInstitute/vector-aixpert)
---
## Contributing and Documentation
-- [CONTRIBUTING.md](https://github.com/VectorInstitute/AIXpert/blob/main/CONTRIBUTING.md): coding standards (PEP8, Google docstrings), pre-commit (`ruff`, `mypy`, `typos`, `nbQA`), branching, and tests.
+- [CONTRIBUTING.md](https://github.com/VectorInstitute/vector-aixpert/blob/main/CONTRIBUTING.md): coding standards (PEP8, Google docstrings), pre-commit (`ruff`, `mypy`, `typos`, `nbQA`), branching, and tests.
**Docs locally:** `uv sync --no-group docs` then `mkdocs serve` → [http://127.0.0.1:8000](http://127.0.0.1:8000).
**Dataset** · 
**Leaderboard** · 
+
+**Authors:** Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum, Deval Pandya, Shaina Raza.
+
+SONIC-O1, a fully human-verified real-world audio-video benchmark with 4,958 annotations across 13 conversational domains. We evaluate multimodal models on video summarization, evidence-grounded QA, and temporal event localization, and release an extensible evaluation suite to support reproducible benchmarking and robustness analysis.
+
+### Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning
+
+**Paper** · 
+
+**Authors:** Shaina Raza (Vector Institute), Ranjan Sapkota, Manoj Karkee (Cornell University), Christos Emmanouilidis (University of Groningen).
+
+Critical survey of responsible agentic reasoning and AI agents.
+
+### Evaluating and Regulating Agentic AI: A Study of Benchmarks, Metrics and Regulation
+
+**Paper** (TechRxiv) ·