Compute Efficiency Benchmark

Overview

LLMs are becoming part of production systems across many industries. As adoption grows, compute is becoming an increasingly important engineering concern. Larger models, longer prompts, and more complex workflows improve capabilities, but they also increase cost, latency, and infrastructure demand.

Managing compute is therefore becoming as important as managing model quality. The goal is not simply to use the most capable model available, but to understand the trade-off between performance and efficiency.

This repository demonstrates one practical approach to managing GenAI compute through context compression. By reducing the amount of information sent to a model before inference, it is possible to lower token consumption while measuring the effect on application performance.

The repository contains a reproducible benchmark that evaluates how context compression and model selection influence accuracy, token usage, latency, and estimated inference cost using a long-context question-answering task.

Questions explored

The benchmark investigates three practical questions:

1. How much context can be removed before model performance starts to degrade?
2. How much compute can be saved through prompt compression?
3. How does model selection compare with context compression as a strategy for improving efficiency?

Benchmark design

A Q&A GenAI application was benchmarked to illustrate the performance-cost trade-off. The evaluation measures answer accuracy and normalized compute cost across:

• three model sizes: gpt-5, gpt-5-mini, and gpt-5-nano;
• four context settings: full context, 20% reduction, 40% reduction, and 60% reduction;
• 300 questions from the QuALITY long-context question-answering dataset.

Only the article context is compressed. The question and answer options remain unchanged for every run.

Accuracy versus normalized compute cost across model and context configurations. Cost is normalized to gpt-5 with full context and is based on measured token usage and relative model price factors.

Main findings

• Compute cost is controllable. gpt-5 maintains its baseline performance at 20% context reduction and declines moderately at 40% and 60%. Smaller models trade accuracy more quickly as context is reduced.
• Model choice dominates cost. gpt-5-mini achieves accuracy close to gpt-5 at roughly one-fifth of the normalized cost, while gpt-5 is approximately 25 times more expensive than gpt-5-nano in this comparison.
• Small accuracy gains can require disproportionately higher cost. Selecting a model and context configuration is therefore a strategic decision, not only a technical one.

The benchmark does not aim to identify one universally optimal configuration. It provides a repeatable way to find an acceptable operating point for a specific application, based on its required accuracy, cost, and latency.

How it works

The workflow has four main stages:

1. Prepare the dataset. Validate the QuALITY input and confirm that each question has four answer options and a valid gold answer.
2. Compress the context. Run LLMLingua-2 locally to generate context variants with keep-rates of 0.8, 0.6, and 0.4.
3. Run the benchmark. Evaluate the original and compressed contexts with each Azure OpenAI deployment.
4. Analyze the results. Aggregate accuracy, token usage, latency, compression statistics, and estimated cost into summary tables and plots.

Compression rates in this repository are described as context reduction:

Context setting	LLMLingua-2 keep-rate	Approximate context reduction
Full	1.0	0%
Compression 20%	0.8	20%
Compression 40%	0.6	40%
Compression 60%	0.4	60%

Repository structure

artifacts/              Publishable figures
scripts/
  analyze.py            Aggregate results and generate plots
  analyze_context_quality.py
                        Compare preservation metrics with QA utility
  benchmark.py          Run Azure OpenAI question answering
  calculate_context_metrics.py
                        Calculate and cache context-preservation metrics
  compress.py           Generate compressed dataset variants
  test_context_models.py
                        Test the optional local metric models
  validate_dataset.py   Validate the QuALITY dataset
results/
  eval.sqlite           Cached benchmark results
  analysis/             Generated summaries and plots
data/
  quality/              Original QuALITY files
  quality_comp_*/       Compressed dataset variants

Requirements

• Python 3.10 or newer
• Azure OpenAI deployments for the models being evaluated
• Internet access for the first LLMLingua-2 model download
• Local disk space for the dataset, model cache, and benchmark results

Dataset validation and context compression run locally. Benchmark evaluation requires Azure OpenAI credentials and incurs API usage charges.

Setup

Run the following commands from the repository root.

1. Create and activate an environment

python -m venv .venv
.venv\Scripts\Activate.ps1

If an existing environment is located in the parent folder, activate it instead:

..\.venv\Scripts\Activate.ps1

2. Install the project

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

3. Configure Azure OpenAI

Copy-Item .env.example .env

Edit .env with the Azure OpenAI credentials and deployment names available in your environment. Do not commit .env.

LLMLingua-2 setup

Context compression uses the following Hugging Face model:

microsoft/llmlingua-2-xlm-roberta-large-meetingbank

No separate download command is required. The first run of scripts/compress.py downloads the model and stores it in the local Hugging Face cache. Later runs reuse the cached files.

The current implementation uses:

• LLMLingua-2 mode with use_llmlingua2=True;
• CPU execution with device_map="cpu";
• paragraph-based chunking with a maximum of 1,400 characters per chunk;
• keep-rates supplied through --rates;
• punctuation and line breaks preserved as forced or chunk-ending tokens.

The first run requires internet access and may take longer while the model is downloaded and initialized. Compression itself is CPU-intensive.

Dataset

The benchmark uses QuALITY: Question Answering with Long Input Texts, introduced by Pang et al. at ACL 2022.

• QuALITY repository
• QuALITY v1.0.1 data

Download the QuALITY v1.0.1 files and place them under data/quality/:

data/
└── quality/
    ├── QuALITY.v1.0.1.dev
    ├── QuALITY.v1.0.1.train
    ├── QuALITY.v1.0.1.test
    ├── QuALITY.v1.0.1.htmlstripped.dev
    ├── QuALITY.v1.0.1.htmlstripped.train
    └── QuALITY.v1.0.1.htmlstripped.test

The files can be copied from a local clone of the official repository:

git clone https://github.com/nyu-mll/quality.git
cp quality/data/v1.0.1/QuALITY.v1.0.1* data/quality/

The benchmark described in this repository uses:

data/quality/QuALITY.v1.0.1.htmlstripped.train

Run the benchmark

Run the workflow one step at a time. Start with small validation and benchmark runs before launching the full evaluation.

1. Validate the dataset

This command checks that the dataset exists, can be read, contains questions with four options, and has valid gold answers. It runs offline.

python scripts/validate_dataset.py

For a quick validation of the first 20 questions:

python scripts/validate_dataset.py --limit 20

The command reports the number of questions validated in the local file.

2. Test context compression

Start by compressing three articles at a single keep-rate:

python scripts/compress.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --rates 0.8 `
    --limit 3

The compressed output is written to:

data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train

Once the test succeeds, generate all compressed variants:

python scripts/compress.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --rates 0.8,0.6,0.4

3. Run a small Azure OpenAI evaluation

Evaluate three questions with gpt-5-nano before running the complete benchmark:

python scripts/benchmark.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --deployment gpt-5-nano `
    --db results/eval.sqlite `
    --n 3

Then evaluate the compressed version:

python scripts/benchmark.py `
    data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train `
    --deployment gpt-5-nano `
    --db results/eval.sqlite `
    --n 3

Results are cached in results/eval.sqlite. Re-running the same dataset, deployment, and prompt version skips completed questions and avoids duplicate API calls.

4. Run the full evaluation

The published benchmark uses the first 300 questions for each model and context configuration.

Full context

python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

20% context reduction

python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

40% context reduction

python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

60% context reduction

python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

5. Analyze the results

python scripts/analyze.py --db results/eval.sqlite --out results/analysis

The analysis step writes summary tables, JSON output, and generated plots to:

results/analysis/

The publishable figure is written to:

artifacts/accuracy_vs_normalized_cost.png

Measurements

For each question and configuration, the benchmark records:

• answer correctness;
• input, output, and total tokens;
• request latency;
• response parsing status;
• compression ratio;
• local compression time.

Accuracy is calculated against the QuALITY gold answer. Monetary cost is derived separately from measured token usage using documented model prices or relative price factors.

Normalized cost is calculated relative to the baseline configuration:

normalized cost = configuration cost / gpt-5 full-context cost

Token savings and latency are directly measured. CO2 should be treated as a proxy unless an explicit energy and emissions methodology is added for both local compression and cloud inference.

Evaluating context-preservation metrics against question-answering accuracy

The first part of this repository measures whether context compression reduces compute while preserving question-answering accuracy. This additional analysis uses those same compressed contexts and benchmark results to explore a related question: does a compressed text receiving a high preservation score mean that the application still performs well?

Common metrics such as ROUGE, BERTScore, and semantic similarity compare the original and compressed texts. They are standardized and useful at scale, but they do not directly test whether the evidence required by the application was preserved. A small omission, such as a number, condition, or exception, may have little effect on overall text similarity while changing the correct answer.

The compressed context also receives the same preservation score regardless of which model subsequently uses it. QA accuracy, however, can vary by model because different models may recover relevant evidence from the remaining context with different levels of success.

Text-preservation metrics describe how much the compressed context resembles the original. QA accuracy measures whether that context still supports the task for a particular model.

Metrics calculated

The analysis considers a broad set of common text-preservation metrics:

• Token retention and token savings: the proportion of the original tokens retained or removed after compression.
• Jaccard similarity: overlap between the sets of unique tokens in the original and compressed texts.
• TF-IDF cosine similarity: similarity between vocabulary-weighted document representations, implemented with scikit-learn.
• ROUGE-1, ROUGE-2, and ROUGE-L: unigram, bigram, and longest-common-subsequence overlap, reported as precision, recall, and F1 using rouge-score.
• BLEU: n-gram precision between the compressed and original text, calculated with smoothing using NLTK.
• METEOR: token alignment based on exact matches, stems, and WordNet synonyms, implemented with NLTK.
• BERTScore: contextual token similarity, reported as precision, recall, and F1 using the BERTScore implementation with distilbert-base-uncased.
• Sentence-level semantic coverage: the semantic similarity between each original sentence and its closest match in the compressed text, summarized by the mean and 10th percentile using all-MiniLM-L6-v2.

Together, these measures provide different views of lexical overlap, semantic similarity, and information retention. They are compared descriptively with QA accuracy; the analysis does not perform correlation, ROC-AUC, or predictive modelling.

Context-preservation scores across the evaluated compression levels.

1. Install the metric dependencies

python -m pip install -e ".[metrics]"
python -c "import nltk; nltk.download('wordnet'); nltk.download('omw-1.4')"

BERTScore defaults to distilbert-base-uncased, while semantic coverage uses sentence-transformers/all-MiniLM-L6-v2. The public models normally require no Hugging Face login and are downloaded on first use. Alternatively, configure local model directories in .env:

CONTEXT_BERTSCORE_MODEL=C:\path\to\distilbert-base-uncased
CONTEXT_SEMANTIC_MODEL=C:\path\to\all-MiniLM-L6-v2

The following command tests both model-backed metrics on one entry pair before a full run:

python scripts/test_context_models.py `
    --original data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --compressed data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train

2. Calculate and cache the preservation metrics

python scripts/calculate_context_metrics.py `
    --original data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --compressed-root data `
    --out results/analysis/context_quality

This writes entry-level results to document_metrics.csv and document_metrics.json. Compatible set_unique_id and compression-variant rows are reused on subsequent runs, so only missing or changed rows are calculated. Use --force to recalculate all rows. This step is offline and does not call Azure OpenAI.

For a lightweight smoke test without the model-backed metrics:

python scripts/calculate_context_metrics.py `
    --original data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --compressed-root data `
    --out results/analysis/context_quality_test `
    --max-entries 3 `
    --skip-bertscore `
    --skip-semantic

3. Combine the metrics with QA accuracy and generate plots

After the preservation metrics and QA benchmark have been calculated, run:

python scripts/analyze_context_quality.py `
    --db results/eval.sqlite `
    --metrics-dir results/analysis/context_quality `
    --out results/analysis/context_quality

The analysis writes metric_comparison.csv and metric_comparison.json, one plot per metric, an all-metrics plot, and a representative overview plot. These outputs place the preservation scores alongside the existing QA accuracy for gpt-5, gpt-5-mini, and gpt-5-nano at full context and 20%, 40%, and 60% context reduction.

BLEURT and SummaC are not part of the default experiment because their additional installation and runtime requirements would make this exploratory analysis substantially heavier.

Interpreting the results

The observed results are specific to this dataset, sample, prompt, model deployments, and compression configuration. They should not be treated as universal performance guarantees.

The main value of the repository is the evaluation pattern:

1. define the required application quality;
2. benchmark multiple model and context configurations;
3. measure the full cost-performance trade-off;
4. select the least compute-intensive configuration that satisfies the application requirements.

This same approach can be adapted to document Q&A, retrieval-augmented generation, enterprise search, meeting analysis, and other long-context applications.

References

• LLMLingua-2 project page
• LLMLingua-2 paper
• Microsoft LLMLingua repository
• LLMLingua-2 model
• QuALITY dataset repository