Skip to content

Add TestRouter leaderboard submission - #183

Closed
TreeTXT wants to merge 2 commits into
RouteWorks:mainfrom
TreeTXT:test-router-leaderboard
Closed

Add TestRouter leaderboard submission#183
TreeTXT wants to merge 2 commits into
RouteWorks:mainfrom
TreeTXT:test-router-leaderboard

Conversation

@TreeTXT

@TreeTXT TreeTXT commented Aug 19, 2026

Copy link
Copy Markdown

Summary

  • add the TestRouter configuration and full/robustness prediction files
  • register qwen/qwen3.7-plus as a universal model and add its token pricing
  • add SinapisAI inference support for the four-model TestRouter pool

Model pool

  • qwen/qwen3.5-flash-02-23
  • deepseek/deepseek-v4-flash
  • qwen/qwen3.7-plus
  • deepseek/deepseek-v4-pro

The Qwen3.7 Plus prices are expressed in USD per million tokens using the supplied conversion rate of 1 USD = 6.74 CNY.

Local evaluation

  • RouterArena score: 0.7036
  • accuracy: 70.68%
  • total cost: $1.237440
  • average cost per 1K queries: $0.1473
  • robustness score: 0.9143
  • optimal selection: 0.7605
  • optimal cost: 0.6439
  • optimal accuracy: 0.8293

Validation

  • python router_inference/check_config_prediction_files.py TestRouter full --check-generated-result
  • python router_inference/check_config_prediction_files.py TestRouter robustness
  • python -m py_compile llm_inference/model_inference.py universal_model_names.py
  • git diff --check

Both RouterArena validation runs passed. The full file contains 8,400 regular entries plus 2,427 optimality entries, and the robustness file contains 420 entries.

@TreeTXT
TreeTXT marked this pull request as ready for review August 19, 2026 07:03
@TreeTXT

TreeTXT commented Aug 19, 2026

Copy link
Copy Markdown
Author

/evaluate

@TreeTXT

TreeTXT commented Aug 19, 2026

Copy link
Copy Markdown
Author

/evaluate

@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: TestRouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7030
Accuracy 70.62%
Total Cost $1.237440
Avg Cost per Query $0.000147
Avg Cost per 1K Queries $0.1473
Number of Queries 8400
Abnormal Entries 14
Robustness Score 0.9143

⚠️ 14 of 8400 queries (0.2%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.7590
Opt.Cost (Cost Efficiency) 0.6406
Opt.Acc (Accuracy vs Optimal) 0.8278

Evaluation completed by RouterArena automated workflow

@TreeTXT TreeTXT closed this Aug 19, 2026
@github-actions

Copy link
Copy Markdown

Router Evaluation Results

Router: TestRouter
Dataset Split: full

RouterArena Metrics

Metric Value
RouterArena Score 0.7030
Accuracy 70.62%
Total Cost $1.237440
Avg Cost per Query $0.000147
Avg Cost per 1K Queries $0.1473
Number of Queries 8400
Abnormal Entries 14
Robustness Score 0.9143

⚠️ 14 of 8400 queries (0.2%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

Optimality Metrics

Metric Value
Opt.Sel (Optimal Selection) 0.7590
Opt.Cost (Cost Efficiency) 0.6406
Opt.Acc (Accuracy vs Optimal) 0.8278

Evaluation completed by RouterArena automated workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant