Feature Request: more nuanced score for models

[paoloricciuti]

Right now, the models are scored solely based on the amount of test that they pass...a more nuanced score that also involves how the tests are passing would be wonderful. This could involve:

• Whether the model is using the MCP server or not
• Whether the model is using the Test tool or not
• The amount of step it took to complete
• The number of tokens it took to complete
• Possibly cost (?)

Other ideas?

[khromov]

For MCP vs non-MCP, I think it's largely orthogonal to the percentage-based scores (because having MCP basically tests how good the model is at using it), so I think the best approach is to run both a MCP and non-MCP version of the same model and have a filter toggle to see results with/without MCP with two "leaderboards".

For test tool, perhaps we just bundle it into the MCP results?

Steps/Tokens/Cost could be sortable columns on the index pages.

PS. I added a very basic temporary index page for testing: https://sveltejs.github.io/ai/

[paoloricciuti]

Yeah obviously we should run both with and without MCP but you are right, it could be just a columns in the table. I would still show them both: Claude 4.5 is definitely better than another model if he can pass the same amount of tests without the MCP as GPT-5 with the MCP

[jesperordrup]

Number of tokens to complete is useful