The same AI model scores differently depending on who runs it
Atallah said the results also change over time, and described the models as non-deterministic.
20VC with Harry Stebbings · OpenRouter CEO: Why Chinese Open Models Are Beating the US | Why Enterprises Fear OpenAI & AnthropicThe same openly shared AI model, whose trained files are publicly released so any company can run a copy, can post very different test scores depending on which company runs it.
Small setup choices can move a score. Providers can use different chips, run compressed versions of the model, set different randomness controls, or add hidden instructions.
Alex Atallah, the co-founder and chief executive of OpenRouter, a marketplace that sends customers to companies that run such models, said his company tests the same models across different hosts and keeps finding different results on fixed, well known tests. He said the results change over time, and called the models "very emotional," his phrase for non-deterministic behavior.
Chinese company Moonshot recently published its own comparison showing its Kimiko 3 model getting different scores on the same test from different providers.
Questions people ask
What does OpenRouter do?
OpenRouter is a marketplace for AI models. Customers send their requests to OpenRouter, and it forwards them to outside companies that run the models, picking the provider based on price, speed, and availability.
What is an AI benchmark?
Benchmarks are standardized tests used to compare AI models on skills like math, coding, or factual recall. They are useful for rough comparisons, but the score can depend on the setup, so a single number does not tell the whole story.