High AI Test Scores Do Not Mean a Smarter Model
Labs keep better internal tests private while facing pressure to chase public scores.

The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch · 20VC: Jev Creator on Why Data Is More Important Than Compute | Will US Open-Source Models Catch Chinese | The Cost of Intelligence and Tokens: What No One Sees But Needs to Know with Diogo AlmeidaA high score on a public AI test does not mean the model will work better for you.
That is the view of Diogo Almeida, founder at TypeSafe, which makes Jev, which is built to make reliable decisions in software. Public benchmarks, the standardized tests used to rank models, are a poor proxy for intelligence and can mislead people who lack time to test models themselves.
Labs hold better internal ways to measure intelligence, but those measures stay private. That leaves the public with noisy scores and little reward for publishing a more honest test.
Once a test is public, it invites benchmark maxing, tuning a model and its setup to lift a known score without making the system more robust or useful in real work.
The incentive to do that is uneven. Top labs that already rank near the top can afford to be mostly truthful and protect morale and real standing, while second tier contenders face more pressure to convince themselves they are catching up.
The example given is Meta's Llama 4, described as unusually overt about chasing scores, and the race to be called the American open source leader, described as a side prize with heavy optics because few people use those models enough to challenge the claims.
The same split applies to Anthropic versus Chinese rivals. Almeida put it this way: "Anthropic models are just the best," even when Chinese models match them on benchmarks, which suggests score chasing and dependable performance are different skills.
Tests for decision making are even weaker because no one understands what good performance is. That leaves more room to tune for the test while missing what matters in real work.
Coding shows what does change real performance. The biggest jump came when coding material became a larger share of training, with later gains looking more incremental. He guesses narrow models could get cheaper and more specialized, and he does not know how good they are.
Questions people ask
What does it mean to game a benchmark?
Benchmark maxing means preparing specifically for a known test, through extra training on similar questions, careful prompt tweaks, or repeated tries, until the score rises. The number improves, but the model may be no more reliable on new problems a customer actually has.
Why is there no trusted scorekeeper for AI models?
There is no single independent auditor most buyers trust, and designing a good test is costly and quickly goes stale once labs train to it. An internal test that guides research can also reveal product plans, so labs prefer to keep their most informative evaluations private.
How should I pick a model if I should ignore leaderboards?
Try the models on samples of your own work, especially the boring or difficult cases where mistakes are costly. Track how often the answer is right without fixes, how consistent it is across tries, and whether failures are easy to catch.
