The Same AI Chip's Output Can Vary About 100-Fold Depending on Software
As chips get more specialized, performance depends less on silicon and more on the translator that tells it what to do.

Standard Capital · Luminal Raises $25M Series A to Break AI's Hardware Bottleneck, Led by Standard CapitalBuy a top Nvidia AI chip and run a model on it with weak software, and you might get about 10 tokens a second while most of the chip sits idle. With much better software, that same chip, drawing the same power and costing the same money, might produce about 1,000 tokens a second in total across many users sharing it.
A token is a small chunk of text, often part of a word, so the rate is a measure of total output. The comparison is a rough example, not a promise that one person will get answers 100 times faster.
That gap comes down to a compiler, says Joe, a co-founder of the startup Luminal, which builds that kind of software. The compiler decides how fully the expensive hardware is used.
The reason starts with a broken trend. Moore's Law once promised that transistors would get smaller and more power efficient every few years, letting the same chip do steadily more work.
When that progress slowed, chipmakers kept improving performance by building more specialized hardware. A general computer processor gave way to the graphics chip, originally built for video games and now widely used for AI, and then to even more specialized circuits built mainly to answer AI queries.
More specialization means more of the silicon is devoted to raw math, and more of the complexity shifts to software. The compiler must handle scheduling and memory handling, including fetching data before the math needs it, chores that decide how much of the chip actually stays busy.
A compiler is a translator. Researchers write models in a friendly, readable system called PyTorch, which makes math simple to express in the Python programming language, and the compiler turns that high level code into precise, device specific instructions the chip can execute.
There are many ways to do that translation, and they use hardware very differently. A weak version leaves most of the chip waiting, while a strong one keeps math units fed with data at the right moment.
The job is getting harder. Joe says Nvidia's newest Blackwell chip is substantially harder to program well than its prior Hopper generation, because it is more specialized.
It gets harder still with highly specialized chips such as Google's tensor processing unit, an AI focused circuit known as a TPU. Programming one well is extremely difficult, and Google's internal tools for doing it are generally not available to outsiders.
That translation gap helps explain Nvidia's lead. The company invested years ago in making sure readable research code would run reliably on its hardware, while rival chips often lack equally polished software.
The consequence is large because the hardware is already paid for. Hundreds of billions of dollars in chips and data centers could, in principle, produce far more answers from the same machines if the software that programs them improves.
Questions people ask
What is AI inference?
Inference is the work an AI model does after training, when it answers a new query. It involves large amounts of repeated math, often matrix multiplication, and it runs on expensive hardware that uses substantial power.
If software matters so much, why do companies keep buying more chips?
Partly because demand for AI answers has been strong enough that even inefficient operation can pay. Better software would still help, because getting more answers from each chip could reduce how many chips and data centers are needed.

