The Transformer Was Built From Spare Parts
Its pieces were already in the literature, and Patel argues AI systems have been unusually good at that kind of remixing in mathematics.

Dwarkesh Podcast · The avalanche that hit math is coming for AI developmentDwarkesh Patel, a podcast host who interviews AI researchers, argues that AI labs care less about grand theory than about recipes that reliably make training more efficient and models smarter.
In mathematics, inventing ideas like topology or group theory matters far more than settling a narrow puzzle. AI labs put things in the opposite order.
Deep learning has advanced for a decade and a half without much new theory, so grinding through promising combinations can be especially useful for machine learning, unlike in mathematics.
His example is the Transformer, the 2017 architecture behind modern chatbots. Before it appeared, researchers already knew attention as a way to train in parallel and carry information across longer passages.
Chris Olah, an AI researcher active at the time, has said, "At that time, my sense is that the transformer architecture was a natural combination of many ideas like attention and residual networks, which had independently been effective, taken to quite aggressive lengths."
His friend Sholto Douglas, also an AI researcher, has a half-finished essay arguing the Transformer could have been built two years earlier. No one knew the right combination of existing ideas until later.
The design pulled ahead only after half a dozen additional tweaks were connected in just the right way. Nearly every one had already appeared in published papers for years.
Finding that package took research taste, but it also took tons of small experiments. That persistent, intelligent matching of dozens of combinations is the kind of work current AI systems have shown they are disproportionately skilled at in their math contributions.
Questions people ask
What is attention in this context?
Attention lets a model weigh different words in a passage when it processes each new word, so it can keep track of distant context. Researchers liked it because it could be run in parallel across many computers, which made training faster.
What is a residual network?
They are shortcuts that let information skip ahead one or more layers in a neural network. The shortcuts make it easier to train much deeper networks, because the training signal does not fade as quickly.
Why does relentless tweaking matter so much in AI research?
Many AI advances come from finding settings and structures that train reliably at large scale, not from proving theorems. A small change can decide whether a model learns well, so trying many combinations and measuring the result often beats waiting for a deeper theory.
