Korva is a scroll of the internet’s best ideas.
A professor said deep learning fails
The professor's warning
The freshman was Sam Altman, working in an AI lab the summer between his first and second years. Nothing in the lab was working. The professor listed all kinds of directions to try, and deep learning was the one exception.
A detour into startups
He believed the professor and pursued other things. By around 2005 it was clear to him that AI was not working, and he ended up in startups by accident and fell in love with them. He had loved AI his whole life, as a nerdy kid who spent Friday nights at his computer and reading science fiction.
Backlash at OpenAI
When OpenAI began in 2015 with the aim of building AGI, the intellectual giants of the field hammered it for the idea. They hammered it again when the company focused on large language models.
Why was deep learning dismissed for so long?
Neural networks fell out of favor after Marvin Minsky and Seymour Papert's 1969 book Perceptrons showed the limits of single-layer networks. Funding dried up in the AI winters of the 1970s and late 1980s. Training large networks also needed far more data and computing power than labs had. Even so, Yann LeCun's convolutional networks were reading handwritten zip codes for banks and the postal service by the 1990s.
How did backpropagation survive the doubters?
Backpropagation is the method that adjusts a network's connections by passing errors backward through its layers. David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized it in a 1986 paper in Nature. A small group of researchers, including Hinton in Toronto and Yoshua Bengio in Montreal, kept working on it through the years when most of the field had moved on. Their persistence is part of why Hinton, Bengio, and LeCun later shared the 2018 Turing Award.
What finally made deep learning work?
The turning point came in 2012 at the ImageNet image recognition contest. A network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, cut the error rate to about 15 percent when the runner-up was near 26 percent. It was trained on graphics processors, which handle the huge number of parallel calculations that neural networks need. Large labeled datasets like ImageNet supplied the examples that earlier decades lacked.
More from David Senra
