Korva is a scroll of the internet’s best ideas.
Nobody can read a neural network
Dots and Lines
A neural network is drawn as circles joined by lines, borrowed from the neurons in a brain. Picture dots on a plane, each one blue or red, and a network that takes a dot's x and y coordinates and answers with its color. With only two training dots, the best guess is a straight line. Each new dot lets the line tilt and refine. Mathematically that is a value for a, b, and c in the expression ax plus by plus c: if the result is greater than 0, the dot is blue, and otherwise red.
Billions of Numbers
Real networks do the same thing with millions, if not billions, of numbers, all set by throwing lots and lots of data at them. The computer works out what every connection should be, aiming at the right answer with high confidence. No one can say that one edge holds its value for one particular reason.
Other Uses
The approach carries over to other fields. Feed a network humidity levels, pressure values, and historical rainfall, and it can learn to predict rain. Feed it monthly ad spending and sales history, and it can predict sales: not 100 percent correct, but confidently correct most of the time.
How does a network find its own numbers?
Training uses an algorithm called gradient descent. The network makes a guess, a formula measures how wrong it was, and each number is nudged slightly in the direction that reduces the error. Backpropagation is the method that works out those nudges efficiently across every layer. David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized it in a 1986 paper.
Why did neural networks take off after 2012?
In 2012, a network called AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the ImageNet image-recognition contest. It reached a top-five error rate of about 15 percent, while the runner-up had about 26 percent. It was trained on graphics processors, whose parallel arithmetic suited the huge number of multiplications involved. That result set off the wave of deep learning.
How old is the line-drawing idea?
Psychologist Frank Rosenblatt built the perceptron in 1958, a machine that learned to separate two kinds of inputs with a line, using weights adjusted after each mistake. It followed a 1943 model of the neuron by Warren McCulloch and Walter Pitts. Marvin Minsky and Seymour Papert's 1969 book showed that a single perceptron cannot handle some simple patterns. Networks with several layers later overcame that limit.
More from CS50
