Korva is a scroll of the internet’s best ideas.
Tangled dots separate in 3D
Drawing the widest line
A support vector machine sorts blue dots from red dots by drawing the line that sits as far as possible from the nearest dots of each color, as close to the middle as it can get. That works while the two colors sit on their own sides of the page. When they are mixed together, no straight line in the plane can divide them.
Adding a third axis
The fix is a third axis. Each dot keeps its X and Y, and its Z becomes its Euclidean distance from the origin. The red dots land low and the blue dots land somewhat higher, so a flat plane at right angles to the Z axis cuts cleanly between them.
Skipping the embedding
The price is that the plane now has to be found in three dimensions, and real data can need far more. Computing that embedding directly becomes a bottleneck that doesn't scale, especially when the data is already high dimensional. The kernel trick sidesteps it: instead of working out the plane precisely, calculate a function K that captures the properties of the plane separating the two classes.
Why does the widest gap matter?
Vapnik's statistical learning theory links a wider margin to a lower bound on how badly a classifier can do on new data. Only the dots nearest the boundary, called support vectors, fix where the line sits. Move any other dot and the line stays exactly where it was. This is why the method can work well even with few training examples.
How did the kernel trick get its start?
Aizerman, Braverman, and Rozonoer described the core idea in 1964, using kernels in a method for pattern recognition. Boser, Guyon, and Vapnik combined it with maximum-margin classifiers in a 1992 paper. That combination turned the support vector machine into a method that could draw curved boundaries. The trick works because training only ever needs the inner products between pairs of points.
How can a kernel reach infinite dimensions?
The radial basis function kernel scores two points by how close they are, with the score falling off smoothly as the distance grows. Its implied feature space has infinitely many dimensions, yet each score takes only a subtraction, a square, and an exponential. No coordinates in that space are ever written down. The optimizer sees only a table with one kernel value for every pair of training points.
More from Computerphile
