Korva is a scroll of the internet’s best ideas.
Massachusetts is 1,536 numbers
Words Far Apart
Take a paragraph that begins with Massachusetts being a state in New England, then moves to a new sentence about its capital. Before about 2017, it was hard for machines to connect Massachusetts to state, because the two words sit far apart, in different sentences. Without giving weight to that relationship, a model pays little attention to either.
How Attention Works
Attention, which Google proposed in 2017, works out dynamically how words relate to each other in a text. A bolder edge between Massachusetts and state marks a real relationship in the training data, while a thin line to a word like is marks little. The words become vectors, each a long list of numbers, and are fed into large neural networks.
From GPT to ChatGPT
OpenAI published its GPT model in 2020, and ChatGPT came out in 2022. GPT stands for generative pre-trained transformer: it generates content, it was pre-trained on lots of publicly available data, and it transforms the user's input into ideally correct output. It was never meant as a branded product.
Why did attention beat older approaches?
Before 2017, most language models read text one word at a time, using designs called recurrent networks. Information from early words faded by the time the model reached later ones. The 2017 Google paper Attention Is All You Need, by Ashish Vaswani and colleagues, introduced the transformer, which compares every word with every other word at once. That also let training run in parallel on many processors, which made much larger models practical.
Why are the lists of numbers so long?
The length of the list is a design choice that trades detail for cost. Longer lists can capture more shades of meaning but need more memory and computing power. OpenAI's text-embedding-ada-002 model, released in December 2022, produces 1,536 numbers for each piece of text. Google's original BERT base model, from 2018, used 768.
How can numbers capture a word's meaning?
Words used in similar contexts end up with similar lists of numbers, and the directions between them carry meaning. In 2013, Tomas Mikolov and colleagues at Google showed with word2vec that taking the vector for king, subtracting man, and adding woman landed close to the vector for queen. The result became a famous demonstration of how vector spaces encode relationships.