Korva is a scroll of the internet’s best ideas.
ChatGPT answers by the odds
Learning From the Internet
The inputs to these networks have been all of the content of the internet: Google search results, Reddit, Stack Overflow, dictionaries, encyclopedias, and any other such works. From the patterns and frequencies of text across all of it, the model learns what usually follows what.
Why Answers Go Wrong
Ask it how are you, and it likely replies good, thanks, how are you, in something close to 99 percent of cases. That is the logic behind every answer: probable, not guaranteed. Misinformation on the internet gets into the mix, so some answers come out wrong.
Navigating the network to an answer means finding the best path through it. Sometimes the path leads to something made up, and the model delivers it very confidently. Those are known as hallucinations, and they happen even with CS50's own duck.
How does a model choose its next word?
The model assigns a probability to every possible next piece of text, called a token, and then one is picked. A setting controls whether it always takes the most likely candidate or samples from the top few. Sampling is why the same question can produce different replies on different tries. Each chosen token is added to the text, and the process repeats until the reply is complete.
Why do language models state false things confidently?
The training goal is to predict plausible text, and nothing in that goal checks a claim against the world. A fluent sentence and a true sentence can look alike to the model. A 2025 paper from OpenAI researchers argued that common benchmarks also reward guessing over admitting uncertainty, which encourages confident answers. Reducing the problem is an active research area.
How much text do these models train on?
In OpenAI's 2020 paper describing GPT-3, the model was trained on roughly 300 billion tokens. Most of that came from filtered web crawls, plus books and Wikipedia. A token is typically a short word or a piece of a longer one, about three quarters of an English word on average. Later models are reported to have used far more.
More from CS50
