Korva is a scroll of the internet’s best ideas.
Training crawls skip the slow pages
What labs skip
A model that was already trained on a page gains nothing from crawling it again, so a fresh web index has little use for a training crawl. Labs also aren't patient enough to wait for slow, random JavaScript to load. The tokens gained per unit of compute on those pages is one to two orders of magnitude worse, and a lab is happy to take some trillions of tokens without them.
Why completionists differ
A search company that is a completionist about the web sees it differently, because the slow pages are the ones others skipped. That difference is why a search index built for agents complements a model rather than copying what the labs already have.
Build or buy
Most LLM work will want web access and search infrastructure tuned for agents: for nine out of ten uses, buyers of LLMs will want it. Labs can build that or buy it, and partnerships are starting to appear.
Why is JavaScript-heavy content harder to crawl?
A page built with JavaScript often arrives as an almost empty shell, and its text appears only after a browser runs the code. A crawler has to render the page the way a browser does, which takes far more computing time than reading plain HTML. Google has described crawling and rendering as separate steps, with rendering sometimes happening later. Pages that take longer to load take longer still to render.
What does a training crawl keep?
Training sets are usually built by filtering a huge crawl down to cleaner text. The 2020 GPT-3 paper by Tom Brown and colleagues at OpenAI described a filtered version of Common Crawl that made up 60 percent of the weighted training mix. Filtering removes duplicates and low quality text, which also removes many odd or slow pages. The goal is high-value tokens per unit of compute.
Why does freshness matter in search?
Google has described a concept it called query deserves freshness, in which some searches need the newest pages. A search about a breaking event is useless if the index only holds last month's pages. Fresh content needs frequent recrawling, and that costs far more than crawling once. A static training set avoids this cost because it is built once.
More from Training Data
