Korva is a scroll of the internet’s best ideas.
Search agents first, index later
Agent before engine
Deep research tolerates a wait of a minute, and some products run ten minutes of research. That is plenty of time to crawl a lot of pages, as long as there is enough of a map to know which to prioritize. So the first product was a search agent rather than a search engine, and its competition was outsourced humans curating data, which is far easier to beat than an established engine on day zero.
The first customers
The early customers were doing insurance underwriting and claims processing, sales data enrichment, and finance work where a person used to send data collection overnight to a team of humans before building a model. Each of those jobs replaced outsourced human work on web data.
Work that built the index
That work also paid for the index. Every customer's job let the index grow incrementally, with no need to spend heavily on infrastructure before serving anyone. It also produced real use cases to run evaluations on, so search for agents could be shaped by empirical cases instead of theoretical ones. The approach trades crawling for inference-time compute.
Why is an index a latency optimization?
Without an index, answering a query would mean reading every document to see whether it matches. A search index flips the arrangement: for each word it stores the list of documents that contain it. Finding matches then becomes a lookup instead of a scan. This structure, called an inverted index, has been the core of text search for decades.
How does a crawler decide what to fetch first?
Crawlers rank their to-do list using signals about which pages matter most. Larry Page and Sergey Brin described PageRank in 1998, which treats a link from one page to another as a vote of importance. Pages with many votes from important pages get fetched sooner and revisited more often. Pages that change often, such as news sites, are scheduled for frequent returns.
How large are open web crawls?
The nonprofit Common Crawl has published monthly snapshots of the web since 2008, and each snapshot holds billions of pages. Its data has been widely used to train language models. A commercial search index has to go beyond that snapshot approach by keeping pages continuously updated. Growing the index one customer job at a time follows that pattern on a smaller scale.
More from Training Data
