To Challenge Google, a Startup Started Without an Index
Instead of paying to map the web upfront, the startup let its bot wander the live internet for minutes per question.
Training Data · Parallel’s Parag Agrawal: Building a New Web for AI AgentsParallel's first product was a search agent that searched the live web after each question arrived. The company did not yet have an index built ahead of time.
That was the starting bet of Parag Agrawal, the former chief executive of Twitter now running Parallel Web Systems, a startup trying to build search for artificial intelligence agents. Three years ago, when he was starting the company, the obstacle was obvious. Many assumed only giants could pay to copy and organize the whole web before serving a first customer.
Agrawal's workaround was to not build it upfront. Parallel figured out how to build a larger and more sophisticated index incrementally, paying for it over time as it solved problems for more customers. The trick was to start with time instead of storage, using the patience of early agent work to go find answers live.
Agrawal describes an index as a latency optimization, a shortcut that delivers an answer instantly because the work was done earlier. Parallel gave up speed at first, and traded it for live computing power. For deep research, a user will wait a minute, and some of Parallel's products did 10 minutes of research. Those minutes gave the software time to visit many pages, provided it could guess which corners of the web to prioritize. In his telling, the company swapped advance preparation for extra computing at the moment of the answer, letting the agent spend computer brainpower at the moment of the question to make up for not yet having everything stored.
That choice also changed the competition. On day zero, Parallel was not trying to beat a search engine. It was trying to beat people.
"We said, it seems like humans sitting on search engines are way easier to compete with than a search engine on day zero," Agrawal said. The alternative for customers was paying outsourced workers to scour the web and curate data by hand, and a bot that could visit many pages during a wait of minutes could beat that.
The first customers came with that kind of grunt work. Insurance companies used the agents for underwriting and claims processing. Sales teams used them to enrich lists of leads. Finance teams, which had sent data collection overnight to humans who curated it for models, started sending it to Parallel instead.
Parallel's position, not an outside finding, is that "human click data is a bug." By click data, it means records of which search results people clicked. The company also says "an agent doing work with search should rely on agent feedback, not human feedback." It argues models are good at compressing information.
Each job gave Parallel real examples of what agents actually needed from search, empirical cases instead of theories. Those customer jobs supplied the test cases it used to work out how to design search for agents.
Questions people ask
What is a search index, exactly?
An index is a giant map and copy of the web, stored in advance so a search engine can answer instantly. Google constantly crawls, meaning it visits billions of pages, saves the text, and organizes it by words, links, and freshness. Without that upfront work, every question would require going out and reading the internet from scratch.
How can a bot crawl the web after I ask a question?
Crawling means sending software to visit web pages, follow their links, and download what they find. At query time, Parallel's early agent would take a question, guess which sites were most promising, and quickly crawl them while the user waited. It works when users tolerate a minute or more of waiting, as they do for deep research, but not for instant answers.
Why is building a web index so expensive?
The web has tens of billions of pages that change constantly, so keeping a fresh copy requires thousands of computers, huge storage, and constant bandwidth. A startup would have to spend heavily for months or years before answering a single customer query well. Parallel avoided that by spending computer power per question instead of building the whole archive first.