After Its AI Failed on Real Code, a Startup Flew Everyone to Brazil

The whole company got on the plane to Brazil.

Scott Wu, co-founder of the startup Cognition, which makes an AI coding assistant called Devin, sat in Brazil with engineers from Nubank, the largest Brazilian bank by stock market value then. His entire team was there with him, watching how the bank's programmers handled one update after another, then digging into the same problems themselves to figure out what Devin would need to know.

It was an all hands effort to make the first deal work. Months earlier, Cognition had gone viral with a demo of an AI that could work as a software engineer. The launch in March 2024 brought fame, and requests for the product, but no customers and no revenue. When people tried it on real company code, the pilots failed.

Wu said those pilots in April and May 2024 could handle toy demos and small projects, but were certainly not ready for real codebases. He said they were making agents with GPT-4, in what he called a very different and much more primitive era, and the pilots were failing, which he described as natural.

So the team stopped asking what the AI could do in theory and asked what kind of work was simple enough for it to do end to end. The answer was deliberately boring. Wu said it had to be repetitive, tightly scoped, and easy enough to check quickly that an agent could diagnose and fix the problem, but still tricky enough that a fixed script could not do it alone. It needed some intelligence, and some wandering around, and it sat between fully automatable chores and deep architecture problems.

That pointed to migrations and version upgrades, the grunt work of keeping old software current. In a large company, Wu said, that can mean a codebase of 50,000 files where programmers must make the same eight changes in each file, as when moving from one version of the Java programming language to the next. Each change takes a little thought about tradeoffs, but the task is very repetitive.

Nubank had one of those big migrations. Cognition built a custom version of Devin that was extremely optimized for that single job.

Wu said they did not deploy a team, they deployed the whole company. They went through candidate tasks one by one, tried some manually to understand the work, then taught Devin how to read the right files and handle the bank's exact problems. It was, in effect, building the product for one company, and Wu said that first success shaped how they picked later jobs.

That work with enterprise code was very different from building a small demo from scratch, Wu said. The team leaned into real problems inside massive products relied on by millions daily, starting with the mundane, repetitive jobs an agent could finish.