KorvaJoin the waitlist
Tech

Google's Frontier Models Product Head Says AI Models Can Surprise Their Makers

Google's Frontier Models product chief says the job is less about controlling training than about spotting unexpected strengths.

AI image
Lenny's Podcast: Product | Career | Growth · What it actually takes to build frontier models | Tulsee Doshi (Head of Product for Gemini)

Tulsee Doshi, who leads product for Google's frontier-model efforts including Gemini, was asked what people misunderstand about how frontier models are made. She said customers and even internal users often expect training to work like traditional product development, with set goals the team can make sure the model hits.

Her answer was an image from Harry Potter. Snape is the potions teacher in the books. Building a model, she said, is "kind of like Snape's potions class," where teams choose the ingredients and should be thoughtful about what they expect.

By intentions, she means the usual product work. Teams set the capabilities they want, choose training data and methods, and name the scores they want on evals, the tests used to grade models.

But Doshi argues the result is not fully predictable. Teams mix their chosen ingredients, she said, but they also have to expect "a little bit of magic" that brings surprising and unique strengths.

That does not mean training is guesswork. She said researchers monitor training runs closely, track metrics as they climb, and intervene when needed, making development "as much of a science as possible."

Her point is that a product manager's job is "not just to try to shape what goes into the model, but it is also to look at what comes out the other side." That means identifying where the model shines in unexpected ways, then shaping that strength into something useful for users.

Doing that requires learning what Doshi calls the "vibes of the model." A test score may show that a capability improved, but product work means seeing how it shows up in a product and how the model communicates, including its behavior and persona.

She said her team keeps learning as it evaluates and tests. They always learn something new "about what the model can do."

Questions people ask

What is an eval?

An eval is a test that grades how well a model performs a specific task, such as solving math problems, writing clearly, or following instructions. Teams use scores to compare models, track progress during training, and decide where more work is needed.

What does learning the vibes of the model mean in practice?

It means studying how the model behaves in real use, beyond its test scores. That includes how clearly it communicates, how it handles follow-up questions, and where a strength shows up most usefully inside a product.

What are training ingredients?

Training ingredients are the data, examples, and training choices researchers use to encourage particular skills. Teams select and adjust that mix with clear goals in mind, though Doshi says the outcome can still surprise them.

More in Tech