Judge AI With Five Prompts You Care About
Gemini product chief Tulsee Doshi says start with five prompts you care about, then grow to about 20 to separate models.

Lenny's Podcast: Product | Career | Growth · What it actually takes to build frontier models | Tulsee Doshi (Head of Product for Gemini)Tulsee Doshi, head of product for Google's Gemini models, says you do not need a massive benchmark to judge artificial intelligence. Start with just five prompts.
The trick is to pick tasks you genuinely care about, where models already show signs of life but still fall short. That gap, in her view, is where you can see differences between models.
For some people that means visual 3D animation and questions of aesthetic style, used as a visual check on model behavior. For others it is creative writing, essays, synthesizing information, or hard financial analysis.
Run your prompts, look at the answers yourself, and write down what good means. That step is surprisingly hard, especially for long, multistep interactions, but collecting examples of good gives you clarity about what you are aiming for.
Then grow the set to about 20 prompts that stress a particular capability. Doshi says even that mini evaluation can be powerful enough to show signal between one model and another, without requiring a large team or complex setup.
The first step is small. Pick one job you want AI to do well, write five versions of it, and judge the results by your own standard before you try to scale anything.
Questions people ask
How do I choose my first five prompts?
Pick one repeated job where you have strong opinions about quality, such as drafting a memo, summarizing research, or making a chart. Write five variations that differ in difficulty, length, or constraints, so you are testing range and not repeating the same request.
What does it mean to define what good looks like?
Look at two or three outputs side by side and note what you liked, what was wrong, and what you would have accepted. Turn those notes into a short checklist, such as accurate numbers, clear structure, and a specific tone, and reuse it each time you judge.
How should I compare two models fairly?
Use the same prompts, in the same order, with fresh sessions and similar settings when possible. Score each answer against your checklist rather than from memory, so you are comparing the models and not your changing expectations.

