Korva is a scroll of the internet’s best ideas.
Code versus decks
Code can be run to see whether it works, so there is a way to validate it from the output. Knowledge work is different. The final deck can't simply be believed, and what matters is the process, the inputs, and the reasoning along the way.
Showing the work
That shapes the product. ChatGPT has to act as a collaborator that shows all the in-progress work, its citations, and its inputs, so the person goes on the journey with the model and knows at the end that the result is right, good, and useful. Open questions remain: whether more citations should appear in the chain of thought, and whether a thread, which suits coding so well, is the right surface for seeing all of that in knowledge work.
Human collaborators
Bringing in human collaborators raises the same question from another side, which is how the model becomes more of a collaborator as people get things done together.
Why can software be tested automatically?
Software tests are small programs that run code and compare the output to an expected result, ending in pass or fail. Kent Beck popularized writing them first in test-driven development, part of the Extreme Programming approach he developed in the late 1990s. Because the check runs in seconds, a developer can run thousands of them after every change. Beck also created the JUnit testing framework with Erich Gamma.
Why does showing reasoning steps help?
A 2022 paper by Jason Wei and colleagues at Google, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, found that prompting a model to write out intermediate steps improved its results on math word problems. Written steps also give a reader something to inspect, since a wrong turn in the middle can be spotted even when the final answer looks plausible. A 2023 OpenAI paper, Let's Verify Step by Step, found that rewarding each correct step beat rewarding only the final answer on a hard math benchmark.
More from Lenny's Podcast: Product | Growth | Career
