Korva is a scroll of the internet’s best ideas.
Phone plus post got 7%
The test came from a campaign in the early 1990s that offered an extra couple of pounds a month for network services on a home phone line, such as call diversion and call waiting. The letters invited people to send back a coupon in a postage-paid envelope or call a number and quote a code. The client, for reasons nobody ever learned, wanted to drop the postal option.
Rather than gamble on removing it, the agency sent three versions of the letter to 50,000 people each, chosen completely at random: post only, phone only, and a choice of both. The response rates were 5%, 2%, and close to the sum of the two, about 7%.
To an economist that result is baffling, since it suggests the biggest factor in whether someone bought was how they could order, ahead of what the product was or what it cost. It also suggests that some brilliant businesses are almost certainly failing at a single psychological hurdle, such as people hating to order by phone, while their owners spend millions improving the product.
Why do random splits make a test reliable?
Random assignment gives each group about the same mix of ages, incomes, and attitudes, so a difference in response can be traced to the letter and not to the people who received it. Large groups shrink the role of luck. With 50,000 people in a group, a 5% response rate carries a margin of error of only about plus or minus 0.2 percentage points.
Why would two reply options beat one?
Different people face different hurdles: some dislike speaking to a stranger on the phone, while others find posting a coupon a chore. Offering both removes whichever hurdle each person meets. Richard Thaler and Cass Sunstein, in their 2008 book Nudge, argue that small frictions such as an extra step can change what people do far more than standard economic models predict.
More from My First Million
