IBM Let 5,500 Marketers Run Their Own Tests
By scrapping a costly, one-man approval system, the company went from 97 experiments a year to 2,822 in three years.

library.hbs.edu · Four Ways to Build a Culture of Scientific Experimentation | Working KnowledgeIn 2015, an IBM marketer with an idea for an online test had to petition the information technology department. There, a single specialist decided which tests were worth running, and he rejected many proposals as not strong enough.
The process was rigid and expensive. Tests had to follow a fixed set of steps, and their cost was charged back to the business unit that had asked for them.
For a global company with more than 5,500 marketers, the result was a trickle. IBM ran just 97 tests that year, and experimentation was not a core activity.
The turnaround came from dismantling the gatekeeper system and letting marketers run tests themselves.
The shift started when testing was taken over by IBM's head of marketing analytics, with support from the chief marketing officer. He gave marketers around the world authority to carry out their own tests.
To make that promise practical, he removed cost and complexity. He supplied simple software, ended internal charges for web testing, offered training for everyone, and created a central support team to provide help.
He also set clear rules for running careful tests, so that thousands of newcomers would work in a consistent way and produce results the company could trust.
Then he pushed for speed and visibility. Marketing units were asked to complete 30 web tests within a single month, and after that blitz, quarterly contests rewarded the most creative or most widely useful tests.
He also used the budget to force the habit. IBM tied part of each marketing unit's budget to its experimentation plan, so testing was no longer optional.
Three years later, IBM was running 2,822 tests a year.
Questions people ask
What kind of tests were those marketers running?
IBM's tests were online tests, such as A/B comparisons of web pages, emails, or ads. Different groups of customers saw different versions, and the company measured which version drew more clicks, signups, or purchases.
What is a center of excellence?
It is a small central team of specialists that supports workers in other departments. At IBM, it provided advice on testing tools and methods, so marketers did not have to become statisticians to run a valid test.
Did running thousands more tests actually help IBM's business?
The account cited here reports only the rise in testing volume, not a specific lift in sales or profit. In general, companies run more tests to find more winning ideas and drop losing ones faster, but any single test may fail or show only a small gain.


