Controlled measurement matters
It’s hard to measure a baby’s height. There’s no point in asking the baby to stand up nice and straight, is there? People who do this all the time use the right tools to measure, and the right technique: stretch the kid out and hold it still.
If you want to do any meaningful analysis in business, you need dependable measurements to work with. You can’t grab the entire economic system and hold it still, of course. Yet there are many opportunities for controlled measurement in real business applications. You can’t control the whole society, but you may be able to control your own actions, and those of your coworkers. You can also control the function of inanimate things like equipment and websites.
Let’s say you have been using the same coupons since the beginning of time, and you’d like to try something new. If you replace the old coupon with a new one today and watch to see what happens, you’re going to get lousy information. Why? Because the world is a squirming baby! Things are changing, and whether the response to your new coupon is fabulous, crummy or so-so, you won’t really know what’s going on. It won’t be the response to a coupon, but the response to a mix of everything affecting your business at that moment, plus a new coupon.
What you need is a controlled test. You need a group of people to get the old coupon – exactly as it has always been. And you need another group to get the new coupon. In order to separate the effect of the coupon change from all the other excitement in the world, you need to match the two groups in every way you can – deliver the coupons at the same time, in the same way, avoiding any kind of systematic bias in who gets which. In other words, don’t send one version to men and the other to women, don’t form the groups by geography, or age, or anything else that might affect results. You want random samples, and the definition of random is that every single person has an equal chance of getting into either one of the two samples.
That way, at least for that moment with its unique economic conditions, you’ll know which coupon worked better.
[There’s a lot more that could be said about analysis methods and continued testing that are relevant to this issue. I’ll write more about those topics down the road.]
Date: February 16, 2012
