Big data and the number crunchers - are you a victim?

Ever get the feeling you were being watched? It’s not just a feeling, says columnist Jeff Cole, as he looks at the increasing influence of big data and the number crunchers.

Here’s a bold statement: Every day you are the victim of a designed experiment. Sounds like words coming from a person wearing a tin-foil hat and living under a bridge, right?

Living in a very similar intersection is a specialized breed of researchers and marketers. While the Big "Y" for your Six Sigma project may be reducing cycle time or improving yield, their Big "Y" is Profit. I’m referring to Big Data, Designed Experiments and the surprising impact they have on us and their ability to change our behavior without us even knowing.

A Case for Slower Cycle Times?

You may be working day and night trying to reduce the cycle time of the processes where you work. But can you imagine a project intended to increase cycle time? Intentionally change behavior to make people move slower?

Such is the case in supermarkets.

Decades ago they discovered a key fundamental– the longer they keep you in the store the more you buy. That’s why, in a typical supermarket, milk is in the back of the store on one side and bread is on the other. They know the exact height of your child in the cart so all the colorful, high-sugar treats are screaming at them from their height. Nothing is random in a supermarket. They experiment with just about every X-factor you can imagine – store temperature, lighting, colors of the wall, placement etc. The closer you get to the register, the smaller the items become (and the higher margin too) for those "impulse" purchases.

Most people simply wander through the market without any inkling of a notion that statistically designed experiments and terabytes of purchasing data have been very carefully analyzed and acted on in order to subtly change and direct their behavior. The only thing a civilian may notice is the odd coupons that spit out at the register when they check out. If the system’s algorithms detect that you bought baby food and bibs, you may get a coupon for diapers.

Ever sign up for a retail "loyalty" card? You’ve made their job even easier as they can now analyze your longer-term purchasing patterns – even when you pay in cash.

This is not often talked about, but one large supermarket chain went so far as to construct an experimental design center in an unmarked warehouse (and they aren’t alone). It was a fully-functional, fully stocked market with one exception – it wasn’t open to the public. Oh yeah – it was also totally wired – sensors on the shelves, cameras, the whole nine yards. You may think this is fantasy, pure rubbish, or not widely practiced.

In response, I offer this example discussed in a 2011 issue of Time Magazine. In an experiment described by Martin Lindstrom, a leading supermarket consultant, a "zone of seduction" was set up to catch shoppers.

Markets know exactly how fast you push your carts down certain aisles. One experiment changed the flooring in one area. Instead of smooth linoleum, they put in a rougher flooring material that caused the cart to make "clickety-clack" sounds. Instinctively, the shoppers slowed down and spent an additional 45 seconds in that area, increasing their spend by up to 73%! While they were unwittingly mired in this "zone of seduction" they were also subjected to a secondary experiment involving the wording on a sign above a stack of soup cans. That experiment was also mildly successful – it netted a 7-fold increase in soup purchases. (For several interesting and short videos talking about additional supermarket tactics, click here or here.)

I’m not going overboard in saying that on a daily basis you and I are on the receiving end of designed experiments and big data. Every major political speech is vetted through focus groups with people in high-tech marketing research centers who turn the dials up and down as they hear things they like or don’t like.

The politician’s Y variable? Your vote. In the U.S., the Obama campaign "Moneyballed" their way to victory – headed by Dan Wagner, the campaign chief analytics officer, they employed a number crunching team 5 times the size of their 2008 campaign. Many pundits cited that as a template for campaigns to come.

Ever watch TV? In pilot season, producers have a choice – roll their budget into one version of a pilot, or experiment – try different theme songs, casts, etc. Focus groups privately view the pilots and vote for the best – it’s been going on for decades. This clip of Gilligan’s Island for instance is a version of the pilot with a different cast. There is a whole sub-culture devoted to finding unaired pilot versions of your favorite shows.

Why experiment though? Money, of course. If they could tweak the show to make more people watch, they could charge more for advertising. High viewership = higher profits. Same goes with catalogs you receive at home, print ads, retail stores, commercials, and let’s not get started on the web. Every keystroke you make on your favorite web sites are captured, cross-referenced, (sometimes sold) and compiled into neat operational statistics with names like web page "stickiness".

Has your behavior changed? Big Data and Designed Experiments do not equal Big Brother. But they are out there and are at least a distant cousin. Something to consider next time you reach for that can of soup in Aisle 7…

That said, rest assured that this month’s column is totally experiment-free and devoid of big-data usage. None of your information was collected or analyzed to drive these words!

[inlinead]