“Trust but verify”: a data analytic cautionary tale

Posted:  April 3, 2013

It’s a phenomenon everyone in the research business faces now and then:  you see something surprising, unexpected in your data, something not particularly positive for your client.  You say to yourself, “Oh no, the clients are going to be so disappointed.  This is going to come out of the blue for them.  They aren’t going to be happy.  They’re going to have a hard time believing it.  I’m not looking forward to presenting this.”

So what do we do?  We wait.

Then there’s the opposite kind of finding in your data.  You say to yourself, “That’s it—that’s a huge finding.  Who would have thought?  I can’t wait for our clients to see this.  They’re going to love it.  It will change the way they act on this opportunity.” 

So what do we do?  We wait.  Much as I don’t want to, we wait.

And why do we wait?  It’s not because we don’t trust our data.  I was taught a long time ago, as a graduate student (and that was a long time ago), to trust your data.  If you’ve followed the principles and best practices on sampling and questionnaire design, the data don’t lie.*

So we trust our data, but we don’t trust it blindly.  We “trust but verify,” meaning we test every possible reason why and how our data could be wrong, before we’re comfortable reporting that surprising, unexpected finding.  We never want to fall into the trap the Russian army recruiter fell into in a parable related by John Allen Paulos in his book, “Once Upon a Number:  The Hidden Mathematical Logic of Stories.”

“A recruiter in the Tsar’s army was riding through a small town and noticed dozens of chalked circular targets on the side of a barn, each with a bullet hole through the bull’s eye.  The recruiter was impressed and asked a neighbor who this perfect shooter might be.  The neighbor responded, ‘Oh that’s Shepsel, the shoemaker’s son.  He’s a little peculiar.’  The enthusiastic recruiter was undeterred until the neighbor added, ‘You see, first Shepsel shoots and then he draws the chalk circles around the bullet hole.’”


It’s the perfect cautionary tale for anyone in our business:  don’t take that surprising, unexpected finding at face value—trust your data, but verify that trust by testing every alternative explanation for it. 

*Every time I think of this phrase, “data don’t lie,” I’m reminded of the line attributed to NBA player Rasheed Wallace (now with the Knicks, formerly with the Celtics, and long-time Piston):  “Ball don’t lie.”  Whenever ‘Sheed  felt a foul called on him was wrong (meaning—if you know Rasheed Wallace—every foul ever called on him), if the foul shooter subsequently missed the free throw ‘Sheed would yell to the ref, “Ball don’t lie!”  The foul call was incorrect—the missed free throw verified it.

A shout-out to Lieutenant Columbo

Posted: March 13, 2013

Peter Falk died almost two years ago.  He suffered from Alzheimer’s disease in the last years of his life.  He reached the point where he no longer recognized people.

Peter Falk was best known as the Emmy award winning star of the TV series “Columbo.”  I was a big fan of “Columbo,” and for me it was both sad and ironic that the disheveled, absent-minded detective lieutenant Columbo, who immediately, instinctively, with few clues, always recognized the evil perpetrator, reached the point in his life where he no longer recognized anyone at all. 

When I taught research methods for political scientists to undergraduates and graduate students in the 1970s I found myself using what I called the “Columbo Method” to describe one of two ways to approach the research report writing process—the other being the "Agatha Christie Method.”  I continue to use this distinction with my colleagues here at The Taylor Group (most of whom weren’t even born in the heyday of the great “Columbo” TV series—and so have no idea, typically, what I’m talking about). 

In Agatha Christie murder mysteries, I explain, the audience never finds out “who did it” until the very end.  The viewer is given clues along the way, building up to the ending climax where the murderer is finally identified—and all the clues along the way finally come together. 

On the contrary, in every episode of “Columbo” the audience knows early on “who did it,” through the wily instincts of old, trench-coated, hunched-over, perpetually-perplexed Lieutenant Columbo.  And then the rest of the episode is taken up with just how Columbo figured it out so quickly and ultimately arrests the perpetrator. 

The research report writing process can follow either the Columbo Method or the Agatha Christie Method.   You either build a series of findings toward a grand conclusion for the reader, or you begin the report with the grand conclusion and spend the rest of the time presenting findings that explain to the reader how you got there. 

I miss you Lieutenant Columbo, but I’ll never forget you; and I figure from the raised eyebrows and “here-he-goes-again” expressions on my colleagues faces when I describe the report writing process, neither will they.

On the value of research experience (and the desire for more)

Posted February 19, 2013

It’s hard to argue against the value of experience.  If you wanted to make such an argument, you would have to take issue with Albert Einstein, who said: “The only source of knowledge is experience.”

I have seen its importance over the years with my younger and older colleagues alike.  They get better at what they do with every project.  And I’ve certainly seen it in myself.  I still feel—actually, I know—I get better at what I do with every project. 

Experience in the world of market and opinion research doesn’t just develop and sharpen skills.  It deepens one’s sensibilities, enhances one’s ability to discover important insights—in survey data, in focus group proceedings, in all methodological approaches.  The more my colleagues and I study our clients’ customers and prospects and employees, the more we know about them, and the more actionable insight we can derive from any given study. 

The value of experience:  the more you see, the more you know.  It’s a simple truism.

And then there’s the corollary, which pushes to the value of experience to a higher level:  the more you know, the more you also see.  It’s because you see more when you know more that the opportunity for genuine insights are multiplied.

As powerful as experience is, by itself it feels insufficient.  Something’s missing.  Experience is simply the slow process of grafting new bits of knowledge onto existing bits of knowledge.  It’s edifice-building for sure, but slow, brick-by-brick. 

What’s missing is the energy, the vital force, the passion that fuels the deliberate brick-by-brick process of experience building.

Somewhere along the way I came across the great mathematician Kurt Gödel’s incompleteness theorem, which proved the incompleteness of mathematics (of all logical systems of any complexity.  It’s not a leap to recognize that knowledge of any complex subject is forever incomplete; no matter how far one’s knowledge extends (whether it be logical systems, or in our case, our clients’ customers and prospects and employees and research methodologies) that knowledge is always incomplete . . . . there is always yet more to know

This is what has always kept me going.  It’s precisely the fact that there is always more to know, no matter how much my colleagues I have learned, that keeps what we do fascinating (for ourselves) and essential (for our clients).

The following is supposedly attributable to Lao Tse (the great Chinese philosopher of the 5th/4th Century, BC):  “The more you know, the less you understand.” 

Don’t get me wrong:  I love Eastern philosophy, but I do have a hard time getting my head around it.  Not knowing what to make of this, I did some research and found this explanation:
“The more you know about knowing and the more you see deep into it, the more penetrating you
become, the more knowing becomes transparent to you – and less and less will you know. One day when you have seen the whole game of knowing, suddenly it evaporates.”

There, that explains it.

What's Up With The Bridges?

Posted January 29, 2013

You may have noticed the prominence of bridges on our site. You may also have wondered, what’s up with that?

The fact is that bridges have long been, for us, the perfect metaphor for the work we do—and not just us, the entire market and opinion research profession.

Bridges connect—one side to another.  They solve a problem—how to get efficiently to the other side without having to wade, swim, or float across, or walk, bike, or drive around.  They create opportunity—without them no connection at all between the two sides is a distinct possibility.  They create advantage—linking people and businesses from both sides for mutual benefit.  

Market and opinion research does the same thing.  We connect our clients with their constituencies, by helping clients understand the needs of those constituencies.  We help our clients solve problems, by gauging customer concerns and identifying ways to address those concerns.  We help clients embrace opportunities, by determining how to retain customers and attract prospects.  We help clients create advantage, by sizing up their competitive strengths and weaknesses, and how best to build on the former and turn around the latter.

We build bridges between our clients and those who matter most to them.

Research-as-a-bridge also has great metaphoric stretchability.  Just as there are different bridges appropriate for different crossings—a log across a small stream, a stone footbridge across a narrow pond, a rope bridge across a chasm, a wooden bridge across a train track, a steel suspension bridge across a wide river—there are different research approaches, different methodologies, appropriate for different needs clients have to connect with their constituencies.  The best research organizations have a variety of tools to assist clients with their research needs—most broadly both quantitative and qualitative approaches, but within each, a full range of methodologies tailored to the specific client need—whatever it might be.

Finally, if you think of it, from that perfectly placed log across that small stream all the way to that elaborate steel suspension bridge across that big river, bridges are models of ingenuity, beauty, and value.  In their variety, they speak to the human pleasure in solving a problem that is universal, yet profoundly sensitive to local conditions.

Our clients come to us with a common goal:  to more effectively understand the needs and interests and concerns of those who matter most to them—their customers, their prospects, their employees, their shareholders, etc.  Why?  So they can take effective action toward gaining competitive advantage.  The bridge to that goal is different for every client, every project.  We help our clients build that bridge.

A personal note:  If you have a bridge that you love, and you’re willing to send us a photo of it and a note on what makes it special to you, we would love to see it.  Our goal is to populate our site with bridges that we love, and that you love.  What you see now is just the start.