And, You Were Expecting?

This comment about the Trump administration’s “attack on the internet” suggests that giving internet service providers unrestricted freedom to screw customers (and each other) is a remarkably bad idea, in part because most customers give their providers even lower approval ratings than the Trump administration itself.   As if we didn’t have enough problems…

Regarding those poor customer ratings: it isn’t just customers who can give ISPs and telecoms poor ratings. I was once helping a team analyze customer surveys for one of the ISP giants. One of the early and simple outcomes was that customers gave this particular company a median customer satisfaction rating of 3.  3 out of 10, that is.

Everyone took this in stride, including our ISP colleagues.  3 out of 10 seemed about right. There wasn’t even any particular concern, amongst anyone, that the rating was so low.

After about two weeks, I received an email explaining that I had misunderstood how the customer-satisfaction question was coded – while respondents indeed gave a 1 to 10 answer, in this case (and no others) a 10 was the worst score, and a 1 was the best.  So 3 out of 10 was really meant 8 out of 10. Hey, it’s a tough business – a satisfaction level of 8 out of 10 is really pretty good.   And everyone was pleasantly surprised, making the new answer even better.

And what was the real mistake in this?   The explanation that allowed a 3/10 to be magically transformed to 8/10 was perhaps suspect, but let’s assume that was a genuine analysis error.  The bigger mistake was to accept either answer (3/10 or 8/10) as valid.   3/10 carried the bias of expectation – we know we’re pretty bad.  This could have been telegraphed to a respondent in quite a few ways, from question phraseology to presenting one question next-door to others that are unlikely to be answered positively.   On the other hand, 8/10 was so much better than expectations, that this result should have been questioned also.

As it turned out, the survey data did reveal that respondents would answer a series of questions very similarly, not bothering to parse subtle differences between each question.   So many of the answers were not independent at all, throwing all individual answers into doubt.  The real mistake in this analysis was to consider any answers outside of the expectations (or biases) of the survey creators.

In my experience, surveys are amongst the data most susceptible to producing what people expect.  But the problem of producing invalid but expected answers is surprisingly prevalent, in part because we all want to do a good job.  Teams of data engineers and analysts naturally want to provide satisfactory answers for their customers, and most customers have some idea of what constitutes a reasonable outcome.   It can be a very handy to learn what our clients expect before doing anything else, and to be productively suspicious when outcomes either entirely align or strongly disagree with those original expectations.   We should explain not only the answers that disagree with expectation, but also those answers that entirely agree with expectations.   Some answers are just too good to be true – regardless of what the data are telling us.

I’ve found asking up front about what clients expect in terms of answers helps with managing bias.  (Arguably asking about expectations is functionally like asking about “bias,” but while few of us want to tell about our biases, most of want to show how smart we are by sharing our expectations for an outcome.)   Biases get worse the more time and effort we put into a project, so asking about expectations up front and then checking outcomes against expectations makes any surprises smaller and less painful.   And a lot of the time, we’ll get to say:  “Look, you were right…”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s