A bias we all encounter is confirmation bias – the tendency to find and interpret information in alignment with our expectations. And then, to stop looking. Mirror, mirror.
We tend to think of ourselves as unbiased. But any time we execute a “first ‘sensible’ answer wins” analysis, that’s confirmation bias. For most of us this will happen every day. Who has time to check out all the possibilities for every little thing?
That approach is OK, until the answer’s value is greater than the cost of examining the reasonable options. Which is a lot of the time, in most research and corporate databases.
A quick story: I once watched a team create a list of retail-store sales by store manager, and before I knew what was happening people were trying to understand why managers in low-performing stores were doing such a bad job. It took a real effort on my part to restart inquiry, and get the team to look for other possible explanations. As it turned out the data couldn’t tell us which explanation was determining – it could have been inventory, location, or the managers.
Scenarios like this can be discouraging – our additional work only showed the limitations of what we actually understood. Confirmation bias already urges us to stop short of full inquiry. Beyond that, no one wants to expend energy and come away with less.
However, we really didn’t come away with less. The supposed conclusion was never supported in the first place – a potentially valuable outcome, if not necessarily what we hoped for.
What we want, hope for, and talk about is “insight.” It’s an overused term, and I think we are conditioned to expect insight just because we have a big data system on hand. Regrettably, our data did not sign up to tell us anything new. Real insight is relatively rare – a genuine surprise that can be sustained through the trial of full and unbiased inquiry. Rare maybe, but worthwhile too – it’s the reason we build this stuff.