Sports Illustrated recently carried a perceptive article on the use of analytics in the NFL, which varies widely across the league. Some teams employ predictive models only as a rough definition of how a player’s attributes would normally relate to their expected performance, so they can knowingly do something outside the model, and the normal. It’s a discipline where success comes from establishing new trends, more than following existing ones.
Something similar happens in baseball. When I interviewed for my first job, my to-be-boss presented me with one of Bill James’ tomes – and I was very interested to learn that James didn’t give a damn about formal analytics. What he cared about was developing new metrics and ways to measure performance. (I still think the “Hall of Fame Watch” is a masterpiece of scorecarding.) Real differentiation comes from understanding when a model will be wrong, for one player or a group of players.
We aren’t playing in the NFL or MLB, brother…. Undeniably, but many engaged users act like they are, and use analytics accordingly – less to predict their situation, than to understand how that situation differs from normality. And then to act, not by the dictates of analytics, but rather by using analytics to gauge how far they’ve traveled into the realm of the unnormal when they overrule analytics.
We might be tempted to see this as a problem. Actually, it might be the best user scenario there is: these are users who are engaged, who value analytics outcomes, who understand the limits of those outcomes, and who augment the outcomes with local expertise to get just what they want.
Using outcomes without question, or not using them at all, are the real problems. We can get either or both, by attempting to shrink-wrap outcomes in a “here is all that matters; do this; don’t do that” format.
Now, when enthusiastic users say that Your stuff is great! Here’s how we use it – we start by pulling a few numbers from this table…. And when we’re finished, we email everyone on our team! – I listen, say “thank you,” and admire their ingenuity.