My Favorite Users

Sports Illustrated recently carried a perceptive article on the use of analytics in the NFL, which varies widely across the league.  Some teams employ predictive models only as a rough definition of how a player’s attributes would normally relate to their expected performance, so they can knowingly do something outside the model, and the normal.  It’s a discipline where success comes from establishing new trends, more than following existing ones.

Something similar happens in baseball. When I interviewed for my first job, my to-be-boss presented me with one of Bill James’ tomes –  and I was very interested to learn that James didn’t give a damn about formal analytics.  What he cared about was developing new metrics and ways to measure performance.  (I still think the “Hall of Fame Watch” is a masterpiece of scorecarding.)   Real differentiation comes from understanding when a model will be wrong, for one player or a group of players.

We aren’t playing in the NFL or MLB, brother….  Undeniably, but many engaged users act like they are, and use analytics accordingly – less to predict their situation, than to understand how that situation differs from normality. And then to act, not by the dictates of analytics, but rather by using analytics to gauge how far they’ve traveled into the realm of the unnormal when they overrule analytics.

We might be tempted to see this as a problem. Actually, it might be the best user scenario there is: these are users who are engaged, who value analytics outcomes, who understand the limits of those outcomes, and who augment the outcomes with local expertise to get just what they want.

Using outcomes without question, or not using them at all, are the real problems.   We can get either or both, by attempting to shrink-wrap outcomes in a “here is all that matters; do this; don’t do that” format.

Now, when enthusiastic users say that Your stuff is great! Here’s how we use it – we start by pulling a few numbers from this table…. And when we’re finished, we email everyone on our team! – I listen, say “thank you,” and admire their ingenuity.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s