The Democracy Of Analytics

Democracy has always been in short supply. With Canada, Germany, and California now among the few nation-states carrying the torch of liberal Western democracy,  this is not a condition quickly to be reversed.

Perhaps coincidentally, the analytics community is filling this void with the idea of self-service analytics – otherwise known as “democratization” of analytics.  Democratization says that if each of us has analytics software and our own personal petabyte (or so) of storage, the march of progress will be radically accelerated – each of us will investigate our information, each of us will craft data models and engaging dashboards, each of will share our insights with an eager audience (when they are not crafting their own solutions), and each of us will propose new and improved answers to pressing world problems, like the decline of Western liberal democracy.

I approve, which has irritated a few of my technical friends, but approval is different than saying “this is a concept without issues.”  There are certainly issues with democratization, and those issues will alter how the technical community supports analytics endeavors in the future.  Nonetheless, I approve.

I approve, because the essential starting point of analytics is a well-formed question, and only those with expertise and context for a particular problem can even begin to craft those questions.

I approve, because with the right support, there is no reason why any person shouldn’t be able to view the essential consequences of their information.  Personally, I don’t like elitism in any form.   Those who can pose the questions and most need the answers should be allowed to implement their vision.

I approve, because a first round of questions and answers almost always leads to other questions, often requiring more involved techniques and analysis and professional assistance.

I approve, because my approval is irrelevant:  analytics democratization is empowering, and right or wrong it seems to be the course of the future.

As for the problems, they relate to how much analysis each and every user must perform.  Democratization could be a productivity boon or a costly fiasco, depending on how it’s implemented.

Democratization as I’ve heard it proposed could easily make matters worse, by asking each user to sort out data assessment, data modeling, appropriate visualization, and exploratory and predictive models.   Then, apparently,  people would vote on what they think is best, with the “best” solutions moved forward to production.  That won’t work – democracy is great, but when everyone votes on everything all of the time, the result is nothing more than a gruesome mess, whether in politics or technology.

The problem with the “everyone votes” scenario is this: knowing the questions and issues involved in setting up data infrastructure, data assessment, data models, exploratory and predictive analysis is a process that takes years.  We cannot, at least with present technology, ask a general user community to craft a star schema (or know the value that brings to a particular class of business questions), or analyze mismatched keys, or ascertain whether an empirical model is robust to extrapolation, or create encapsulated object classes – and then vote on what’s best.  Those are still the necessary activities of IT professionals, and they will remain necessary.  No one, professional or otherwise, will obtain the right outcome from a poorly-crafted and -validated data model, data, or predictive model. Or even know how wrong the answer is.  Even in a democracy, the stage must be set.

With the stage set, we may just see the potential from democratization advertised by its supporters. But without appropriate platforms for this new analytics democracy, analytics could actually regress.   For this democracy to succeed, there will challenges for both the communities of full-time data professionals, and for newly-empowered general analytics communities.

We IT professionals may soon be stepping away from the day-to-day business of solving conventional analytics problems or building standard visualizations. None of us should really have an issue with that – after all, the value and excitement of standard analysis comes from the investigation and results, far more than the mechanics.

Instead, IT people may soon find themselves having to solve “meta analytics problems” – essentially anticipating and defining a class of questions and techniques, and presenting those in a form that allows general users to focus on investigation rather than mechanics.  We shouldn’t underestimate that challenge – it’s significant.   Among other things, solving the “meta problem” will require tools and techniques that allow users to easily assess limits in their results – something that is now pretty rare.

The general user community brings context to particular analytics solutions, the value of which is difficult to overestimate.   For this community, the biggest challenge may be appreciating limits intrinsic to any analysis.  In an age when many software packages can be had for a free download, it’s also a challenge to know what tools are likely to bring success, and which will bring only pain.   Just like professionals, a “no black box” rule can be a pretty good idea, both for transparency and for assuring we know what we’re getting.

So there will be challenges for everyone, and that’s good for everyone. Otherwise we’d all very quickly be out of a job.   Analysis democratization?  Bring it on – the roles of IT and general users may shift, but each should have a critical role in bringing about a democracy of analytics that ultimately will serve all of us better.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s