Engineers and scientists will testify with near-unanimity: it is surprisingly simple to create a complex solution, and surprisingly complex to render a simple solution.
Particularly as our systems grow larger, complexity tends to happen all on its own, without any help from us. Still, we do sometimes give complexity help it surely doesn’t need. We expand our systems with data of dubious value, to the point they are difficult to maintain; we twist problems to meet the tools we know; we ask algorithms to do jobs that could be better handled by an improved problem formulation or data representation.
Complex constructions all too quickly yield systems where contact with the questions and outcomes is uncertain for users, and even developers.
The cost of complexity is simple: opaque and complex systems may pass muster when outcomes are expected, or unexamined, or simply ignored. However, when people do not understand an unexpected or interesting result, they will not accept the outcome. When we need outcomes the most, complexity defeats their useful application.
It’s simple enough to complain about complexity. But more to the issue, we often take simplicity for granted. Simple solutions are often very unassuming, naturally connecting inquiry to answers. And because they can be unassuming, we find ourselves assuming that a what is simple now was also simple throughout its development, rather than a process of iteratively refining questions, models, transforms, and continually removing what is unessential and complicating.
Perhaps ironically, tools now falling under the umbrella of “data science” are some of the most potent simplifying tools available for maintaining simplicity, offering the opportunity to simplify data models and representations, and set aside data with little probative value. These tools are best applied at design time and used throughout development, for simplicity is much easier to sustain than to retrofit from a large, complex, and essentially immobile system.
How often do we add in low-value information to our systems, making them large and inflexible, motivated by our worry that when our system becomes large and inflexible we’ll not be able to make a change later? I understand the reasoning, but a smaller and more manageable system should never have this issue – new information with genuine value can always be added later. That’s as it should be: asking good questions is almost always an iterative process, rarely gotten entirely right the first time through.
I often see over-complex systems, too big to change, long after change is precisely what’s needed for better adoption or improved inquiry. However, I don’t believe I’ve ever seen a business intelligence or analysis system that truly had to be that way. Simplicity isn’t impossible to achieve, but it is hard work.
Those crafting simple and transparent solutions deserve our appreciation for what truly is a complex – but very worthwhile – task.
Particularly as we contemplate larger and larger data systems, simplicity is a worthy design and development goal. It is simplicity – not complexity – that brings solutions to our most challenging problems, where questions, answers, data, and metrics will evolve to their final form. Simplicity offers us comprehension; comprehension brings challenges to our early outcomes; those challenges bring improvement; from improvement we reach adoption; and with adoption we can assure impact. And all of that change and iteration is only possible with solutions that begin – and stay – as simple as possible.