Be in the know.

The Platfora blog.

Going from Information to Decisions, Big Data Style


Since the industrial revolution, we’ve been fantastic at applying new technology to automating action.  We’re now at a point that you can run a factory floor by computers and robots, with just a few workers to keep the machines up and running.  There’s still innovation to be made there, surely, but the potential incremental gain is getting smaller and smaller.  Information systems (databases, spreadsheets, ERP systems, et cetera) have similarly experienced the same type of innovation – allowing for record keeping to occur rapidly and relatively easily.  However, the way we make decisions hasn’t evolved nearly as rapidly, and that innovation started much more recently.

It’s surprising that in today’s fast-paced world where information is available at our fingertips, corporate leaders often have to make decisions the same way that they did  years ago – by using their gut instincts.  Sure, they’re using computers to tabulate the data, build reports, and create dashboards.  But the data is growing faster than traditional systems can manage and coming from less structured sources – making it hard to pull all of it together for a meaningful analysis. There’s too much at risk to make business decisions from a limited view.

Forward-thinking companies have been looking to analytics technologies to give them the ability to derive business insights using data from all of their customer touch-points.  Early adopters of analytics software focused on specific pain points, where an organization was looking to reduce the cost of asking a few specific, high value questions. Today we’re seeing a push towards asking more varied questions across all relevant data sources – but that requires a class of tools that are usable by a broader audience.

Early adopters found success in adopting these new technologies to revolutionize their business, and now are way ahead of their competition. Thanks to the growing amounts of data from the Internet of Things and other machine-generated data sources, there will be a bigger push to innovate at a strategic level throughout a broader group of organizations.  The innovations in Big Data Analytics now allow companies to pose deeper strategic questions:  questions such as, ‘Can my company make more money by paying our employees more?’ or ‘What is the impact of customer sentiment to my business after negative press?’ can now actually be answered, as long as you are collecting enough of the right data in the same place.

Today, we finally have the infrastructure in Hadoop that’s necessary to tackle some of these higher-order questions.  But Hadoop isn’t the entire story.  From there, you need tools that can actually formulate the questions.  Programming languages and frameworks like R and MapReduce help, but it isn’t easier for a human to understand the output of those tools than it is to understand the raw data going in (trust me, I’ve tried).  It also isn’t easy to formulate their questions using these tools.  So either we’re going to reserve these insights for organizations who are willing to invest in lots of big brains, or we’re going to come up with a new class of tools that can drive this Big Data infrastructure in an easy to understand way.

I firmly believe that while both brilliant minds and better tools are important to moving the needle, true innovation in this space is going to come in the tool front – allowing more users with disparate skillsets the opportunity to ask their own questions in an easy-to-understand format.  Those tools should be accessible and approachable by a broader group of decision makers throughout your organization, and not just the big brains you hired to move the needle forward in the past.  These tools should allow for self-service access to your data holdings to allow for quick iteration over your problem set – allowing business decisions to avoid being bottlenecked by IT.  And they certainly need to be easy enough that Big Data neophytes shouldn’t be intimidated by a high learning curve.  Oh yeah, and they should be able to help you derive actionable insights too.

Big Data Analytics solutions like Platfora don’t really fall into either of those broad categories I mentioned earlier, or rather, they fall somewhere in between the two.  Finally we have a technology that can successfully bridge the gap between the systems that control the action and the systems that manage the information.  Tools that drive Big Data Analytics, like Platfora, are at the forefront of building out this new technology bridge, and that’s what gets me out of bed everyday.  There’s still a lot of work to be done to prove that these new technologies are actually going to change the world, but I’m excited to see what’s coming next.

Keith McClellan leads up Federal Engineering at Platfora, and has been focused on Big Data and related technologies for most of his career. If you’re interested in his random musings, he tweets @keithmcc and occasionally writes for the Platfora blog.