When we started this journey back in 2011, Hadoop was largely seen as the place to land and prepare data for use in relational data warehouses. A few bleeding-edge organizations used Hadoop directly as a poor-man’s data warehouse, but struggled with Hive queries that were hard to write and would take minutes to hours to complete. The idea of interactive BI (business intelligence) directly against Hadoop seemed like a pipe-dream.

At Platfora, we made a bet that Hadoop’s destiny wasn’t simply to be a cheaper, slower cousin of the relational data warehouse. Rather than creating a pale imitation of the status quo, we saw a way to leverage Hadoop’s strengths and offset its weaknesses to do a better job of exploratory BI and analysis than had been possible before.

Hadoop is superb at two things — it provides a near-infinite data reservoir where data of all kinds can be landed without needing to figure out how it will be used ahead of time, and it is a slow lumbering freight-train of an engine for crunching and aggregating batches of millions or billions of rows.

Platfora uses Hadoop the way it was intended — as a batch work engine — and we built our own scalable in-memory engine designed for interactive (sub-second) response that automatically drives Hadoop to distill raw data into in-memory aggregates, while serving up lighting fast queries and a complete BI experience built native for this Hadoop stack. Ying and yang, woven together to allow superb performance with a never-before-seen degree of agility to get at any Hadoop data today without waiting for an IT project.

Our inspiration was the iPhone — a beautiful synthesis of design and technology that delights everyday users and just works. That’s a high bar, and I don’t want to claim that we’ve attained Steve Jobs-esque levels of sublime design. But we’ve done what hasn’t been possible until now — business users going from raw datasets to visual insight in a day (not 6 months, 12 months, or more).

Along the way, we’ve met hundreds of companies that are committed to Hadoop and the agility that comes with a data reservoir architecture. Fortune 500 organizations, big banks and retailers, web and advertising companies, media and telcos. We’re overwhelmed by the reactions of users inside customers like Riot Games and Edmunds.com that are using data in ways that just weren’t possible before Platfora.

We’re entering a new age. You don’t have to believe us — listen to our customers. The relational data warehouses and traditional SQL-based BI products of the past are becoming the legacy mainframes of the data age. With Platfora, BI is better on Hadoop.

- Option 1 — buy into the traditional MPP database approach of Teradata, Oracle Exadata, etc., and spend 12+ months modeling and implementing schemas, ETL pipelines, aggregation jobs, maintenance scripts and more. It is expensive to purchase, and once implemented is rigid and hard to evolve, but on the plus side provides standard SQL support and good performance. Let’s call this the “Legacy Database Option.”

- Option 2 — store the data in Hadoop, and query it directly using MapReduce, Pig and Hive (Hadoop’s SQL-ish syntax). This option is low cost, scalable and requires no decisions or modeling up front — just write raw data files in any format and figure out how you want to use the data later. However, querying it down the road is complex and requires highly specialized IT staff. Keeping track of what data you have in the cluster can be just as hard and performance is inadequate. Each question can take minutes or hours across tens or hundreds of nodes. Let’s call this one the “Hadoop-Centric Option.”

Both options fail to achieve what customers are telling us they want now: more rapid (but secure) self-service exploration and analysis that allows business users to answer questions that span disparate datasets in unanticipated ways. The solution must be consistently fast, responding to user queries in sub-second time frames, and feeding our appetite for today’s powerful data discovery tools.

Impala solves a part of the problem

Based on Google’s F1 architecture, Impala is an open source Cloudera-led project to improve the performance of Hive by bypassing MapReduce and layering an MPP database-style architecture on top of HDFS. In many ways a hybrid of Options 1 and 2, it improves Hive query performance by about 5-10x. It’s a clear shot across the bow of the traditional MPP database vendors. Impala will lag behind them on raw performance, but it has the potential to replace them by making ad hoc queries on smaller datasets nearly interactive (seconds instead of minutes).

The success of Impala will be great for the Hadoop ecosystem. However, it takes us back into the land of Option 1 (the Legacy Database), with the need for DBAs to manage transformation and maintenance jobs, design and implement aggregations, tune performance, etc.

Impala is quick at querying small amounts of data, but laws of physics dictate that querying terabytes or petabytes of data, is slow. It is slow even with the highly tuned I/O subsystems of Teradata and Oracle Exadata, and far slower in any Hadoop-centric architecture like Impala that needs to pull unoptimized raw files from HDFS. This isn’t news to any DBA, and in response, they manually build and maintain ‘aggregate’ tables — much smaller tables of rolled-up data that can be queried in seconds but lack fine-grained details — and instruct their data analysts to use them rather than the big slow tables that clog up their systems when queried.

But the presumption that a DBA knows what data will be important up-front is usually wrong, and it creates a never ending loop of change orders between analysts and the DBAs building the aggregates. This is exactly the challenge that popular BI tools have when they operate against big data. If a desktop BI user finds interesting data that they want to drill in on, or use as a dimension to slice a different dataset, they must go back to their DBAs for new aggregations that include the data they want; a tedious and slow process that slows down their work.

Worse, if a desktop BI user hits the wrong tables (i.e. the raw data), or submits a complex query, they can chew up vast amounts of cluster resources and dramatically impact other users. This is not the scalable big-data architecture of the future, and it is exactly the painful world that every customer we talk to is trying to escape.

Platfora makes Hadoop consistently fast and self-service

Here is where Platfora comes into the picture. Our platform instantly turns raw data in Hadoop into interactive in-memory business intelligence. Platfora connects in minutes to any Hadoop distribution and automatically generates MapReduce jobs (w/ added Impala acceleration on the roadmap) to build and maintain scale-out in-memory aggregates.

Our scale-out middle tier is simultaneously an ‘aggregate cache’ of the data below, and a lightning fast in-memory analytical query engine to the users above. It holds ‘lenses’ — automatically materialized data marts that roll up raw Hadoop data and can be refined with a click of a button (using our Fractal Cache(TM) technology) to hold whatever level-of-detail is most interesting to users at any point in time. Performance is consistently sub-second, and offloads work from the Hadoop cluster to turn it into a scalable enterprise resource.

The front-end is a completely web-based (HTML5 Canvas) exploratory BI framework, in the spirit of modern data discovery BI tools, but natively built for Hadoop. Now users can interactively explore and visualize, build dashboards, collaborate and storytell seamlessly against any volume and diversity of Hadoop datasets. The front end is tied directly into the middle-tier, giving analysts the first closed-loop exploratory framework that lets them reshape their aggregations or add additional dimensions in a self-service manner without IT involvement.

This is the future

Lets not pine for Hadoop to emulate Option 1, with DBAs architecting and managing every aspect of the data warehouse — constantly modeling, tweaking, and maintaining systems for better performance while hopelessly guessing the needs of their business users. Nor should we put up with Option 2, which is slow, complex and provides inconsistent performance. The future can truly be better than the past — a world of consistently fast, scalable, and modern business analytics and BI for all of your data.

We’ve been living in the dark ages of data management. We’ve been conditioned to believe that it is right and proper to spend a year or more architecting and implementing a data warehouse and business intelligence solution. That you need teams of consultants and IT people to make sense of data. We are living in the status quo of practices developed 30 years ago — practices that are the lifeblood of companies like Oracle, IBM and Teradata.

When I ran product at Greenplum, we understood this reality. Working with brilliant folks like Joe Hellerstein (UC Berkeley) and Brian Dolan (then at Fox Interactive), the team developed practices to navigate around the outmoded approaches of the past. Joe coined the name ‘MAD Skills’ (Magnetic, Agile and Deep).

But we could only distort reality so far. At the end of the day it was still a big relational database. When the rubber met the road, DBAs were doing what they always do — designing data models, building ETL jobs, and tuning indexes and aggregates.

The insight for Platfora came a number of months after leaving Greenplum (post EMC acquisition). I’d been spending a lot of time thinking about Hadoop and why it was gaining so much momentum. Clearly it was cost-effective and scalable, and was intimately linked in people’s minds to companies like Google, Yahoo and Facebook. But there was more to it. Everywhere I looked, companies were generating more and more data — interactions, logs, views, purchases, clicks, etc. These were being linked with increasing numbers of new and interesting datasets — location data, purchased user demographics, twitter sentiment, etc. The questions that these swirling datasets would one day support couldn’t be know yet. And yet to build a data warehouse I’d be expected to perfectly predict what data would be important and how I’d want to question it, years in advance, or spend months rearchitecting every time I was wrong. This is actually considered ‘best practice’.

The brilliance of what Hadoop does differently is that it doesn’t ask for any of these decisions up front. You can land raw data, in any format and at any size, in Hadoop with virtually no friction. You don’t have to think twice about how you are going to use the data when you write it. No more throwing away data because of cost, friction or politics.

Which brings us to the insight.

In the view of the status-quo players, Hadoop is just another data source. It is a dumping ground, and from there you can pull chunks into their carefully architected data warehouses – their ‘system of record’. They’ll even provide you a ‘connector’ to make the medicine go down sweet. Sure, you are back in the land of consultants and 12-18 month IT projects, but you can rest easy because you know the ‘important’ data is safely being pumped into your multi-million dollar database box. Just don’t change your mind about what data or questions are important.

But lets go through the looking glass. The database isn’t the ‘system of record’ — it is just a shadow of the data in Hadoop. In fact there is nothing more authentic than all of that raw data sitting in Hadoop. With just a bit of metadata to describe the data, it’d be possible to materialize any ‘data warehouse’ from that data in a completely automated way. These ephemeral ‘data warehouses’ could be built, maintained, and disposed of with a click of a button.

Imagine what is possible. Raw data of any kind or type lands in Hadoop with no friction. Everyday business users can interactively explore, visualize and analyze any of that data immediately, with no waiting for an IT project. One question can lead to the next and take them anywhere through the data. And the connective tissue that makes this possible — bridging between lumbering batch-processing Hadoop and this interactive experience — are ‘software defined’ scale-out in-memory data marts that automatically evolve with users questions and interest. Enter… Platfora.

Through the looking glass, there is no need for a traditional data warehouse. It is an inflexible, expensive relic of a bygone age. It is time to leave the dark ages.

The status quo is an intricate and expensive supply chain of data that is progressively stored, manipulated and processed until value is seen by a business user. The steps along this chain include storing data in a big (expensive) relational database such as Oracle or Teradata, loading data via a complex ETL (extraction, transformation, load) solution such as Informatica, and building reports and dashboards using business intelligence products such as Business Objects or MicroStrategy. Companies can spend 12-18 months architecting, planning and implementing all of these pieces. When a user’s questions evolve, or new data sources need to get integrated, count on months of lead time for these change orders.

These systems are like a very organized filing cabinet. Everything has its place – except when it doesn’t, in which case there’s nowhere to put it without a major reorganization. But this won’t cut it in a Big Data world. Companies realize that their data is a critical asset, and they need to get increasingly good at answering questions (about user behavior, fraud, product interactions, etc.) that they cannot anticipate today but will be critical in the future. Any attempt to build out our “filing cabinet” ahead of time is guaranteed to be wrong and force us to throw away essential data.

Enter Hadoop – the open source technology designed by pioneers at Yahoo!, Facebook and other companies based on prior work done at Google. The core of Hadoop is a reliable distributed file system (HDFS) and a parallel data processing framework (MapReduce). In other words, you can store lots of data very cheaply (e.g. Facebook has 100 Petabytes in their cluster), and you can write programs or query the data with performance that scales with the number of nodes. By the analogy, Hadoop is more like a huge cardboard box – ready to be filled with endless amounts of raw data without any need to agree on data models or make any assumptions about data usage up-front.

Hadoop is an incomplete solution in many ways, and yet it is rapidly gaining adoption in Global 2000 enterprises. We’ve spoken in-depth to over 100 large enterprises and web companies about their Hadoop initiatives. What we’ve heard repeatedly is that these projects are driven by mandates to stop throwing away data that may be useful. This data is their lifeblood, and they must not squander it. However, the rapidly increasing variety, rate of change, and size of datasets has swamped their traditional systems, and storing it in Hadoop is their only viable way of keeping up. For all of its weaknesses, Hadoop is the only credible answer to the Big Data challenge of moving beyond the “filing cabinet.”

Hadoop is irresistible for this reason, but the big question that remains is how to use the data there once you’ve stored it. The challenge is that Hadoop is a very different architecture to traditional data warehouses. It is a batch engine — a lumbering freight train that can process immense amounts of data, but takes a while to get up to speed, so even the simplest question requires minutes of processing.

Given this, be very skeptical of the legions of business intelligence (BI) and data warehouse vendors who are racing to announce their “Hadoop connector.” Simply treating Hadoop as another “data source” for their legacy product won’t work – the data isn’t structured like a relational “filing cabinet”, and even if was, no user wants to wait an hour for the next page when they drill-down in a report.

The good news is that a new class of much more intelligent Hadoop-native solutions are on the horizon – leveraging Hadoop correctly for predictive analytics, interactive/exploratory business intelligence, agile ETL, and a multitude of vertical solutions. Here at Platfora we are laser focused on this next phase of Hadoop. The result won’t just match the status quo, but exceed it in flexibility and the ability to scale and adapt to changing requirements. Exciting times are ahead – stay tuned.

THE ‘BIG DATA’ SHIFT

“Big Data analysis is usually iterative: you ask one question or examine one data set, then think of more questions or decide to look at more data. That’s different from the “single source of truth” approach to standard BI and data warehousing.” — PwC 2010 Technology Forecast

You could be forgiven for thinking that ‘big data’ is just a new name and a lot of hype around the same data work that IT has been engaged in for decades. But you’d be mistaken – there is a fundamental change underway that will transform how every industry uses data.

The way we’ve managed data for the last 25+ years assumes that IT has a complete picture from the get-go of all data sources and the requirements of their end-users. Based on this they spend 12+ months building a data warehouse that hard-codes these assumptions. Users are trapped in this limited view of the world, and even simple changes can require six months of engineering. All in all not a great solution, but sufficient in an age when data sources rarely changed and users were satisfied with the same canned queries and reports every day.

What has changed – the heart of the ‘big data’ shift – is only peripherally about the volume of data. Companies are realizing that there is surprising value locked up in their data, but in unanticipated ways that will only emerge down the road. These data sources could span functional silos and organizations, and include external ones such as Twitter and foursquare, weather and economic sentiment or pop trends. There is no way that IT can predict up front which data will be needed or how the users will need to combine or distill this data to test hypothesis and converge on answers to each question.

Before, we only stored the data that we knew we needed. Now we understand that every scrap of data may play a role in answering next year’s unanticipated questions. The cost of keeping this data – in commodity clusters such as Hadoop – is next to nothing. The cost of throwing it away is untold.

FIRST STEPS INTO A BRAVE NEW WORLD

We talk to a lot of companies that are navigating this big data shift today — across industries such as advertising, finance, media, retail, telecommunications and government. Given the choice of throwing away data or entering a brave new world, they are consistently choosing the latter. They are starting to load all of this data into Hadoop, to keep it, often without any real understanding of how to empower everyday business users to take advantage of that data.

This was driven home at a couple of big data events that I spoke at over the past month. The first was the Accel Big Data Conference, a superb event run by Ping Li, where we discussed enterprise usage of big data with other panelists from Facebook, JP Morgan, and Cloudera. The second was the Berkeley DataEDGE event, in which Quentin Hardy from the New York Times led a really insightful conversation with our panel about startup innovation in the big data space. In both, the need for simple, democratized access to big data by everyday business users rose up as the most critical issue needing to be solved.

THE BLACKSMITH AND THE DATA SCIENTIST

“What Big Data is seeing now looks like the classic industrial curve. There is the first discovery of something big, leading to establishing principles like scientific rules. Science moves toward engineering as a means to manufacturing, resulting in mass deployment. Then things really change.” – Quentin Hardy, How Data Gets Real (NYTimes 06/04/12)

Today, there is a fundamental lack of tools and technologies that make big data usable by everyday business users. Even the simplest questions require developers or data scientists to painstakingly wrangle the data into some useful form. In ‘industrial revolution’ terms, we are in the pre-industrial era of artisanship that proceeded mass production. It is the equivalent of needing to engage an expert blacksmith to forge the forks and spoons for our dinner table, rather than being able to easily and cheaply buy mass-produced flatware.

Since every company of any scale is going to need to leverage big data, as an industry we either need to train up hundreds of thousands of expert blacksmiths (aka data scientists) or find a way into the industrialized world (aka better tools and technology that dramatically lower the bar to harnessing big data).

If we can achieve the latter, then data scientists still have a critical role — as experts applying their sophisticated techniques and training to questions that demand that attention. And yet, most business questions won’t require that expertise, so with these tools everyday users could use their intuition and domain expertise to work with data in productive and meaningful ways.

That is my bet — that it won’t be long before we see the first signs that we are entering the industrial age of big data. We’ll begin to see tools and techniques that truly democratize access and allow business users to serve most of their own data exploration and visualization needs, while allowing data scientists and more sophisticated users to collaborate and impart their unique value. Exciting times ahead.

We’ve got a few silver bullets though, and we just added another one. We had wrapped our Series A, and were done thinking about financing for now. But when Mike Speiser from Sutter Hill Ventures reached out and wanted to chat, I couldn’t say no. We are huge fans on Mike’s work an CEO/entrepreneur (most recently at Pure Storage) and on the venture side, and he is one of the handful of investors that we’d go to extreme lengths to get involved at this stage.

Mike had done his homework on us, and in the blink of an eye we’d added another Series A closing to include Sutter Hill Ventures (bringing our total to $7.2 million) and added Mike to our board of directors.

One of Mike’s passions and talents is building incredible engineering teams – e.g. at Bix/Yahoo!, VERITAS/Symantec and Pure Storage. He has superb instincts and experience assembling remarkable talent that executes at a different plane than other companies. We’ve been living this philosophy since day one of Platfora, so together we’re going to go even faster as we bring together the best of the best in the face of this hyper-competitive market.

We’re very fortunate to have what we believe is the best group of investors on the planet. With Scott Weiss from a16z, Mike from Sutter Hill, T.J. Rylander from In-Q-Tel, and a group of the smartest and most passionate angels around.

The common thread that connects us and all of our investors is an understanding that we live in historic times. Marc’s recent op-ed in the Wall Street Journal, “Why Software is Eating the World,” eloquently laid out how even the most traditional industries – from agriculture to health care to national defense — are at a tipping-point and are being transformed by new software ideas.

But the counterweight to the power of software is in the data it generates. Software can be used to process and understand data, and software is also a massive generator of data. In fact “machine generated data” – data produced by software systems – is growing overwhelmingly faster than “human generated data.”

This data is the future. For any company, data holds the ability to understand, predict and better service customers. Data tells the tale of interrelated events related to fraud or threats. Datasets answer questions about health trends and risk factors, financial market sentiment and inefficiencies, and social influence and behavior. We must be mindful of issues like privacy and security, but there is no escaping that data is increasingly capturing the pulse of the world.

In the same way that ‘software is eating the world’, companies around the globe will either sink or swim with data. But existing data warehousing and business intelligence products are relics from another age. They contemplate carefully organized, structured and curated datasets that grow and change slowly. Every single company I’ve talked to over the past six months has shared tales of how those systems are breaking. My friend Mike Driscoll, CTO of MetaMarkets, recently tweeted, “Prediction: there will be a mass extinction event among BI vendors in the next decade. They won’t survive the data deluge.”

If the old is breaking, what is there to replace it? Hadoop is the industry’s best hope. Based on Google’s MapReduce, it is universally in use at web companies (Facebook, Yahoo, Netflix, etc.) and is making deep inroads in every one of these industries facing the data deluge. It is flexible, scalable, and has the promise of taming terabytes or petabytes of data. But despite Hadoop’s amazing potential, it is low-level infrastructure that has similarities to batch-processing system from the 1960s. Everything takes minutes to hours to run, and jobs need to be carefully submitted by experts. It isn’t interactive like a traditional database, so just layering a BI product on top is asking for disappointment. Interactive reporting for business users simply doesn’t exist for Hadoop.

This is where Platfora comes in. We bring business intelligence into the 21st century, giving business analysts the intuitive and richly interactive tools to explore and produce business insights from massive and rapidly evolving datasets. Whether a company has gigabytes, terabytes or petabytes of data, our platform eliminates the need for traditional data warehouses, ETL tools and the legacy BI products of the past. We replace complexity and scaling pain with simplicity and beauty.

Platfora’s breakthrough is a combination of server technology, user experience innovation, and data science. Our platform works with existing Hadoop clusters (Cloudera, MapR, Amazon EMR, etc.), and automatically turns the questions of business users into Hadoop jobs that synthesize and distill Hadoop datasets into dimensional and predictive dashboards, reports and insights. The system intelligently drives Hadoop to create and maintain ‘work products’ — highly compressed partial results that are refined at the click of a button to achieve subsecond report delivery, analytics overlay, and drilldown performance.

We are hard at work building our product, and working closely with a handful of industry leading companies to get it right. More importantly, to get there we are assembling a superb team of data and distributed systems architects/engineers, UI and UX developers and data scientists. The team is still small, but growing fast, and we’re focused on high-energy teamwork and building product that changes the world. Come join us.

The Reality. That one scratch below the surface and you realize that this whole thing is at a much earlier stage than most people appreciate. We’re still living the first baby steps — vendors battling to provide the best low-level Hadoop plumbing. Once you get the cluster up and running, now what? You start to pump data into it. Then what? That’s when things start to unravel. You can write some basic Hive or Pig queries, and wait minutes or hours for jobs that take seconds on MPP databases. You can get down to the metal and try your hand at MapReduce programming in Java, or maybe adapt some Mahout machine-learning code. However you cut it though, unless you are willing to get your hands very dirty, it is going to be a rocky ride. This technology is a long way from main street.

The Future. That’s where we come in. We’re not ready to talk about what we’re doing yet, but we believe that the Hadoop that the world needs (or is able to consume) is a far cry from what you see today. The big data explosion is real and imminent, and companies worldwide need a way to get ahead of the onslaught and be ready to put that data to work. The Hadoop that we envision is not just evolutionary enhancements of today’s stack (although there are plenty of vendors that have that angle covered). It is tangibly more. That’s all we can say about our plans for now.

At Platfora, we are assembling an extraordinary team of big data and scalable systems experts, rock-star data scientists, and sublime UX/visualization designers. If this describes you, then we’d love to talk. It is going to be quite a ride.