Since we started, the Hadoop ecosystem has grown bigger and become stronger. In early 2011 one could argue that it was unclear whether Hadoop would be a major success in the enterprise. Today it would be impossible to make the same point. Almost every Global 2000 CIO is investigating how Hadoop can impact their business, and many of them have already put Hadoop into production in their infrastructure.

The state of the commercial market is strong, led by a pack of trail-blazing Hadoop distributions, and getting stronger with each new entrant and advancement in technology. We’re often asked what the distributions think about Platfora. Are they happy about us building on top of the Hadoop? The answer is clearly “yes.” Along with our announcement today, Cloudera, Greenplum, Hortonworks, and MapR have all shown their support by talking about our partnership publicly. We have a great symbiotic relationship with these companies. They provide the underlying storage and raw processing power for big data, and we provide the “intelligence layer” on top. During our beta program we had customers running each one of these distributions of Hadoop and the combined products worked together seamlessly.

We think this is very important. Our customers want to know that they will be able to take advantage of the rapid advancements in Hadoop, and we ensure that. Going forward, Platfora will continue to invest in supporting the latest technology in the Hadoop ecosystem, including the emerging category of faster SQL-style interfaces to Hadoop. With Platfora, end users will have access to all the data in Hadoop through an interface designed for them and IT can provide consistently fast access without the hassles of managing ETL and a data warehouse.

Although we want to ensure our customers that we will continue to support all the major Hadoop distributions, we did want to publically thank Mike Olson and the team at Cloudera for the advice and connection brokering over the past two years. We look forward to continuing the partnership now on the other side of GA.

Please read MapR’s blog about the Platfora announcement.

Please read supporting statements from our partners about our GA announcement.

However, over the past few years, three positive factors in enterprise computing have created the perfect environment for real innovation in the market of business analytics. First, the exponential growth of data that is generated by our digital world, or “big data.” Second, the trend that started around the time I purchased my first iPhone and I asked the IT team to support it, or “the consumerization of IT” (which implies that I would want to work with the same type of technology at work as I do on my personal time.) And third, the incredible success of open source Hadoop, which can be best described as a movement, in both enthusiasm of its developers and enterprise adoption.

There is a fourth factor, a negative one, that has an amplifying effect in this environment. It is the status quo of business analytics. Without describing it in too much detail, the process of getting data from collection to an analyst is ugly. Three legacy tools — ETL software, the data warehouse, and business intelligence — form an inefficient and inflexible pipeline that takes way too long to answer questions. It is even worse in a world where the questions are frequently changing.

So, Platfora started with a simple question: if we were to rebuild the entire experience from data collection through analysis, what would it look like?

1) The user experience should be beautiful and intuitive. It’s got to run completely in a web browser, on any platform, and it needs to be collaborative with other workers. It’s 2012!
2) The interaction with data has to be responsive. There can’t be a delay because the data is too big.
3) The user shouldn’t have to open an IT ticket to make it work. The components need to work together seamlessly and automatically, like they were made to work together.

That’s tall order. There was no product on the market that spanned from Hadoop up to the business analyst in a truly interactive web-based BI application.

So we built it from scratch, and today we introduce Platfora.

Platfora is the first business analytics product utilizing a closed-loop, iterative approach to working and refining large data for analysis, driven by users. We call this the “Interest Driven Pipeline” and it has been core to the Platfora vision since the beginning. Of course, it is not economically possible to work with petabyte datasets in memory, so making them interactive means managing what data is fast. The Interest Driven Pipeline puts the business analyst at the center of the process and it’s a completely new experience.

Platfora provides a 100% HTML5 interactive BI application. It turns out there is a reason this hasn’t been done before, at scale. When you are working with complex visualizations, web technology simply has not been fast enough to provide interactivity with the data. We’re using “Canvas” technology that lets Platfora render 100,000+ marks in a single visualization and make them interactive, which is a feat on its own. Plus, Platfora incorporates collaborative features to make working together easier and there are no per-user license fees to prevent everyone from having access.

At the heart of Platfora is an entirely new scale-out, in-memory data processing engine that progressively refines details from raw data in Hadoop. We call it Fractal Cache technology. It allows the user to work with various levels of detail of the data and easily move between them without having to rebuild a data warehouse in the process.

So that’s Platfora. Which brings me to the final, and most important factor, in innovation: finding the incredible team to carry it out. Platfora has been driven by an exceptional team of engineers, designers, and visionaries. We’ve been pulled together by our own deadlines and goals. It’s a team of real believers that did what we said we’d do. After all the hard work of the past 18 months this team deserves to step back and realize they have created a new category — something truly innovative, the first of it’s kind.

For me, the most striking change was the shift in discussion of Hadoop as infrastructure (and the community, projects, and roadmap to advance that infrastructure) to the applications being developed around Hadoop to make it useful for the enterprise. The customer-led sessions — the ones where real users were sharing their Hadoop battle stories and and best practices — were so popular that standing room only turned into people standing in hallways outside the halls.

My favorite sessions were customer-led as well. Sears Holding Company CTO Philip Shelley did a fantastic talk in which he described the modernization of their data analysis infrastructure from mainframes running batch COBOL jobs to Hadoop and 400 lines of Pig. He walked through a few use cases in detail (i.e. calculating price elasticity for all their products, local pricing, etc.) At one point he put up a slide that detailed all the various systems and transformations that they had weaved together to get what they needed. There was one Hadoop tool vendor in there for a small slice of the work. It was complex. It left me feeling like this customer was an early explorer in the wilderness, coming back to tell the next wave of explorers the story of his journey.

Electricit’ de France (EDF) Project Manager Marie-Luce Picard described her company’s struggles modernizing their reporting infrastructure for the smart grid. In the old world they measured each customer’s power consumption two times per year. Now, they were measuring each customer’s consumption every ten minutes and generating time series visualizations for millions of customers. (There is a good question to ask: why only every ten minutes? How about every second?) Again, this was a complex recipe of home grown software, components of the Hadoop ecosystem, and lots of time from smart engineers and data scientists to achieve the required results.

The state of Hadoop use in the enterprise can be summed up by one of my favorite tweets of the week:

There is a ton of custom work being done to achieve every use case. Every bit of success and every failure is being passed on between users that must custom craft solutions even for common business problems.

The first generation of tools available for the enterprise has helped, but we’re a long way from Hadoop being useful and simple to the average enterprise out of the box.