Is the Centralized Data Warehouse Dead?

In the past I’ve heard some criticisms around Teradata’s founding “vision” -- criticisms that have been bolstered by the misperception that Teradata has admitted that its original vision of a “single store of business data” is no longer correct. Allow me to clear a few things up…

Teradata started with the vision of “bring data together” so users could ask “any question of any data at any time.” At our heart we started as an analytic software company. We revolutionized a new paradigm of MPP software that was able to scale to (at the time) unfathomable amounts of data; we were often told that “no one will ever need a Terabyte of data!”

The vision was about bringing data together to drive analytics, but unfortunately the technology of the day did not match our vision. Data transfer speeds and cost were prohibitive to be constantly moving data around networks, compute in single instances were not able to effectively manage the massive amounts of data in a timely manner, and storage was still measured in megabytes.

We had to build a new class of hardware to execute our vision. We took discrete compute and storage instances and put them together to look like a single system (sound familiar?). We had to overcome the limitations of technology and, not only logically, but physically, bring the data together in order to remove the barriers to realizing our vision.

And we succeeded. I was fortunate enough to be part of the team that delivered the first system with over a Terabyte of usable space in a single system at Wal*Mart back in late 1992. 

The landscape changed….

Of course, back in the 80’s and 90’s it was much easier. All the data scattered across the corporate landscape was basically relational by design. It was created, managed, and used for reporting purposes. Once we conquered the problems of scale and compute, it was simply a matter of adding more of the data silos into the “centralized data warehouse” (a term I really don’t like, truth be told).

But, our vision was not to simply store data together, it was to integrate the data so that is was usable for a variety of purposes and served the enterprise as a whole, not as a series of departmental answer sets. Our vision was about data and analytic integration.

Cut to the 2000’s and lots of things were changing. Data started being generated in many different formats and relational models were not the ideal place to store and manage them.  Compute and storage medium were getting better, faster, and cheaper. The ability to scale up systems was becoming easier. The cloud was beginning to take shape, meaning the need to physically size and install very large systems in corporate data centers was being questioned.  Hadoop was bringing the concept of parallelism and MPP processing out to other platforms so activities that used to be done on Teradata, such as ETL, now made sense outside the warehouse.

Regardless, our vision stayed the same

With all this happening, Teradata made some significant, and at the time, innovative changes. Much of this started with our acquisition of Aster in 2012 and development of our Unified Data Architecture (UDA) in 2013, all this culminating with QueryGrid in 2014 which allowed us to run queries seamlessly between the Hadoop, Aster, and Teradata platforms. At the time, I was questioned about the “validity” of a centralized warehouse. My response was then, as it is now, our vision is the same and it continues to be more fully realized as technology allows. 

We have never said that Teradata is the only player in the environment, but that the players need to have cooperation and integration, otherwise you get anarchy.

Our vision drives our future

With all that historical perspective, it is fair to say our vision was not the co-location of data for storage’s sake but the bringing together of data for analytics’ sake. If that means that we need to cooperate with the other players in the environment, so be it. We cannot dictate to clients, nor should we. We need to leverage all the technology of the day to drive the highest value to the business.

As Paul Harvey used to say, “and now you know…the rest of the story.”  Signing off, I am Rob Armstrong.