Learn more about the origins of Big Data and how you can maximize it for your business.
Overview
At the height of the tech boom, the opportunity of Big Data computing seemed like a data scientist's dream. But with inadequate tools and complex systems, Big Data quickly became an IT nightmare. Teradata entered the industry with solutions that continuously deliver answers at scale.
One of the earliest mentions of “Big Data” came at the height of the tech boom, with Internet and connectivity expanding along with data processing and storage capabilities. Computer scientists wrote: “Just as search engines have transformed how we access information, other forms of big-data computing can and will transform the activities of companies, scientific researchers, medical practitioners, and our nation’s defense and intelligence operations…Big-data computing is perhaps the biggest innovation in computing in the last decade.”
Big Data's promise was its ability to capture and keep all of an enterprise's data on an unprecedented scale. Enterprises could keep data they may previously have thrown away in order to better understand customer behavior, make predictions about market and environmental outcomes, and much more. Analysts projected that Big Data would generate $50 billion or more in total revenue.
However, after embracing platforms such as Apache Hadoop to perform batch processing of large amounts of data and execute ETL (Extract, Transform, and Load) jobs, the enterprise encountered hurdles. Tools like Hadoop didn’t adequately support traditional analytics required to run day-to-day operations. No SQL database and object storage providers emerged to fill this storage and management gap and to provide agile platforms for real-time, geospatial, and other analytics use cases. As an expert writes, “Hadoop never became fast enough to truly replace the data warehouse.”
Another challenge came with the rise of mobile and the Internet of Things (IoT) as enterprises now had to support a wider variety of data sources. Soon Big Data became a foregone conclusion — business leaders now sought flexibility in analytics and platforms in order to understand the context of data across clouds and across sources. Along with capturing and storing massive amounts of data, IT leaders now looked for applications, platforms, and cloud infrastructure vendors that would make data analytics, integration, and replication more agile and rapid.
Meeting the enterprise’s demand for Big Data processing was a key principle behind Teradata’s founding. In fact, the company’s name comes from our capability to manage terabytes of data. (Today that number has grown to many petabytes of data!). Back then, storing and analyzing large volumes of data was limited by the hardware available at the time. We created the Teradata Database to effectively manage data at scale to enable enterprise analytics, solving the technical issues and demonstrating the power of integrating data in a relationship model.
Today, as the data warehouse runs tens of millions of queries per day and supports mission-critical operations, many enterprises have become more protective of their Big Data. The enterprise often restricts access that prevents data scientists and business analysts from seamlessly running reports and gleaning insights. This often causes these users to create their own workarounds using their own tools, resulting in analytics silos.
To integrate Big Data analytics, Teradata Vantage expands the types of analytic engines within the data environment, allowing them to be used across a variety of use cases. Rather than constantly replicating the functions, Vantage brings them together into a logically central architecture where they can be called upon when needed. Teradata extended from just having a SQL Analytic Engine to include Machine Learning and Graph engines — and that’s just to begin with. All types of programmers can now reference the engine and function they need, minimizing time and data movement and improving consistency and performance.
Vantage also gives the engines access to much more diverse sets of data, and we provide the necessary connectivity across data storage environments. This accommodates a range of data needs, from data that’s used repeatedly by many groups and requires structure and strict governance, along with data that just needs to be quickly stored, with unknown or variable structure, and accessed by a limited set of users.
Finally, we understand the downsides of limiting the tools that users can work with when it comes to data science, so we abstracted tools and languages so that any programmer can write in the tool of their choice. This drives more insights faster and at lower cost as analysis becomes accessible to more users.
Big Data has played a crucial role in digital technology for decades. But its magic doesn’t come from its size — Big Data’s power comes from its ability to help the most people get to the right answers, faster. From Teradata’s earliest days, we worked to integrate Big Data technology, knowing that this was key to increasing business understanding and actionable insights. Today we’re continuing that legacy with Vantage, integrating Big Data analytics to ensure that the enterprise can get to answers faster and remain agile in driving business success.