Type to SearchView Tags

Leveraging Data Warehouse on Cloud to Scale Enterprise Insights
Kamal Anand Senior Enterprise Architect, Digital & Analytics | December 19, 2019
440 Views

In the current economic landscape, data is the new oil. This is true, both metaphorically, and quite literally, when taken in the context of global enterprises that base their numerous daily decisions on a data-led premise. With a massive upsurge in both the amount of data being created, as well as the business needs that use that data, organizations increasingly find themselves looking to scale their analytical capabilities and enable their staff to analyze data in real time.

With cloud-built DWH technology, more information can be instantly scaled to support multiple workloads.

Traditional enterprise data warehouses (EDWs) are not capable of fulfilling all these needs.

Production systems are transactional and require databases that can store, write, update, and delete information. The systems that make all of this possible are online transaction processing databases. In addition to storing information, they also provide an insight into what’s happening to the data based on the information in online transactional processing (OLTP) databases. Organizations need to know not only the knowledge of the earned revenues, but also where this revenue is exactly coming from, this entails knowing profiles of the customers making these purchases, business trends, and transactions that are taking place. Data-savvy businesses need to know how to ensure customer loyalty and retention along with growth. Insights into all these questions are necessary to plan a strategy and develop new products that will help in a growing business.

Cloud data warehouses outshine their on-premise counterparts in terms of speed, security, and ease of use.

Acquiring these insights requires a lot of accumulation, computing, and analysis of data from OLTP databases. Aggregation of all this data leads to the creation of large sets for analytics. However, OLTP systems are not optimized to analyze of large datasets. This has resulted in the emergence of data warehousing solutions.

Data warehouses (DWHs) will hold a copy of the data stored in OLTP databases along with a large amount of data accessed by organizations including internet data, cloud-born data, and machine-generated data related to IoT. Also, the cost of maintaining these legacy systems continue to increase along with mushrooming scalability challenges faced by organizations. This is one of the key reasons behind a movement towards cloud-based technologies that can be leveraged to effectively manage operational costs, along with providing advanced capabilities for data analytics.

By facilitating the adoption of Cloud to existing EDW platforms, companies can allow greater flexibility, more agile growth, and better costs for different workloads and business needs. DWHs currently deployed in most organizations are traditional, which were developed decades ago and were built for on-premise data centers. These still exist alongside the current generation of data lakes based on Hadoop (big data). But neither of them can elastically scale-up, scale down, or remain suspended per the business needs to catch up with the continuously varying demands of today’s enterprises. Due to this, a lot of attention and focus needs to be diverted towards low-level infrastructure tasks that are not helping IT and data science teams as their focus should be on analytical projects that will give insights, and will help in growing the business.

With modern, cloud-built DWH technology, more information can be gathered from multiple data sources and can be instantly scaled to support multiple users and workloads. All of this needs to be accomplished with the assurance of integrity and consistency of a single source of truth without having to worry about computing resources. Data warehouse as a service (DWaaS) is generating a lot of traction these days. It is an outsourcing model in which a service provider configures and manages the hardware and software resources as the data warehouse requires, and the customer just has to provide the data and pay for the managed service.

The key reasons for leveraging cloud-based technologies for existing DWH platforms are:

  • Elimination of capital costs
  • Elasticity and scalability
  • Renewal of license costs
  • Using PaaS to reduce operation overheads
  • Advance data analytics capabilities

The good part is that technology has evolved to address the demands of data-driven insights with innovations available in the market such as:

  • Cloud
  • Columnar Storage
  • MPP
  • SSDs
  • Vectorized processing

Cloud data warehousing is a cost-effective way for organizations to take advantage of these latest technologies, innovations, and architecture without a high upfront cost, installation concerns, and the configuration of hardware, software, and infrastructure. Cloud data warehouses outshine their on-premise counterparts in terms of speed, reliability, security, and ease of use. This allows users to modernize their processes at the speed of technology and make it simple for the entire organization to access data in real time.

Moving EDWs to the cloud involves important design and migration considerations as these are important to ensure that current system functionality moved to the cloud is seamlessly integrated with workloads that are on-premise with minimal downtime and are flawless with no impact to actual users.

Some of these considerations are:

  • Data integration and access
  • Team’s existing skills, tools, expertise, and experience
  • Cost and speed
  • Meet current and future needs
  • Data security
  • Data resilience and recovery
  • Reliability

A few use cases that organizations are considering and its corresponding cloud data warehouse benefits are outlined below:

 

  • Ad-hoc Analysis: - Users can join, aggregate, and scan data in whatever kind of table they want; built-in statistical functions make it easy to build queries and disk usage is optimized by using a column store table format.
  • Machine Learning and Data Science: - Large variety of data formats can be used; rapid sandbox configuration allows for quick experimentation, even as load requirements change; inter-operational data preparation and statistical tooling.
  • Real-time and Operational Analytics: - Data can be queried in real time, even as events happen; higher availability and fewer outages; easy to enrich and de-dupe data; processes repeat queries quickly.
  • Application Backend Data store: - Storage and compute can be scaled-up or down as usage fluctuates absorbing spikes in usage while also keeping costs down; results in data integrity, security, and availability.
  • Mixed Workload Analytics: - A single source of data for any query or use case that an organization may need; supports a broad range of queries without requiring additional hardware or complicated data configurations; secures sensitive data; supports broad data ingestion, allowing streaming and batch load data inputs.

Some of the companies that are providing DWaaS cloud solutions are:

  • Amazon Redshift: is a fast, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using SQL and your existing business intelligence tools
  • Google Bigquery: is a fully managed, powerful Big Data analytics platform that enables super-fast SQL queries using the processing power of Google's infrastructure
  • Snowflake: is a fully managed data warehouse built for the cloud, for structured and semi-structured data
  • Yellowbrick: is based on the Yellowbrick architecture for native flash queries, unlocking the true speed of flash memory to power analytics in the hybrid cloud
  • Actian Avalanche: is a fully managed, third-generation, cloud data warehouse that delivers industry-leading performance, simplicity, scalability, and savings

Soon, we can expect enterprises to turn to these companies to help propel insights from data in new directions and at new speeds, regardless of the size of the business or the industry in which it competes.