The retail sector has been at the forefront of innovation with data underpinning several exciting new use cases. Every day, new data-driven approaches are emerging that are enabling businesses to unlock hidden insights, take actionable and predictive decisions and fuel business growth.
Organizations have been managing data on premises for a long time and they are now in the process of modernizing their data management initiatives by shifting to the Cloud. However, modern enterprises have reached a point now where the quantity, velocity and variety of data available to them can no longer be managed without making significant changes in the fundamental infrastructure and the conceptual model they use to govern it.&
According to Gartner, DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. Similar to DevOps, the goal of DataOps is to deliver value faster by creating a predictable analytics delivery model . DataOps uses technology to automate the design, deployment, testing, management and monitoring of data delivery pipelines with appropriate levels of governance. DataOps breaks down silos and closely connects Data Engineers, Data Analysts, Data Stewards, Operations SMEs, Data Scientists, Citizen Developers and business users to accelerate business decision.
Most of the organizations have majority of their data stored on-premises with either a part of it being migrated to cloud or all of it in the transit for cloud migration in the near future.So how can DataOps help these organizations strengthen their Data operations?
Real World Example
Let us explore a recent case study.
HCLTech has recently been engaged with one of UK’s leading retailers, a British super market chain, to assess its current state of data operations and recommend the best approach to achieve productivity improvement and operational efficiency. This organization’s data operations include multiple BI tools, ETL pipelines, Data warehouses/Data Marts, Cubes and Databases, to name a few.
To assess the current state of the data operations and how this can be tuned to achieve desired outcomes, HCLTech adopted a three-dimensional approach around People, Process & Technology.
The assessment started with workshops and meetings with various data stakeholders, to develop a detailed understanding of the existing processes and pain points. HCLTech analyzed previous year’s incidents from Service Now, to delve deeper into the issues and determine the root causes.
Recommendations:
Most of the assessment findings were from technology and ops process dimensions with a touch of people dimension in each finding and plenty of opportunities to automate. They are as follows:
- Automate Deployment: Automate integration and deployment of on-prem scripts, pipelines, enhancements, and reports.
- Agile: Introduce Kanban and streamline multiple deployment directly from Repos. (Database/EDW enhancements, BI enhancements, Pipeline fixes etc)
- Proactively fix Data issues: Build DQ framework to proactively fix upstream source data quality issues.
- Upskill and cross train team: Upskill team across technologies, to address incidents across Ingestion, Cubes and Reporting to ensure quick resolution. This helped to meet SLA and also improve on that to ensure early availability of data to business to unearth hidden insights.
- Continuous Data Testing: Automate reconciliation steps both at the fag-end of ingestion pipelines and EDW load pipelines. This step provided automated testing, which is dynamic in nature to ensure data is correct both from numbers perspective and business context, e.g. Sales figures in Christmas week this year compared against the sales figures from the previous year.
- Holistic Monitoring: Automate monitoring of BI Servers (Tableau/Power BI etc) using Accelerators.
- Intelligent Alerting & Monitoring: Automate thousands of alerts generated every day and prioritise for faster remediation.
HCLTech integrated its proprietary accelerators for Agile Project Management, DQ automation, Continuous Testing, Holistic Monitoring, and Intelligent alerting to boost operational efficiency and save time for faster delivery of data and insights.
This DataOps initiative helped the Retail giant gain significant productivity improvement while at the same time cut time and cost to assure data quality throughout the delivery pipeline. Be it DevOps or traditional data Operations, DataOps extends efficiencies delivered by the Ops team.
Unlocking the value of data has an undeniable business impact — to do it right you’re going to have to start thinking seriously about DataOps and leveraging this would surely bring in incremental business value.
