Co-authored by Chinmay kale .
According to the 2015 survey by IDG Enterprise, over the past year, there has been a 125% increase in the number of organizations who have been leveraging Big Data by deploying data – driven projects. There has been a significant reduction in the skepticism regarding Big Data projects over the last one year. IDC research indicates that 60% of the organizations have started using new analytic techniques, 33% have started using new data types and 32% have introduced new KPI’s. The same IDC research also shows that 70% users are complaining about the Big Data technology resources available to them to support analysis and decision making. These data points indicate that organizations definitely believe in the Big Data potential. However, they are facing challenges in executing on their ‘Big Data’ vision.
Fig. 1: Benefits of Big Data
Analysts say by 2020, there will be a 4300% increase in annual data generation. Irrespective of the accuracy of this estimate, the critical question is, how organizations will make sense of this enormous data pool (which includes transactional data on its products and services, market feedback, social inputs, product lifecycle data, supply chain transactions, employee engagement and performance, financial performance of the business units and the organization itself, and so much more). Organizations have always been capturing and analyzing data from its systems, so what’s new? Big Data technologies allow you to pool data from a much larger set of structured and unstructured data sources and re-organize it for better analysis. Big Data technologies allow you to analyze and correlate cross functional information to derive previously unknown insights which enable the organization improve its internal efficiency and market effectiveness. Organizations are likely to discover correlations between employee engagement and product quality OR better correlate product upgrades to consumer behavior. The possibilities are endless and exciting. However the road is far from easy. Embarking on this change without careful evaluation of goals and roadmap, is likely to prove very costly for the organization.
Typical Challenges in Big Data Adoption
- Lack of strategy: Research shows that most companies often embark on Big Data adoption without a clear view of their goals and strategy. Most of them try to do “everything” because of the fear of missing out on opportunities.
- Choosing Data Management frameworks and Analytic Tools: Organizations are not able to envisage solutions which would be scalable and re-usable. Today there is a plethora of tools with diverse capabilities and with their own set of promises. Poor tool and platform selection in the context of the organization leads to unnecessary costs and even leads to dropping of Big Data initiatives.
- Insufficient knowledge of disparate data sources:Besides structured data, big data technologies can help analyze various unstructured data formats like audio, video, text logs, social media streams and web streams. However the organization would need to plan diligently and even need to revamp its data management processes to successfully integrate disparate data streams across channels (legacy to upcoming).
- Big Data Talent Gap: Though there is a huge supply of application engineers, other skilled resources like Big Data architects, data scientists etc. seems to be much less than the demand. Building in-house talent involves a steep learning curve and is a longer term exercise. This gap poses a serious impediment to the initiation of Big Data adoption.
How can HCL ERS help in your BIG DATA journey?
Fig. 2: Big Data Maturity Assessment Model
HCL’s Big Data Center of Excellence can help you overcome the typical and other challenges in your Big Data journey. HCL ERS uses a simple and intuitive assessment to determine the big data maturity of the client and articulates the most appropriate course of action thus helping you to leverage the Big Data potential. Depending on your maturity with big data, HCL ERS can help you with identification of use cases to experiment with Big Data, creation of data lakes, creation and deployment of Hadoop data management platforms, and develop analytic applications.
Reach out to HCL ERS Big Data Center of Excellence at Contact-BDA@hcl.com to know more about how we can accelerate your big data journey.
- IDC Whitepaper: Six patterns of Big Data Analytics Adoption