December 14, 2016


The migration at work– from requirements to use case-driven data gathering

Traditionally, BI’s functions involved gathering requirements from stakeholders, identifying attributes that would address the needs identified, and modeling data accordingly. However, from source applications, only a portion of data is transferred to the data warehouse and is transformed for reporting purposes. It is also established that data transformations negate raw data’s impact on analytics. As a result, the traditional approach of using partial data for reporting must be scrutinized.

The overall cost of storing data is now largely rendered irrelevant with Big Data emerging as a revolutionary force. Professionals now recommend that data be obtained from source applications. This results in uploading source data to the data lake. Since stakeholders need rapid and effective decision-making capabilities as well as report generation, operational reports and analytics has steadily gained importance.

Traditional data warehouses/data marts offer operational reports in an attempt to action a transition from BI to Data Lake. In the Big Data environment, stakeholders and IT professionals are endeavoring to glean new approaches to traditional techniques along with consistent collaboration to effectively address the analytics arena.

It is generally known that a function comprises a set of sub-functions, and a sub-function consists of a set of processes. A set of functions makes a business enterprise. Likewise, a user story is made of a set of use cases and each use case includes a set of requirements.

For instance, supply chain has sub-functions like vendor management, demand planning and order management. More than one application can apply to a sub-function, implying data from multiple sources must be cleansed and combined for analytics.

A use case can relate to product life-cycle tracking; it will touch multiple sub-functions in the supply chain space, necessitating data integration (not aggregation) from multiple applications. The use case scenario methodology has also undergone a transformation –

  • All source data relating to a use case is uploaded to the data lake
  • The data model is logical and not necessarily physical
  • Reliance on integration, instead of aggregation
  • Creation of transformational libraries to address data at the lowest level of granularity
  • Data access is pre-dominant over reporting

It’s also important to note that at the integration layer, not all data attributes may be captured and child tables will be filled up for those who need data.

When viewed holistically, an organization is fundamentally engineering a paradigm shift: journeying from reporting requirements to use cases to user stories, moving towards a more integrated environment.