Streaming Systems - A Way forward for Organizations and IT Systems of the Future | HCL Blogs

Signalytics Enterprise Event Hub

Signalytics Enterprise Event Hub
November 27, 2020

The current context of the global pandemic has brought forward the realities of rapid ecosystem changes and unpredictability beyond business strategies, planning, and management. No amount of planning can be as perfect as the variations in scenarios that nature and the limits of imagination can pose. Systems cannot be built based on a pre-configured ecosystem of actors and effects, but responsive fluid systems that can accommodate the unforeseen actors and ecosystem signals and help mend the ways to handle the effects/impacts can be a solution.

Intelligent Enterprise with an adaptable digital signal based framework for changing times.

The existing IT system landscape already captures the ecosystem signals with varying variety and velocity. The carburation of these signals is a determinant of value and responsiveness of the hyper-systems of the future. The rate of change in the ecosystems and signal patterns in a converged and intelligent hyper-system, requires adapting to the ecosystem of the signals through a real-time Signalytics solution.

The principle for these intelligent hyper-systems is based on composable living entities, continuously interacting, responding, and repurposing themselves, and guided by business objective inputs, to support thriving enterprises. The solution for this Signalytics capability is to enable a centralized streaming data ecosystem for an enterprise through an Enterprise Event Hub (EEH). This will be the brain behind the living and thriving enterprise [Also see]

The Enterprise Event Hub will be used as a platform to deliver the brain behind the living enterprise through data-intensive applications leveraging distributed, elastic, and adaptable models. Such a capability needs to have the following characteristics applicable to the entire platform, as well as for the individual components:

  • Reliable: The solution should behave consistently to the maximum extent in case of adverse events
  • Scalable: The solution should leverage containerized deployment, enabling transparent scaling
  • Extendable: The solution should be extendable to new types of actors, events, or signals
  • Composable: The solution should be modular to plug-in additional actors and processing capabilities
  • Flexible: The solution, as a whole or in part, should enable transparent moves across cloud/on-premise applications
  • Secure: The solution should provide the security mechanism for data in-transit, and data at rest

This kind of enterprise event hub platform to enable Signalytics solution intelligence should be devised based on the following principles:

  • Event is viewed as a time series data for both real-time and batch processing
  • A hybrid framework for streaming system and batch data-intensive application treatment
  • Effective storage for real-time events, and efficient long-time raw event archival

Conceptual Architecture– Enterprise Event Hub

The conceptual view of the EEH platform’s key concept is the separation of the analytical layer from the operational layer for event streams that are forked into different paths for isolating different signal batching/response patterns such as reservoir and lake.

Table 1: Key Reservoir and Lake Usage Considerations

Key Reservoir

Logical Architecture– Enterprise Event Hub

This section provides the logical architecture diagram of the EEH platform. The diagram depicts different source and consumer types along with different components/functions required in the different modules of EEH:

Key Reservoir

Figure 2: Logical Architecture View

We envisage that each module of the components such as sensors, neurons, memory, and the knowledge base, as part of the EEH platform, may be running on resources/clusters of its own. Considerations such as capacity, isolation level, orchestration, scaling, monitoring, management, and so on, should be defined during the detailed architecture finalization for the specific enterprise based on the events involved and intelligence expectations.

The final architecture would be recommended based on typical capabilities to support

  • Real-time operational event streaming system via a hybrid framework from first-, second-, and third-party sources
  • Event stream processing and correlations for event refinements
  • Containerized cloud-native microservices for automation and extension
  • Securing real-time event reservoir and data APIs for downstream consumption
  • Long-term data retention
  • Finding historical trends and pattern using BI tools
  • Data scientists to develop machine learning models
  • Horizontal robustness and vertical scalability
  • Meeting performance SLAs

Intelligent Adaptive Enterprises on Enterprise Signal Streaming Foundation

The target technology platforms and solution architecture should be configured to support the above aspects for enterprises based on the signals traversing the breadth and depth of the systems and services. The consolidation of all these signals into the knowledge base of enterprises with the required connectivity, edge, and central processing, and an active memory to support all the intelligence needs would be essential for enterprises in the future, in a constant flux, in terms of demand, supply, operating parameters, and ecosystem dependencies. The overall business objectives needs to be maintained within acceptable levels of variations, while striving always toward the North Star.