Synergy of High Performance Computing and Artificial Intelligence | HCL Blogs

Synergy of High Performance Computing and Artificial Intelligence

Synergy of High Performance Computing and Artificial Intelligence
December 24, 2019

Synergy of High Performance Computing and Artificial Intelligence

In the recent past, data has seen a drastic increase both in volume and velocity due to the digitization of global business processes. According to a leading research firm, the volume of data created globally is expected to increase from 33 Zettabytes (ZB) in 2018 to 175 ZB by the year 2025. Data is now the new business currency. As organizations embark upon data-driven analysis/insights from simulation and modeling fueled by high performance computing (HPC) are the key requirements for any data-driven organization to gain a competitive advantage with their digital transformation initiatives.

Relationship between HPC and AI

HPC supports modeling and simulation to accelerate business growth and digitization in different fields ranging from Life sciences to financial services, energy and geo sciences, Manufacturing, Media and Entertainment and Retail. Artificial intelligence (AI) is considered to be the foundation for cognitive computing – computer models that mimic human cognitive functions to recognize trends, patterns, and associations in vast data streams to make decisions in real-time, and predict future outcomes.

To achieve success in AI, it is important to leverage enough computing power to propel real-time data analytics. Thus, AI workloads such as machine learning (ML) and Deep Learning (DL) are being built on the top of HPC infrastructure to deliver the speed and performance required for the discovery of precision medicine, weather predictions and climate modeling, fraud detection in financial services and arrive at better-informed decisions. More companies are using AI-powered HPC solutions to enable high-performance data analytics for training ML and DL models to gain insights from the ingress of digital data. Some of the key players in the HPC market are HPE, Dell, IBM, and Lenovo.

Leverage high bandwidth interconnection, low latency compute with HPC to improve the performance of AI workloads.

Today, HPC and AI programs can no longer be considered separate domains, when it comes to digitization. These complementary technologies are swiftly converging so that organizations can gain greater value from the data that is being captured and stored. This synergy between HPC and AI is based on the common element i.e. data. While huge amounts of data and insights are generated by HPC simulations, at the same time, AI requires tons of data to train the models. Therefore, the effectiveness of HPC simulations can be improved if we use AI programs to identify the parameters that can significantly impact the simulations and then the following iterations of the simulations can be based on those parameters.

The convergence of AI and HPC seems inescapable at an infrastructure level because both the workloads require high computing power, high-speed interconnect networks and a huge amount of storage.

The merging of HPC and AI is reflected in HPC’s growth projection as well. In line with the forecast of a renowned research company, the global HPC server-based AI market is expected to expand at a CAGR of 29.5% to reach more than $1.26 billion in 2021, up by more than three-fold from $346 million in 2016.

The intersection of AI and HPC is occuring because of the following trends:

  • With data growing at an exponential rate, DL techniques are hitting a performance bottleneck.
  • Scientific computation and simulation models which are considered the core application areas for HPC are making use of AI algorithms to make sense of the data and transform it into more human-friendly formats.
  • HPC brings in high bandwidth interconnection, low latency compute and can be leveraged to improve the performance of AI workloads.

Below are some of the prominent use cases where AI augments HPC workloads:

  1. Precision Medicine for Cancer Treatments

    Precision medicine known as personalized medicine technique is now being used by doctors as a tool to customize cancer treatment for patients based on genetic understanding, and the type of cancer affecting them.

    HPC's adoption in cancer research is growing as it not only helps increase compute capability but also integrating patient data from multiple databases, improving memory bandwidth, use of heterogeneous hardware, and improving S/W efficiency. Through HPC researchers are simulating protein interactions to understand cancer development. AI techniques can be leveraged here to complement HPC and provide more efficient simulations while improving productivity through automated information extraction and analysis of patient records.

  2. Weather Prediction and Climate Modelling

    Data that is used in weather prediction systems come from several sources such as land, ocean sensors, air, radar, satellite, and data for climate modeling realized upon historical observation. In both cases, a huge amount of data is generated. Due to volatile climatic conditions, it is very important to have improved prediction capabilities.

    Data that is used in weather prediction systems come from several sources such as land, ocean sensors, air, radar, satellite, and data for climate modeling realized upon historical observation. In both cases, a huge amount of data is generated. Due to volatile climatic conditions, it is very important to have improved prediction capabilities.

    HPC is widely used in weather prediction and climate modeling to help meteorologists and scientists put mountains of data to work, improve the accuracy of weather forecasts, and simulate weather patterns. AI platforms, along with HPC can be leveraged in enhancing weather applications to create early warning signals for catastrophic weather events and obtain refined and accurate weather forecasts. As an example, Numerical Weather Prediction analyses data from various sources provide short-term weather forecasts for rainfall, snowfall or extreme weather events, and long-term climatic prediction based on carbon emission scenarios.

  3. Autonomous Vehicles (AV)

    Autonomous vehicles (AV), known as self-driving cars or driverless cars are vehicles that can guide themselves with little or no human intervention.

    These vehicles make use of various technologies to perceive their surroundings such as radar, lidar, sonar, odometry, cameras, GPS, storage systems, and HPC networks. Continuous streams of data are captured from their surroundings and HPC analyzes this data to ensure safety and efficiency.

    Road conditions can never be predicted accurately, and pre-programmed responses will not be able to accommodate the unexpected. It is very important for AV to continuously learn and make favorable decisions under all circumstances. Therefore, HPC-powered AI solutions can be used in AV as HPC is capable of handling petabytes of data generated by such vehicles. At the same time, AI programs will ensure smart decision-making and safety of the person.

  4. Fraud Detection in Financial Service Institutions

    HPC-driven AI systems are used in financial service institutions (FSI) in the following ways:

    • Customer Engagement: FSI’s use AI-driven systems to bring themselves closer to the customer. Chatbots and voice analytics tools are being used to manage customer interactions. Intelligent systems are being used to optimize the marketing of products and services. Data from sources such as transaction history and social media sentiments are now being analyzed to anticipate customer needs and provide tailored recommendations.
    • Credit Risk Assessment: FSIs are increasingly using AI systems in mortgage and retail banking to accelerate the credit risk assessment and make informed decisions on the creditworthiness of loan applicants. This is helping them reduce losses associated with defaults on loan payments.
    • Fraud Detection: Fraud detection is a key requirement in the financial sector. AI algorithms are helping banks identify unusual and fraudulent transactions. HPC-powered ML algorithms are being used by master cards to detect and stop fraudulent payment card transactions.
    • Regulatory Compliance - By Leveraging AI, FSIs can automate the process of identifying, collating, and analyzing data from diverse systems to meet compliance requirements and reduce the possibility of human errors that might lead to non-compliance.
  5. Cybersecurity: Real-Time Cyber Threat Analysis

    The volume and complexity of cyber threats have seen a sharp increase. Traditional cybersecurity solutions are not adequate for emerging threats like advanced persistent threats, botnets, and zero-day vulnerabilities. Cybersecurity vendors are adopting approaches such as behavioral analytics, server logs, and VPN logs which can then be fed into AI algorithms to identify suspicious patterns.

Benefits Arising from the Confluence of HPC and AI

Organizations are now looking for IT systems that have workload-optimized architecture and can process different types of workloads ranging from HPC to AI. By combining HPC with AI, organizations can take advantage of high-bandwidth compute along with state-of-the-art neural networks to improve the efficiency of cyber threat analysis.

The convergence of AI and HPC workloads is an important transmogrification for enterprises and institutions that are looking for a comprehensive infrastructure solution in their digital transformation journey.

By running HPC and AI workloads in a single cluster environment, an organization will achieve double the cost benefit. From the perspective of capital expenditure, it will reduce the need for making new investments while gaining maximum value from the existing cluster environment. From the perspective of operating expenditure, the cost of running and maintaining infrastructure will be reduced by running the workloads on one cluster rather on multiple clusters.

The convergence of HPC and AI platforms will help organizations hold the full spectrum of computing capabilities that will be needed to support emerging business needs. It will also help accelerate scientific discoveries, shorten the time-to-market of new products, and produce real-time business insights on the road to digital transformation.