Introduction

Log analytics is a crucial area for customers to address application issues, application security issues, infra-security incidents and unwanted outages. Many customers operate in a hybrid environment, extending their on-premises network infrastructure to a cloud infrastructure provided by a Cloud Service Provider (CSP). Most customers need to store logs for a specific timeframe to meet regional and organizational compliance, but storing large-scale raw log data is also a challenge due to high storage costs. A centralized log management solution on AWS can overcome these challenges by allowing customers to collect, analyze and display logs in real-time and effectively manage storage.

While customers from financial services, especially, will reap the benefits of this solution, customers from all industries can leverage it to manage infrastructure and application logs efficiently and run analytics on it to detect anomalies.

Customers from financial services, especially, will reap the benefits of this solution, but customers from all industries can leverage it to efficiently manage infrastructure and application logs, as well as run analytics on it to detect anomalies.

Share  

The HCLTech solution for security log analytics

To address these demands and improve the log analytics process in an AWS environment, we have provisioned a solution we call the 'Centralized Log Analytics Pipeline.' This is built using a rich set of AWS services like Amazon OpenSearch Managed Cluster, Amazon OpenSearch Ingestion Pipeline, Amazon S3, Amazon SQS, Amazon CloudWatch and Amazon Kinesis Data Firehose Delivery Stream, along with one open-source software: FluentD.

1. The centralized log management solution uses FluentD (Open-source) to collect log data from various on-premises sources like servers, firewalls, web proxies and NIPS devices, as well as AWS Kinesis Data Firehose Delivery Stream to collect Amazon CloudWatch Logs from multiple accounts and regions.
2. All the raw logs from on-premises and cloud will be transferred to Amazon S3 in the respective log category folder. S3 Intelligent Tiering is enabled to manage the log storage.
3. An AWS OpenSearch Ingestion pipeline will be triggered by an S3 PUT event with the help of SQS to read, parse and process the log data. It also ingests processed data into indexes into Amazon OpenSearch Service, which contains a visualization tool called OpenSearch Dashboard that can perform analysis, visualization, dashboard, anomaly detection, alerting and reporting.
4. As an added feature, GenAI flavor can be added to this solution while sending the alerts to users and searching the logs from the customized interface. Anomalies, error strings from OpenSearch Service, can be sent to relevant administrators using SNS. Subsequently, admins can query for strings using REGEX, hostname, timestamp and more. Furthermore, to ease the querying using NLP and get relevant solutions for issues reported in the OpenSearch Dashboard, the event details being sent to the admin can be enriched with possible solutions by integrating internal corporate Knowledge/Solution repos (can be hosted on AWS) with MLOps using different AWS GenAI services, as well.

Key benefits

This solution was initially developed to address the challenges of one financial customer. But it can be leveraged for all customers focused in this area. The following are the key benefits of this solution:

1. Manage terabyte-scale data for storing the logs
2. Automated log collection
3. Provide an aggregator storage to store the raw logs
4. Centralized and secure log storage location for analysis and display
5. Re-ingest the data in case of any ingestion failure
6. Reduce infrastructure cost
7. Reduction of efforts in terms of future manageability

Architecture

HCLTech's log analytics solution is fully managed, automated, centralized and fault-tolerant. This solution can seamlessly store the logs, analyze logs, detect anomalies and send alerts. It uses AWS native services, making it easily configurable, customizable and deployable in any customer AWS cloud environment with easy upgrades and updates while reducing the overall TCO compared to other commercial software products.

Conclusion

HCLTech has rolled out a managed solution in AWS Marketplace, which will help customers build centralized, automated, cost-effective and fault-tolerant log management solutions quickly in their environment. To know more about the HCLTech Log Management solution (CLAP), inquire about the Proof of Concept, or discuss unique business needs, connect via AWS Marketplace Offering or write to us at AWSEcosystemBU@hcl.com.