Enabling an API factory to ensure integrations at scale

Enterprises are diversifying rapidly. A distributed landscape driven by remote working, platforms, ecosystems, innovations, digital transformations, and regulations is triggering the need for full lifecycle API management. Along with a full lifecycle API management, what is increasingly being observed is the need to build APIs at scale and drive efficiencies while doing so.

Gartner defines full lifecycle API management as an approach to manage the entire API lifecycle, which includes planning and design, implementation and testing, deployment and operation, and versioning and retirement.

There is a plethora of market offerings that do this; however, we will focus specifically on Microsoft Azure as the base integration platform with respect to all references, scenarios and examples in this article. We will also delve into the second aspect in detail, i.e. how to build APIs at scale and drive efficiencies in the process through a ‘Factory Approach’.

Why do we need a ‘Factory Approach’ for integration?

We have observed the following challenges in our interactions with our customers over the past couple of years:

1. Non-optimized approach to delivering integrations.
2. Lack of structured investments and vision for an integration strategy.
3. Inability to grasp the complexity of the hybrid enterprise landscape and future-proof it.
4. Legacy approaches to integration and lack of standardization and governance.

We can leverage this offering whenever we see economies of scale where integrations need to be enabled with consistency and predictability

To overcome these challenges, an end-to-end service offering was envisaged focusing exclusively on delivering integrations through a factory execution model, cross leveraging the core capabilities of Azure IPaaS. We call this service offering – ZiFaaS – Azure-powered Integration Factory As A Service.

The ‘Why’s’ and ‘How’s’ of ZiFaaS– Driving an API Factory

We have already discussed the ‘why’ with respect to the need to come up with a POV to optimize integration in enterprises – specifically the need to build APIs at scale in a factory model. We can leverage this offering whenever we see economies of scale where integrations need to be enabled with consistency and predictability backed by strong governance and assets – accelerators and templates underpinned by the capabilities of the modern and scalable Azure IPaaS platform. Overall, the intention is to drive operational efficiencies in addressing high volume of integration use cases. ZiFaaS offers solutions to these challenges and guidance to these scenarios by providing a broad-spectrum service catalog that details the approach of a factory execution model – the phases, tasks, activities etc. needed based on the Azure IPaaS reference architecture. The focus is also to roll out an efficient engagement model that pairs both the tech and ops aspects seamlessly. Let us delve into this in a little more detail.

Service Catalog and Capabilities

Figure 1 ZiFaaS Service Catalog

The ZiFaaS service catalog and capability snapshot detailed above tries to address some challenges such as non-optimized approach and lack of standardization in integration scenarios. This is done by providing clarity on consumers of the service, components in the integration ecosystem, technical capabilities and services to be leveraged and the overall governance wrapper that would make the engagement a success.

It recognizes three key types of consumers that would benefit out of it – internal consumers, citizen developers and the external ecosystem. The offering works in collaboration with customer’s existing CoE’s, architecture and design guidelines to strategize a roadmap for accelerated API development and rollout. From a technical capability standpoint, all the core areas are addressed. These areas include:

• API and application integration: Functional, service integration with orchestration and management of APIs, workflows, messages, queues, events, streams etc. on Azure IPaaS
• File-based integration: Batch, EDI, MFT etc.
• Data integration: ETL, Data models and data pipelines etc.

Cross-cutting competencies around compliance, security and DevSecOps along with organizational architectural principles, guidelines and SLA’s are also leveraged as part of ZiFaaS to ensure a seamless integration engagement while being agnostic and reusable stays close to the specific needs of a particular customer and is cognizant to GEO, region, country, and culture specific asks.

Approach to a Factory Service enabled by our accelerators

ZiFaaS comes with a host of micro capabilities that are enriched with HCLTech’s IPs and accelerators. A few keys ones are highlighted below along with examples of how they will benefit our customers.

1. Reusability and automation
   • This is achieved by interface reusability and by available low-code integration and connecters that can be used while enabling an integration program.
   • HCLTech’s IPs like ADvantage Code.NET and Sketch are utilized to enable this.
   • There are also templates, checklists, and other governance artefacts that ensure reusability and automation.
   • The aim is to improve the speed of execution while ensuring standardization, consistency, and repeatability.
2. Accelerated setup with optimized execution
   • Our discovery framework ensures that the disparate needs of integration in a hybrid enterprise landscape are properly assessed and addressed.
   • Templates, checklists, guidance documents, and governance artefacts ensure that learnings in previous engagements and best practices are applied even before day zero.
   • Automation is also baked into the execution as part of DevSecOps with IAC offerings and capabilities.
3. Tooling, security, testing at the heart of Ops
   • Execution guided by HCLTech’s core - flexi model.
   • Strong focus on testing harnessing the capabilities of our DevSecOps expertise for differentiated testing like A/B testing.
   • Automation in ops is enabled with HCLTech’s Kalibre, Academy and flexible TOM with our Converged Ops approach as part of HCLTech’s ASM 2.0.
   • Ensure adherence to industry and country regulations, industry standards and architectural principles while ensuring automation backed by HCLTech’s tools and IP’s – e.g., HCLTech’s AppScan, OneTest, RunAdvisor etc.

Execution methodology

From an execution standpoint, we consider there are two key aspects to apply ZiFaaS for our customers. This also addresses the known challenges around planning structured investments and envisioning an integration strategy.

1. How do we assess the requirements, needs, and use cases for integration? And what tasks and activities should we undertake to roll-out the API factory model?
2. How do we assess complexity to ensure that the effort planned is commensurate with the complexity of the task(s) at hand while also ensuring seamless implementation?

For the first, we have a phased approach with a detailed ETVX (Entry, Task, Verification, eXit) for every phase in ZiFaaS. These are:

We also initiate the platform, environment setup, detailed ways of working, RACI between Customer and HCLTech, define the tools and accelerators and the service catalog of how integration will be offered internally in a repeatable model. This phase ensures the readiness for the factory execution including Integration CoE setup etc.

1. Discovery – Typically comprising 6-10 weeks where we kick-off the engagement, identify and align stakeholders, perform analysis of the integration landscape, and come up with the roadmap, baseline delivery plan and rough order of magnitude (ROM) estimates.
2. Foundation – This phase follows the Discovery phase and usually consists of 8-16 weeks. We build on the work undertaken as part of Discovery and focus on defining the integration architecture and design including HLD (hi-level design) as needed.
3. Execution – The execution phase is agile-driven with repeatable sprints (defined as part of earlier phases) where APIs are developed in a DevSecOps lifecycle – continuous exploration, design, build and unit test; specialized testing including performance tests, SIT and UAT; deploy/publish/release; and finally support.

We pair the execution phase with a feedback loop. This ensures continuous improvements based on real-time feedback as the implementation progresses. There is also considerable intervention both technically – identifying and leveraging the right tools, assets, and accelerators we’ve talked about before and also on ops – coordination and management across product teams, Enterprise Architecture Board, platform and governance stakeholders, and business and services management groups to ensure change management, communication and stakeholder management is optimized.

The overall process is depicted in the snapshot below:

Figure 2 Approach to Integration Factory

Coming back to the second aspect with respect to assessing complexity, ZiFaaS offers a T-shirt based integration catalog that draws upon our expertise in integration engagements to define the parameters plus corresponding effort to build APIs and classify the complexity across Simple, Medium, and Complex categories. We also have guidance for the effort needed for support. It allows us to address a key challenge mentioned earlier - grasp the complexity of the hybrid enterprise landscape from day zero, plan for optimized analysis, and try to future-proof the enterprises’ integration asks. Unit pricing of APIs for integration is derived from this catalog, depending on the total volume of work.

The catalog is fine-tuned based on customer scenarios and use cases and heavily leveraged in the initial discovery and foundation phases. From an execution methodology standpoint, we have covered both the approach build APIs and the way to estimate and gauge complexity. The final area we manage and enable is around platform consumption – our team leveraging the ZiFaaS offering identifies the key Azure services that are needed in these engagements and ensures that the consumption and platform cost(s) are kept at a minimum while execution is happening in parallel.

Benefits of ZiFaaS

We envisage that our customers will be benefitted in the following ways:

1. ‘Integration as product’ approach to add speed to market with economies of scale.
2. Utilize full spectrum of integration capabilities end-to-end across Azure IPaaS.
3. Flexible and scalable operating model that can adapt to any managed, co-managed, unique/other service model.
4. Strong focus on governance with SLAs and KPIs that can be leveraged as part of the ‘offering definition’ or tailored to existing metrics in the organization.
5. Agile and DevSecOps-driven with focus on continuous improvements at every stage.
6. Harness HCLTech’s expertise across industry use cases where it has been leveraged already and delve into our capabilities enabled by our Microsoft partnership and managed execution experience (mentioned by analysts like IDC etc.).
7. Predictable catalog-based pricing.

Our execution experience using ZiFaaS

In this article, we have touched upon all the key aspects needed to ensure success for a factory execution model for integrations including the challenges, the use cases, the tasks, activities, and the technical foundation on Microsoft Azure on which such an engagement can be built upon.To conclude, I would like to share our execution experience with one of our key customers that went a long way in detailing and shaping up this offering.

• The customer in question is a Mining Anglo-Australian multinational and the world’s 2nd largest metal and mining corporation having over a dozen upstream and downstream systems requiring integration.
• In a bid to improving supply chain performance and ensuring beyond-par experience for the key end-user personas, it was imperative for the mining group to look at a ‘Connected Enterprise’.
• The ZiFaaS-enabled API factory model was rolled out to ensure an engagement that was API-led, cloud-neutral, scalable, sustainable, and efficient, incorporating automation at all levels.
• There was close alignment with the customer’s integration standards and enterprise architecture guidelines leveraging end-to-end cloud-native capabilities available as part of Azure PaaS. Specific authentication and integration patterns were assessed and utilized.
• The project was successfully rolled out in 2020 and is ongoing. It has now matured and operating in a sustainable way ensuring that all integration use cases across the organization are serviced through this program successfully.