It is now quite common for the modern-day software systems to churn a massive amount of data and synthesize it internally to produce meaningful insights using AI. Such systems consist of software and hardware components that manifest into many products & services we use every day. It could be a cab booking application, online shopping, movie streaming, social networking, infrastructure monitoring, etc. One common characteristic among all these applications is that these are massively parallel, responsive, highly available, and resilient, and these applications provide key insights that help us to make decisions on time. Architecting these systems is always an interesting and challenging exercise. This paper is an exploration of a few learnings from developing such applications.

The example discussed in the paper strives to build a highly scalable, distributed yet minimalistic system. Though the example is specific to ingesting video streams of data, the architecture components and principles may be reused for any other data ingestion system. The paradigm shift from architecting traditional monolith system to microservices based distributed systems creates many opportunities, if done correctly. This involves enabling new business models like SaaS and Pay-per-Use, as well as operational efficiencies like ability to change and deploy individual microservices without taking down the entire system. This paradigm shift from monolith to distributed architecture poses new challenges as well. The paper aims to discuss addressing these challenges based on learnings and some of the best practices. The example discussed in this paper is available as a working code licensed under MIT Open Source by the same authors.

DOWNLOAD THE WHITEPAPER

It is now quite common for the modern-day software systems to churn a massive amount of data and synthesize it internally to produce meaningful insights using AI. Such systems consist of software and hardware components that manifest into many products & services we use every day. It could be a cab booking application, online shopping, movie streaming, social networking, infrastructure monitoring, etc. One common characteristic among all these applications is that these are massively parallel, responsive, highly available, and resilient, and these applications provide key insights that help us to make decisions on time. Architecting these systems is always an interesting and challenging exercise. This paper is an exploration of a few learnings from developing such applications.

The example discussed in the paper strives to build a highly scalable, distributed yet minimalistic system. Though the example is specific to ingesting video streams of data, the architecture components and principles may be reused for any other data ingestion system. The paradigm shift from architecting traditional monolith system to microservices based distributed systems creates many opportunities, if done correctly. This involves enabling new business models like SaaS and Pay-per-Use, as well as operational efficiencies like ability to change and deploy individual microservices without taking down the entire system. This paradigm shift from monolith to distributed architecture poses new challenges as well. The paper aims to discuss addressing these challenges based on learnings and some of the best practices. The example discussed in this paper is available as a working code licensed under MIT Open Source by the same authors.