If you listen to the evangelical proponents of NoSQL, Hadoop (HDFS) centric, columnar and semantic database management systems, you would be convinced that we are at the dawn of a new age in computing that breaks the bonds of traditional relational database management systems (RDBMS). The question is, are we?
For those of us that have been around a while, we can see some old patterns emerging. There was a time before when object oriented, multi-value, XML and knowledge databases promised to dethrone the relational database king, but alas they did not. Is history repeating itself? Will IBM, Oracle, Microsoft and Teradata simply gobble up the upstarts, extend their functionality, and crush this new wave?
Before I answer that question, let's also reach way back and remember that there was once a time when RDBMSs were the challenger. In those days IBM's Virtual Storage Access Mechanism (VSAM) and the IDMS (CODASYL network database) dominated after the age of simple indexed sequential files. People scoffed at that time, but change does happen. The point is - it's not an immutable law of nature that RDMSs remain entrenched forever. Is now that time?
I believe the answer is yes. This is not a fading, hollow trend and we are living through but a legitimate inflection point where the market is changing and will not revert. I am going to qualify my opinion by pointing out that this is not a zero sum game. Put another way, RDBMSs are not going away (just like VSAM has also never gone away) but as the demand and volume of data expands, new more specialized data management capabilities will be used to satisfy a disproportionate percentage of that new demand.
The primary differences between previous fads and now are: economy of scale, legitimate new demand and developer mind share.
Economy of Scale. This is an opaque reference to the 'trifecta' of open source community, cloud (backed by big players Amazon and Google) and the support centric cost model as opposed to the older software license and maintenance model used by RDBMS vendors. Cash strapped companies now have legitimate options for lowering their operational spend without risk (or at least with controllable risk) and with proven support. This is a combination of options that simply did not exist in the past and has changed the business and consumption model.
Legitimate New Demand. I will not repeat what you have heard so many times before and know to be true, but it's simply a fact that data volume, variety and the velocity is increasing exponentially. When RDMSs were challenged in the past, this dynamic did not exist and it was largely a question of ripping and replacing the incumbent with a new mouse trap that would, for the most part, provide the same end result. The business case was thin and based on cost savings, indirect speed to market and innovation advantages. The challenge today is obvious. You have to go no further than your own personal devices and apps to understand the desire to have that same capability in the context of the corporate world.
Developer Mindshare. All great technology shifts, require the development community to be completely bought in and for there to be viral waves of continuous innovation. That was the spark behind RDBMS, client/server, web, apps, mobile and is now prevalent in the data space. In fact, if you look where the energy and excitement is amongst young developers, it's all behind non-RDBMS data technologies and how web, mobile and social technology can be integrated to create new applications. Of course, there is a downside when one generation tends to forget or not learn required lessons about corporate computing, but it’s undeniable that where their energy flows - the market heads.
Even though we are at an inflection point, progress is not a straight line. There will still be winners and losers within the mega trend, but when we look back in ten years’ time, we will recognize the birth of a new landscape of data management capabilities and architectures that started in the times that we live in now.