- More Storage Less power: Now the issue is space and power usage. For any large-scale enterprise, scalability is a big factor. Especially when it is exabytes of data. Each rack in Facebook’s datacenter uses just 2 kilowatts of power instead of the 8 kilowatts that any traditional storage rack uses. But it is able to store 8 times the volume of data stored by standard racks. How does it manage this? The hardware itself is not radically different, but it uses a technology called shingled magnetic recording that features partial overlapping of contiguous tracks, thereby squeezing more data tracks per inch. The system is architected such that different “chunks” of data don’t share same power supply or top-of-rack switch so as to avoid a single point of failure that would cause loss of data. And if a user deletes data, it is deleted from cold storage as well. The overlapping-tracks architecture may slow down the writing process since writing to one track overwrites adjacent tracks and requires them to be rewritten as well. Device-managed SMR devices hide this complexity by managing it in the firmware, presenting an interface like any other hard disk, while other SMR devices are host-managed and depend on the operating system to know how to handle the drive, writing only sequentially to certain regions of the drive.
- Raid in large Hadoop Distributed File System (HDFS): The social network giant implemented the technology, which includes RAID in HDFS, to reduce the replication factor of data in HDFS. RAID is a way of storing the same data in different places (thus, redundantly) on multiple hard disks. HDFS is the primary storage system used by Hadoop applications. It provides high-performance access to data across Hadoop clusters and has become a key tool in enterprises for managing big data and supporting big data analytics. The default replication of a file in a HDFS is three, which can lead to a lot of space overhead according to Facebook’s engineering team. HDFS Raid technology helped the company cut data replication and effectively reduce this space overhead.
- Erasure Coding replacing RAID: Erasure coding (EC) is a method of data protection in which data is broken into fragments, expanded and encoded with redundant data pieces and stored across a set of different locations or storage media. Although it seems similar to RAID, it is entirely different in functionality. The goal of erasure coding is to enable reconstruction of data that becomes corrupted at some point in the disk storage process by using information about the data that's stored elsewhere in the array or in a different location all together. Erasure coding creates a mathematical function to describe a set of numbers which can be checked for accuracy and recovered if one is lost. Referred to as polynomial interpolation or oversampling, this is the key concept behind erasure codes. In mathematical terms, the protection offered by erasure coding can be represented in simple form by the following equation: n = k + m. The variable “k” is the original amount of data or symbols. The variable “m” stands for the extra or redundant symbols that are added to provide protection from failures. The variable “n” is the total number of symbols created after the erasure coding process. For instance, in a 10 of 16 configurations, or EC 10/16, six extra symbols (m) would be added to the 10 base symbols (k). The 16 data fragments (n) would be spread across 16 drives, nodes or geographic locations. The original file can be reconstructed from 10 verified fragments.
Today we see some of these technologies that tackle issues associated with huge amount of data already being adopted by enterprises. The enterprises are relying on one of the three possible ways to do so: advance storage offerings, open source developments or custom software. Each of these emerging data storage technologies has both a long-term as well as a short term vision. Software-defined storage and scale-out storage come with an immediate solution to data growth problems faced by most enterprises. Software-defined storage solutions are today positioned quite well in the industry, but still have limited functionality in a couple of major areas.
On the other hand, technologies such as Erasure coding have short-term use case for storing passive data, and is definitely something that can easily replace the RAID technology.
The whole idea here is to be proactive in adoption of these technologies because the most important innovations take place when people solve problems before they escalate in magnitude.