DiskReduce by B.Fan et al. was published in 2010 [8]. In the paper, the aim is to reduce disk space based on Redundant Array of Independent Disks (RAID) data storage scheme. DiskRe-duce is a kind of network RAID. Due to the performance concern, DiskReDiskRe-duce executes the data encoding in the background. The redundant blocks are not deleted during an encoding phase until the encoding phase completes. Thus, the available space in the system will grow after encoding process.
3.3.1 Encoding Type
The study proposes two prototypes: RAID5 and RAID6. Each scheme has different capacity of reducing the space. By default, three copies are used in the system.
Figure 3.2: The RAID5 and RAID6 Scheme in DiskReduce
RAID5
This scheme maintains a fully mirror version and a RAID5 parity set. The parity set is required to recover the data if all mirrored copies of the block are not available. In a nutshell, one of the copies of the block is encoded as a parity set. The reduced storage overhead is 1+1/N, where N indicates how many blocks are calculated as a parity set.
RAID6
The difference between RAID5 and RAID6 is the number of full backups of blocks. The feature of RAID is encoding additional copies instead of the full backup. The scheme can also tolerate double copies failed like RAID5 scheme. It maintains one copy and double parity set.
The storage overhead only remains 2/N, where N denotes how many blocks are calculated as a parity set. Both of them are implemented on userspace of HDFS. The results show, RAID5
scheme can reduce 40% overhead in storage capacity, and RAID6 can reduce 70% overhead because RAID6 have two parity set instead of two full backs.
3.3.2 Background Encoding
HDFS replicates the block in the background. The concern of running in the background is to facilitate the overhead of implementation. The process spends little time during encoding.
In general, increasing the number of copies can improves performance. The paper describes
three issues to discuss because the copies in DiskReduce implementation are less than or orig-inal HDFS. The below will shows three reason reduce copies number might cause low perfor-mance. First reason is duplicated tasks. During Map/Reduce processing, once the computing node crashes or long time to response, the job which the node have that will be taken by an-other node. Above is called Backup task. The attention is, the choice is prefer the node which have local copies which same the original node. Otherwise, that will consume much network bandwidth because the backup task will occupy part of the bandwidth. Finally the computing performance will decrease more than no backup task mechanism. Second reason is disk band-width. According this paper, popular file usually is small size data. If decreasing the number of copies, a part of nodes which have the popular file will become a bottleneck. The GFS design allows a file have many copies that over three copies if necessary. Third reason is load balanc-ing. The data are randomly distributed within the nodes, but it may not be evenly distributed.
The result causes execute Map/Reduce with choosing node has local copies preferably is load-ing unbalance. If have many copies, the job tracker can efficiently assign tasks such that nodes loading will be balanced.
The method is implemented on Hadoop. The experimental result shows, it reduces the overhead of space from 200% 125% to by using RAID6 scheme. Even if only one copy of blocks serves the requests, the performance gets little penalty. Hence the study is very practical and useful to reduce the high overhead of space.
3.4 Summary
The studies solve the problem of availability, performance and space respectively on a clus-ter. Implementing on HDFS is mainstream because HDFS is a very popular platform for build-ing a cloud storage. The researches have the common goal is how to minimize cost to maximize performance under guaranteeing the availability situation.
Hadoop Rack-Awareness
Although the study solves the problems, there are some issues are not being considered or bring new issue. Through well-planing DataNodes location, replicas will be placed into appro-priate node. If the network structure is built with the racks layout, the system has the ability to handle power outage or network accidents. However, the network environment and servers
should match the assumption constructed as tree hierarchical network topology. Furthermore, manager manually maintains a list to record the location of DataNodes in racks. Unfortunately, many environments are non-structural, which implies general network environments with lim-ited cost have no well-planed physical network environment. To recode the location of servers, maintaining thousands of servers is trouble for system administrators.
System Parameter based
The studies of system parameter based can accurately measure various factors in a system by using network and system characteristics. The result shows it requires many replicas to improve performance and availability. In contrast, high replica number comes with several times space overhead is a serious problem. Although raising replica factor of popular file or lowering replica factor of unpopular file is dynamic. On average the number of replicas is greater than four. That is, only less than a quarter of the real space can be used. Furthermore, the method does not consider the location of the node that cannot guarantee availability. Once all replicas of a block of a file are placed in identical node, the file will lose when the node crashes. Thus, placing replicas according to the nodes location is better than increasing replica factor for availability.
DiskReduce
DiskReduce applies the concept of RAID in HDFS. By using data encoding that effectively reduces the space overhead spending additional space to store replicas for data protection. Spe-cially, performance is not reduced too much. However, DiskReduce has the lack of flexibility which cannot let a user specify which file has to be encoded. On the other hand, decoding the encoded file is difficult because the system cannot control which files to be grouped for RAID encoding.
According to the drawbacks, next chapter we present our method, whose one of functions enhances Hadoop’s rack-awareness, and the function of reducing disk space is more flexible than DiskReduce.