In Peeraid, all nodes in the system are organized into more than one overlay network by the DHT module, which also provides the mechanisms for message routing and key lookup. Comparing with most peer-to-peer file systems based on DHT, Peeraid separates file access from the DHT protocol. User files are not directly stored at the nodes mapped from their keys. They are distributed to nodes chosen according to the policy that users assign to provide more flexibility for service applications.
File-object
Figure 3-4 Publishing a File-Object.
/user/foo.txt
Figure 3-3 Retrieving a File-Object.
15
3.2.1 DHT-based vs. directory-based
In the early design, storage systems built on top of DHT usually use content hashes of data objects as their object keys, and store them at the nodes directly mapped from their keys by DHT, which is called DHT-based storage systems. The advantages of this design are simple and easy for backup management, because every node is required to maintain a list of its neighbor nodes in these systems.
However, a serious problem of these systems is lack of flexibility. In these systems, users cannot do anything to control locations of their files, so files are often stored at the nodes far way from those users who need to access them frequently. This will significantly reduce system performance. Previous research such as [11] had focused on server selection and file caching on the routing path. Nevertheless, server selection cannot take effect when only a few nodes are available. File caching on the routing path will greatly increase the storage cost. Specifically, for an open network environment such as Cloud Computing, file confidentiality is one of the most important concerns. Users might not be willing to store their files at too many nodes that they cannot trust.
B. G. Chun and F. Dabek et al. had proposed another architecture of DHT [5], which is called directory-based storage systems. In directory-based storage systems, a data object can be stored and backed up at arbitrary nodes, and the location pointers to this object are stored at the nodes mapped from its key. Figure 3-5 illustrates the difference between DHT-based and directory-based systems. This kind of design separates the policy of file distribution from the DHT protocol, and provides users more flexibility. However, accompanying with the benefit of flexibility is how to choose the appropriate nodes for storing files and how to manage user files so that data
DHT Directory
Figure 3-5 DHT-based vs. directory-based.
availability can be ensured. Peeraid combines the concept of group and file distribution to address these two problems, and we will discuss it in the later section.
Peeraid integrates both architectures to locate data objects according to their types.
SysInfo-Objects and Metadata-Objects are stored with DHT-based in order to locate them efficiently, whereas File-Objects are stored with directory-based so that users are allowed to control the location of their files. Users who need to access a File-Object should retrieve its corresponding Metadata-Object by the DHT module first to obtain location information of file shares recorded in the Metadata-Object.
3.2.2 Kademlia
Kademlia [23] is a distributed hash table that has the similar basic idea like those introduced previously. However, there are three main differences between Kademlia and other DHTs. First, Kademlia uses XOR metric for calculating the distance between nodes instead of numerical difference. XOR is a symmetric operation, allowing
Kademlia participants to receive lookup queries from precisely the same distribution of nodes contained in their routing tables.
Second, for the ith entry in the routing table, each Kademlia node keeps a list of〈IP address, UDP port, Node ID〉triples for up to k nodes of distance between 2i and 2i+1 instead of only one node like other DHTs. Therefore, the routing table of Kademlia is called k-buckets, and k is chosen such that any given k nodes are very unlikely to fail within an hour. Maintaining information about multiple nodes within an interval of identifier space makes Kademlia more robust and tolerant towards network churn and DoS attacks.
Third, each message sent by a node must include its node ID, so that the recipient can learn about the sender and update its k-bucket for the sender’s node ID. This minimizes the number of messages required by a node to learn about each other, which spreads automatically as a side-effect of key lookups. As a result, the cost of building and maintaining routing tables in Kademlia is lower than other DHTs. Besides, additional useful information about senders can be appended to the message sent (e.g.
latency between two nodes).
Peeraid uses Kademlia protocol as the DHT module due to its low cost of initializing and maintaining routing tables. Kademlia minimizes the number of messages a node must send to learn about each other. This property can be deployed to support group management. Every user on Peeraid has a global routing table for key lookups among all nodes in the system. However, every group has their own routing table for key lookups only among group members. This will ensure that all lookups are carried out within the group and enhance the security of file access. The efficiency of lookups among group members is also improved because the number of group members is usually far less than the number of all nodes in the system. Figure 3-6
Global
Figure 3-6 Every group on Peeraid has their own routing table for key lookups.
illustrates this feature.
3.2.3 Kademlia protocol
The original Kademlia protocol consists of four RPCs:
PING: probes a node to see if it is online.
FIND_NODE: takes a 160-bit key ID as the argument. The recipient of the RPC must return up to k〈IP address, UDP port, Node ID〉triples that it knows to be closest the key.
STORE: instructs a node to store a〈key, value〉pair for later retrieval.
FIND_VALUE: behaves like FIND_NODE with one exception. If a
corresponding value is present on the recipient, the associate data is returned.
For participant peers, the most important function is to locate the k closest nodes to some given target ID, which is called a node lookup. Kademlia accomplishes node
lookups with a recursive algorithm. First, the initiator picks α nodes closest to the target ID from its local k-bucket, and then sends parallel, asynchronous FIND_NODE RPCs to these α nodes.
After receiving any response from peer nodes, the initiator again picks α nodes closest to the target ID it has learned about and resends FIND_NODE RPCs to these nodes. By repeating this process recursively, the initiator learns about nodes with node ID closer and closer to target ID. The lookup terminates when the initiator has queried and obtained responses from the k closest nodes it has seen.
The DHT module on Peeraid includes PING and FIND_NODE RPCs, and the node lookup function, but without implementing STORE and FIND_VALUE. As mentioned before, Peeraid uses both DHT-based and directory-based architectures to locate data objects according to their types, and users must be authenticated by other peers when they try to access files. Therefore, we separate file access functions from the DHT protocol, and implement them on the top layers.
3.2.4 Revising Kademlia
Since every message a node sends is requested to include its node ID in Kademlia, additional useful information about the sender can be appended. In Peeraid, timestamps are piggy-backed on every message for measuring round-trip time between nodes. With this information, users can route queries through a low-latency path and select low-latency nodes from multiple sources to download files.