Servers may fail, and it causes errors. An AOM n tests if its predecessor and successor fails periodically and sends the AOM RECOVERY message to the another neighbor and it contains informations of its neighbor. When an AOM receive this message, it checks the status of the message. If the status is ”Native”, the AOM just passes the message to the next, otherwise it saves this information.
When an AOM detects the failed neighbors, it tries to ask the informations of
conrre-spondding node according to the recovery information to fix its failed neighbor.
Figure 3.12: Failure recovery
3.2 Client
There are three components in our client, the memcached protocol, the distribution method and the AOM protocol. The memcached protocol is responsible for operating the memcached servers and it is used by the user API, the distribution method. The AOM protocol for clients synchronizes the servers in the server poll by communicating with AOMs. Figure 3.13 shows the architecture of servers and clients.
When a client starts, it sends the CLIENT JOIN message to any AOM in the network according to the AOM protocol, then it will receive the servers list of the network in a short period of time. When the client needs to load or store the object to a server, it runs the distribution method to decide which server is the target, then it uses the memcached API to visit the memcached server.
Our client runs the client part of the aom protocol to receive the server information. Clients will receive three types of messages AOM JOIN, AOM LEAVE and AOM MOVE. A client adds the server into its servers list when receiving the AOM JOIN message, deletes the server when receiving the AOM LEAVE message and updates the server when receiving the AOM MOVE message.
The distribution method is the API for users, and it is the normal consistent hash method with the argument of replications r when store and the argument tries t when load. The client
Memcached client
stores an object into continuous r servers started from the server picked up by the normal consistent hash method along the searching direction and tries to load the object from the server picked up by the normal consistent hash method. If it loads miss, it tries to load the object from the previous node(predecessor) until cache hit or the number of times reaches the tries t. In our system, there are two cases that a cache miss will happen. The one is the object a client tries to load is not existed, and another is that the server stores the object has moved. We choose the predecessor as the next node to load because the movement of servers is counterclockwise on the ring. Figure 3.14 shows the example of distribution method of store and load suppose r and t are both 3.
3.3 Implementation of system
Both AOMs and clients are implement in C language with ”libevent” library. The libevent API provides a mechanism to execute a callback function when a specific event occurs on a file descriptor or after a timeout has been reached. Furthermore, libevent also supports callbacks due to signals or regular timeouts. Memcacehd uses libevent to develop the part of networking, too.
The AOM protocol relies on the exchanging of the message that we mentioned before. The
Position of
message format is in figure 3.15. There is a header(figure 3.16) and a content in a message.
The header describes the type, status of the message and length of the content. Type points the action of the producer, and it can be join, leave or something we mentioned before. Status defines the direction if the message must be pass. Length of message writes the size of content in the message. The content writes the position, load and IP of the nodes even more. These nodes can be the producer of message or the neighbors of the producer. The content is composed by the minimum unit called ”content token”(figure 3.17). Every message ends by
”\ n”.
Header(3 bytes) Content(N bytes)
Figure 3.15: The format of a message
Type (1 byte) status(1 byte) length(1 byte)
Figure 3.16: Header
Once the AOM or the client receives a message, it parses the message and forward or backward the message according to the column ”status” in the header, then it handles the message according to the ”type” in the header and the content of message according to the AOM protocol we talked in the previous chapter.
Content token Content token ...
(a) Content
Node(1 byte) information(1 byte) value(4 bytes)
(b) Content token Figure 3.17: The format of content and content token.
The communication of AOM and memcached depends on the ”stats” in memcached proto-col. An AOM can get many informations of memcached by ”stats” but it just cares about the
”bytes” row and the ”limit maxbytes” row at present. The first points out the used memory and the second points out the total memory.
The user API of our client is listed as follows. User must call start() at begining and stop() in the end.
1. start():
Initial and start the client.
2. store(key, data, length, stores):
Store the object(key, data) by the distribution method of store with the argument of
$stores.
3. load(key, item, tries):
Load the item with $key and $tries from memcached servers by the distribution method of load.
4. stop():
Stop the client.