}
StatCallback existsCallback = new StatCallback() {
public void processResult(int rc, String path, Object ctx, Stat stat) { // Process the result of the exists call
} };
ZooKeeper exists call. Note that the call is asynchronous.
Watcher implementation.
exists callback.
As we will see next, we’ll make extensive use of this skeleton.
The Master-Worker Example
Let’s now look at how we deal with changes of state in the master-worker example. Here is a list of tasks that require a component to wait for changes:
• Mastership changes
• Master waits for changes to the list of workers
• Master waits for new tasks to assign
• Worker waits for new task assignments
• Client waits for task execution result
We next show some code snippets to illustrate how to code these tasks with ZooKeeper.
We provide the complete example code as part of the additional material to this book.
Mastership Changes
Recall from “Getting Mastership” on page 51 that an application client elects itself master by creating the /master znode (we call this “running for master”). If the znode already exists, the application client determines that it is not the primary master and returns.
That implementation, however, does not tolerate a crash of the primary master. If the primary master crashes, the backup masters won’t know about it. Consequently, we need to set a watch on /master so that ZooKeeper notifies the client when /master is deleted (either explicitly or because the session of the primary master has expired).
To set the watch, we create a new watcher named masterExistsWatcher and pass it to exists. Upon a notification of /master being deleted, the process call defined in masterExistsWatcher calls runForMaster:
StringCallback masterCreateCallback = new StringCallback() {
public void processResult(int rc, String path, Object ctx, String name) { switch (Code.get(rc)) { public void process(WatchedEvent e) {
if(e.getType() == EventType.NodeDeleted) {
In the case of a connection loss event, the client checks if the /master znode is there because it doesn’t know if it has been able to create it or not.
If OK, then it simply takes leadership.
If someone else has already created the znode, then the client needs to watch it.
If anything unexpected happens, then it logs the error and doesn’t do anything else.
This exists call is to set a watch on the /master znode.
If the /master znode is deleted, then it runs for master again.
Following the asynchronous style we used in “Getting Mastership Asynchronously” on page 56, we also create a callback method for the exists call that takes care of a few cases. First, it tries the exists operation again in the case of a connection loss event because it needs to set a watch on /master. Second, it is possible for the /master znode to be deleted between the execution of the create callback and the execution of the exists operation. Consequently, we need to check whether stat is null whenever the response from exists is positive (return code is OK). stat is null when the node does not exist. Third, if the return is not OK or CONNECTIONLOSS, then we check for the /master znode by getting its data. Say that the client session expires. In this case, the callback to get the data of /master logs an error message and exits. Our exists callback looks like this:
StatCallback masterExistsCallback = new StatCallback() {
public void processResult(int rc, String path, Object ctx, Stat stat) { switch (Code.get(rc)) {
case CONNECTIONLOSS:
masterExists();
break;
case OK:
if(stat == null) {
state = MasterStates.RUNNING;
runForMaster();
} break;
default:
checkMaster();
break;
} } };
In the case of a connection loss, just try again.
If it returns OK, run for master only in the case that the znode is gone.
If something unexpected happens, check if /master is there by getting its data.
The result of the exists operation over /master might be that the znode has been deleted.
In this case, the client needs to run for /master again because it is not guaranteed that the watch was set before the znode was deleted. If the new attempt to become primary
fails, then the client knows that some other client succeeded and it tries to watch /master again. In the case the notification for /master indicates that it has been created instead of deleted, the client does not run for /master. At the same time, the corresponding exists operation (the one that has set the watch) must have returned that /master doesn’t exist, which triggers the procedure to run for /master from the exists callback.
Note that this pattern of running for master and executing exists to set a watch on /master continues for as long as the client runs and does not become a primary master. If it becomes the primary master and eventually crashes, the client can restart and reexecute this code.
Figure 4-1 makes the possible interleavings of operations more explicit. If the create operation executed when running for primary master succeeds (a), the application client doesn’t have to do anything else. If the create operation fails, meaning that the node already exists, the client executes an exists operation to set a watch on the /master znode (b). Between running for master and executing the exists operation, it is possible that the /master znode gets deleted. In this case, the exists call indicates that the znode still exists, and the client simply waits for a notification. Otherwise, it tries to run for master again, trying to create the /master znode. If creating the /master znode succeeds, the watch is triggered, indicating that there has been a change to the znode (c). This notification, however, is meaningless, because the client itself made the change. If the create fails again, we restart the process by executing exists and setting a watch on /master (d).
Figure 4-1. Running for master, possible interleavings