ZooKeeper presents a consistent state to all of the client processes that are using it. When a client gets a response from ZooKeeper, the client can be confident that the response will be consistent with all other responses that it or any other client receives. There are times when a ZooKeeper client library loses its connection with the ZooKeeper service and can no longer provide information that it can guarantee to be consistent. When a ZooKeeper client library finds itself in this situation, it uses the Disconnected event and the ConnectionLossException to express its lack of knowledge about the state of the system.
Of course, the ZooKeeper client library vigorously tries to extricate itself from this sit‐
uation. It will continuously try to reconnect to another ZooKeeper server until it is finally able to reestablish the session. Once the session is reestablished, ZooKeeper will generate a SyncConnected event and start processing requests. ZooKeeper will also re‐
register any watches that were previously registered and generate watch events for any changes that happened during the disconnection.
A typical cause of Disconnected events and ConnectionLossExceptions is a ZooKeeper server failure. Figure 5-3 shows an example of such a failure. In this example a client is connected to server s2, which is one of two active ZooKeeper servers. When s2 fails, the client’s Watcher objects will get a Disconnected event and any pending requests will
return with a ConnectionLossException. The ZooKeeper service itself is fine because a majority of servers are still active, and the client will quickly reestablish its session with a new server.
Figure 5-3. Connection loss example
If the client doesn’t have any pending requests, all this will take place with very little disruption to the client. Apart from a Disconnected event followed by a SyncConnec ted event, the client will not notice the change. If there are pending requests, however, the connection loss is much more disruptive.
If the client has a pending request outstanding, such as a create request that it just submitted, when the connection loss happens the client will get a ConnectionLossEx ception for synchronous requests and a CONNECTIONLOSS return code for asynchronous requests. However, the client will not be able to tell from these exceptions or return codes whether or not the requests were processed. As we have seen, handling the con‐
nection loss complicates the code because the application code must figure out whether the requests actually completed. One very bad way of dealing with the complication of handling connection loss is to code for the simple case, then shut everything down and restart if a ConnectionLossException or CONNECTIONLOSS return code is received. Al‐
though this makes the code simpler, it turns what should be a minor disruption into a major system event.
To see why, let’s look at a system that is composed of 90 client processes connected to a ZooKeeper cluster of three servers. If the application is written using this simple but bad style and one of the ZooKeeper servers fails, 30 client processes will shut down and restart their sessions with ZooKeeper. To make matters worse, the session shutdown happens when the client processes are not connected to ZooKeeper, so their sessions
will not get explicitly shut down and ZooKeeper will have to detect the failures based on the session timeouts. The end result is that a third of the application processes restart, and the restarts may be delayed because the new processes must wait for the locks held by the old sessions to expire. On the other hand, if the application is written to correctly handle connection loss, such scenarios will cause very little system disruption.
Developers must keep in mind that while a process is disconnected, it cannot receive updates from ZooKeeper. Though this may sound obvious, an important state change that a process may miss is the death of its session. Figure 5-4 shows an example of such a scenario. Client c1, which happens to be a leader, loses its connection at time t1, but it doesn’t find out that it has been declared dead until time t4. In the meantime, its session expires at time t2, and at time t3 another process becomes the leader. From time t2 to time t4 the old leader does not know that it has been declared dead and another leader has taken control.
Figure 5-4. Revenge of the living dead
If the developer is not careful, the old leader will continue to act as a leader and may take actions that conflict with those of the new leader. For this reason, when a process receives a Disconnected event, the process should suspend actions taken as a leader until it reconnects. Normally this reconnect happens very quickly.
If the client is disconnected for an extended period of time, the process may choose to close the session. Of course, if the client is disconnected, closing the session will not make ZooKeeper close sooner. The ZooKeeper service still waits for the session expi‐
ration time to pass before declaring the session expired.
Ridiculously Long Delay to Expire
When disconnects do happen, the common case should be a very quick reconnect to another server, but an extended network outage may introduce a long delay before a client can reconnect to the ZooKeep‐
er service. Some developers wonder why the ZooKeeper client li‐
brary doesn’t simply decide at some point (perhaps twice the session timeout) that enough is enough and kill the session itself.
There are two answers to this. First, ZooKeeper leaves this kind of policy decision up to the developer. Developers can easily implement such a policy by closing the handle themselves. Second, when a Zoo‐
Keeper ensemble goes down, time freezes. Thus, when the ensemble is brought back up, session timeouts are restarted. If processes using ZooKeeper hang in there, they may find out that the long timeout was due to an extended ensemble failure that has recovered and pick right up where they left off without any additional startup delay.