In this section, we will describe the execution flow of thttpd. Thttpd uses single-process implementation of HTTP 1.1 protocol [8], and it divides the procedure of handling a request into two stages, namely reading and sending. Figure 5.1 shows the stage flow of request handling in thttpd.
Reading Sending
Got an incoming connection
Finish Sending Response
Generated
Start End
Figure 5.1: Stage Flow of Request Handling in Thttpd
When thttpd starts, it creates, binds, and listens on a TCP socket. Then, it probes for incoming requests by performing select() on the TCP socket. If there is no request, the server keeps on probing. When the server gets a request, it creates a connection entity structure for this request and the connection enters into the reading stage. The connection entity structure contains almost all the information needed to construct the total state variables of thttpd. This will be discussed in the next section. In the reading stage, the server reads the request from the client, parses the request, and generates the corresponding response. After the response is generated, the connection enters into the sending stage. In this stage, the server writes out the response to the client.
Note that connections are processed concurrently. Different connections may be in different stages. Moreover, request probing and processing are also handled concurrently. It is worth to mention that, reading and sending stages in thttpd are divided into more fine-grained sub-stages. In reading stage, the server doesn’t perform blocking read operation. Therefore, several read operations may be needed to get the full request. And the server will try to handle other requests between two successive read operations. Similarly, in the sending state, the server will also try to write a part of the response and then handle other requests or accept new requests.
5.1.2 Zero-Loss Restartable Thttpd
In this section, we describe how we modified the thttpd to make it zero-loss restarable. The modified version is called ZLR_thttpd. First of all, we replaced all socket read operations in thttpd with hread(), and applied the reregist() and the
getreginfo() on thttpd. We will not describe the detail of these modifications because
that they are simply like what we have mentioned in Chpater 4.The other modifications are described in the following. First, we identified the state variables of thttpd. In the previous section, we mentioned that the connection entity structure is used to represent a connection in thttpd, and it contains all the information of an on-processing connection. Therefore, we can get the state variables of a connection by extracting the fields in this structure that are necessary for the recovery procedures. Instead of separating the original data structures in thttpd into state variable part and no_state variable part, we make a copy of all state variables and store the copy into the shared memory. And, we update the variable in shared memory before the corresponding variable in thttpd is modified. These can avoid large modifications to the original thttpd.
struct httpd_state_var {
Figure 5.2: State Variables of Thttpd
Figure 5.2(a) shows the per-connection state variables, the http_state_var structure. The fields in this structure can be divided into three parts. The first part contains the most important fields of a connection, conn_stage and conn_fd. The
conn_stage field represents the connection stage, and the conn_fd field represents the
data socket used for communicating with the client. The second part is updated during the reading stage of a connection. The expnfilename represents the requested file name in the server, and the method field indicates the HTTP method. These two fields are generated after the request is parsed. The last part is updated during the sending stage. The bytes_sent field represents how many bytes of response have been writtento the data socket. In addition to the per-connection state variables, the global variables are maintained in the global_state_var structure as shown in Figure 5.2(b).
The listen_fd represents the socket that the server uses to receive requests from the clients. The num_connects field indicates how many connections are currently processed in the server. And, the max_connect field indicates the maximum number of connections that the server can process simultaneously.
The second modification we made was using shared memory areas to store the
state variables of thttpd. Totally, four shared memory areas are used. As Figure 5.3
shows, these four areas are pointed by four pointers, shm_pointers, httpd_state_vars,char_area, and global_vars, respectively.
shm_pointers
○1 expnfilename
httpd_state_vars
chars_area expnfilename
The area pointed by httpd_state_vars is used to store the httpd_state_var structures of all the connections. The char_area points to a fix-sized shared memory area that is used to store the expnfilename fields of all the httpd_state_var structures.
The above two pointers are stored in another shared memory area, which is pointed by
global_vars
○2
○1 sizeof ( httpd_state_var )
○2 fix size chars area
○3
○3 sizeof (global_state_var )
Figure 5.3: Four Shared Memory Areas in Modified Thttpd
the shm_pointers pointer. In addition, the global_vars points to a shared memory area that stores the global_state_var structure of the server.
The final modification we made was to add a recovery path which will be executed during the service recovery phase. Since thttpd is a single-process application, the path contains a single recovery procedure only. In order to facilitate the recovery, we added a new stage, processing, between the reading and sending stages. As shown in Figure 5.4, the stage is entered when a request is completely read and parsed. In this stage, the server will use the parsing result of the request to generate the response. This stage is added to prevent the service going back to the reading stage while the request has been parsed. The recovery procedure performs two jobs. First, it restores all the variables from the shared memory area. Second, it handles the recovery of each on-processing connection according to the connection stage. For the reading stage, it tries to finish the reading and parsing job. For the
processing stage, it tries to read the requested file again into the memory space of the
thttpd. For the sending stage, the recovery procedure reads the requested file again into the memory space of the thttpd, and then sends the response to client. Note that only the remaining response will be sent since we have recorded the number of sent bytes.Figure 5.4: State Diagram of Connection in ZLR thttpd