I MPLEMENTATION

The system is implemented as a kernel module in Linux 2.4.18. Similar to FT-TCP, we insert the NSW and SSW layers by intercepting several socket-related and TCP-related functions. We also intercept write system call to recover the states of file streams. In addition, we add two kinds of kernel threads for fault detection and packet spoofing.

Figure 4.9: Kernel Threads for Fault Detection and Packet Spoofing

For the socket-related functions, we intercept sock_recvmsg() to insert session identifier and record HTTP requests. We also intercept sock_sendmsg() for session identifier extraction.

Besides, these two functions are intercepted to perform the recovery of the socket streams.

For the TCP-related functions, we intercept tcp_v4_rcv() to hijack the incoming TCP packets.

Outgoing TCP packets are hijacked by intercepting the ip_queue_xmit() and the ip_build_and_send_pkt() functions. All these functions are intercepted in order to record the

TCP traffic and establish the fake 3-way handshakes during the recovery period. Finally, for recovering the states of the file streams, we intercept sys_write(), which is the in-kernel service routine of write system call.

The kernel thread health_monitor is used to monitor the status of Squid. It triggers the recovery operations when Squid fails. For each session that needs to be recovered, the health_monitor creates a spoofer thread, which spoofs packets for re-establishing the corresponding connections (as shown in Figure 4.9).

C HAPTER 5

EXPERIMENTAL RESULTS

This chapter evaluates the fault-tolerant proxy system in four ways. First, we present traces of client-proxy TCP connections to demonstrate the effectiveness of recovering client sessions. Second, we show the overhead on the proxy application and other applications running on the modified operating system. Third, we measure the connection performance of the client sessions which are recovered since the intentional termination of the proxy.

Finally, we show the relation between size of log buffer and number of connections and the relation between recovery time and the number of connections.

5.1 E

E

We use an Intel Pentium 4 2.0 GHz PC running Squid-2.5.STABLE4 [25] as the proxy machine and a server machine which is identical to the proxy machine in hardware running Apache 2.0.40 [5]. Three Pentium 4 1.6 GHz PCs are used as client machines. Each machine is equipped with 512KB cache, 256MB DDR RAM, one 100Mbps Ethernet adaptor, and one Gigabit Ethernet adaptor. All the machines are connected by two independent Ethernet switches (i.e., 100Mbps Ethernet and Gigabit Ethernet).

5.2 E

Figure 5.1: Traces of Client-Proxy TCP Connections

In this experiment, the client downloads a 25MB file from the web server through the proxy. We use tcpdump [16] to check the TCP ACK sequence number of each outgoing packet sent by the client. Figure 5.1 shows the results. The normal line represents the case that the client completes the downloading normally. In the other two cases, we terminate the proxy application when the client has downloaded 12MB of the file. The FT-PROXY line represents the recovery performance of our approach and the FT-TCP line represents that of the FT-TCP mechanism. Since FT-TCP make the proxy reconnect to the server and download the file from the beginning again, its recovery time is longer.

5.3 O

In this section, we characterize the overhead of the fault-tolerant proxy system in terms of the impact on the performance of proxy application and other TCP/IP-based applications.

In Section 5.2.1, we measure the performance of Squid while logging states of client sessions with WebStone benchmark [17]. We show the overhead occurs in an Apache web server running in the modified operating system in Section 5.2.2.

5.3.1 S

P

To present the impact of states logging on Squid performance, we use the standard workload of WebStone 2.5 but make each request be forwarded to the web server (all request are direct access). We achieve this by modifying the cache policy of Squid. This workload shows the impact of session insertion/extraction to associate the socket streams mentioned in Section 4.1.1.

Figure 5.2: Performance of Squid (Connection Rate)

0

Figure 5.3: Performance of Squid (Connection Throughput)

0

Figure 5.4: Performance of Squid (Response Time)

0.00%

Figure 5.5: Decrease of Squid Performance

Figure 5.2, Figure 5.4, and Figure 5.5 illustrate the runtime overhead caused by the state logging operations. The x-axis stands for the number of clients simulated by WebStone. The shaded bars present the performance of Squid running on the modified operating system (the state logging operations are performed) and the empty bars present the performance of Squid running on the normal operating system. Figure 5.5 shows the decrease of Squid performance in the three criteria. It is worth to mention that the overheads are around 1.5%

to 3.8% only.

5.3.2 A

P

We install Apache web server 2.0.40 on the proxy machine. Since the fault-tolerant kernel module checks all TCP traffic and socket read/write operations for logging, the performance of Apache may drop. We measure the overhead with WebStone 2.5. The client machine is the same as Section 5.2.1.

Figure 5.6 illustrates the throughput of the client-server connections. The shaded bars present the performance of Apache running on the modified operating system and the empty bars present the performance of Apache running on the normal operating system. The connection throughput decreases obviously while only one client is simulated. The reason is that the web server has no request to process when the only request is delayed by the modified operating system. Figure 5.7 illustrates the performance decreases of Apache in three ways. Not surprisingly, the performance decrease in each criterion is around 2% while only one client is simulated. The performance decrease of each criterion is lower then 0.5%

even lower than 0% while more clients are simulated. Those small or even minus difference numbers are caused by the inaccuracy of measurement. In summary, the overheads of the modified operating system have little impact on Apache web server.

0

Figure 5.6: Performance of Apache (Connection Throughput)

-0.50%

Figure 5.7: Decrease of Apache Performance

5.4 P

In this section, we measure the performance of Squid which experience failure and recovery. The client requests one file from the server machine through the proxy in each run.

We turn off the caching functions of the proxy in order to measure the worst case of recovery latency. Squid process is being terminated intentionally when the client receives 1Kbytes of data in each run. The transmit time is measured in Gigabit Ethernet environment.

Table 5.1 shows the results of the experiment. According to the result, the recovery latency is around 720 ms in average.

Since the newly establishing connection (server-proxy connection) cannot receive any packet before completing the re-establishment process, the server-side TCP stack should retransmit the packets which have not been acknowledged. However, the long recovery time of Squid (about 200 ms) leads the expiration of retransmission timer. The server-side TCP retransmits those packets again according to the RTO value, and the value will increase twice if the corresponding ACK packet is not received. By tracking the server-proxy connection, we found that the newly established connection cannot continue transmitting data immediately because of the increasing RTO. We solve this problem by receiving the retransmitted packets in the SSW for the newly establishing connection. According to the result shown in Table 5.2, about 240 ms in average of the recovery latency can be reduced.

File Size (Mbytes) 5 10 15 20 25 30

Normal Transmit Time (sec) 0.258050 0.482425 0.690599 0.925669 1.028636 1.217583

Transmit Time including

Recovery Latency (sec) 0.976818 1.228987 1.458313 1.647249 1.768556 1.928813

Recovery Latency (sec) 0.718768 0.746562 0.767713 0.721579 0.739921 0.711230 Table 5.1: Transmit Time of Different Size of File which Experience Recovery

File Size (Mbytes) 5 10 15 20 25 30

Optimized Transmit Time including Recovery Latency (sec)

0.654799 0.961413 1.139041 1.348164 1.478254 1.701996

Optimized Recovery Latency (sec) 0.396749 0.478988 0.448442 0.422494 0.449619 0.484413 Table 5.2: Transmit Time of Different Size of File with Optimization

5.5 S

T

R

Table 5.3 shows the relation between number of on-line sessions and the size of required log buffer. The required buffer size increases while the number of sessions increases. The reason is that we store unhandled data (mentioned in Chapter 4) in the log buffer. Intuitively, the amount of unhandled data increases when the load of Squid increases. Only 132 Kbytes of memory is required to log the state of one session while the proxy is serving 30 sessions.

Number of Sessions 5 10 15 20 25 30

Required Log Buffer (Kbytes) 2.78 28.97 31.06 80.55 127.22 131.94

Table 5.3: Relation between Number of Sessions and Required Log Buffer Size

Table 5.4 shows the relation between number of sessions need to recover and the total recovery time. We intentionally terminate the Squid process when proper number of sessions (e.g., 5, 10, 15) are served. Each client session requests a 25MB web page.

Obviously, to recover more sessions takes more time. And the recovery latency is acceptable.

Number of Sessions 5 10 15 20 30 50

Recovery Time (ms) 102.814 147.626 445.338 525.354 839.004 1751.086

Table 5.4: Relation between Number of Sessions and Recovery Time

C HAPTER 6

CONCLUSION AND FUTURE WORK

6.1 C

We purpose a subsystem to make a proxy service become fault tolerant. Transient faults on a proxy application (either caused by hardware, administrator or the proxy application itself) can be recovered in a way transparent to the clients, servers, and the proxy application.

The subsystem can recover the ongoing requests which are processed when the failure occurs.

In addition, the feature of transparency enables the fault tolerant proxy system functions without any support of the clients and the servers. The subsystem is implemented as a Linux kernel module. TCP traffic and system calls are intercepted in kernel-level to log the state.

The experimental results show low overhead for state logging and acceptable recovery latency.

6.2 F

W

Currently, we focus on the errors happened on a proxy application only. In the future, we will address the operating system failures. Operating system crashes such as kernel panics could be detected by another machine, and the states could also be logged in that machine.

Therefore, we can use a logger machine to monitor the proxy. When the proxy fails, the state of the proxy would be migrated to another backup server. Instead of using an additional machine, the system crashes can also be detected by an intellectual network interface card [3].

With the help of fast system restart techniques such as LOBOS [18] which stores the state in a safe memory area that survives after restarting the system, the state recovery can be performed.

In addition, we plan to extend our fault tolerant mechanisms to other network services in the future. In network services such as Web Service [11] and peer-to-peer network [21], each host may play the role of client and server at the same time. This is similar to the proxy service. Therefore, we will evaluate the possibility to extend our approach to provide fault tolerance for those services.

REFERENCES

[1] Navid Aghdaie, Yuval Tamir, “Client-Transparent Fault-Tolerant Web Service,” 20th IEEE International Performance, Computing, and Communications Conference, Phoenix, AZ, pp. 209-216, Apr. 2001.

[2] Navid Aghdaie, Yuval Tamir, “Fast Transparent Failover for Reliable Web Service,” In Proceedings of the International Conference on Parallel and Distributed Computing and Systems, Marina del Rey, California, pp. 757-762, Nov. 2003.

[3] Alacritech, “Alacritech quad port server accelerator,” available at http://www.alacritech.com/html/100x4.html

[4] L. Alvisi, T. C. Bressoud, A. El-Khashab, K. Marzullo, D. Zagorodnov, “Wrapping Server-side TCP to Mask Connection Failures,” In Proceedings of the IEEE INFOCOM, Anchorage, Alaska, pp. 329-337, Apr. 2001.

[5] Apache Software Foundation, “The Apache Web Server,” available at http://www.apache.org/.

[6] T. Briso, “DNS Support for Load Balancing,” IETF RFC 1794, April 1995.

[7] A. Brown, D. A. Patterson, “To Err is Human,” In Proceedings of the 2001 Workshop on Evaluating and Architecting System dependabilitY, Göteborg, Sweden, July 2001.

[8] V. Castelli, R. E. Harper, P. Heidelberger, S. W. Hunter, K. S. Trivedi, K. Vaidyanathan, W. P. Zeggert, “Proactive Management of Software Aging,” IBM JRD, Vol. 45, No. 2, Mar. 2001.

[9] Cisco Systems Inc., “Cisco DistributedDirector,” available at http://www.cisco.com/univercd/cc/td/doc/product/iaabu/distrdir/dd2501/ovr.htm

[10] Cisco Systems Inc., “Web Cache Communication Protocol,” available at http://www.cisco.com/en/US/tech/tk122/tk717/tech_protocol_family_home.html

[11] Harvey M. Deitel, “Web Services: A Technical Introduction,” Prentice Hall, Aug. 2002.

[12] P. Enriquez, A. Brown, D. A. Patterson, “Lessons from the PSTN for Dependable Computing,” In Proceedings of the 2002 Workshop on Self-Healing, Adaptive and self-MANaged systems (SHAMAN), New York, June 2001.

[13] R. Fielding, J. Gettys, J. Mogul, H.Frystyk, L. Masinter, P. Leach, T. Berners-Lee,

“Hypertext Transfer Protocol -- HTTP/1.1,” RFC 2616, Jun. 1999.

[14] S. Horman, “Creating Redundant Linux Servers,” In Proceedings of the 4th Annual Linux Expos, Durham NC, May 1998.

[15] S. Iyer, A. Rowstron, P. Druschel, “Squirrel: A Decentralized, Peer-to-Peer Web Cache,”

21th ACM Symposium on Principles of Distributed Computing (PODC), Monterey, California, Jul. 2002.

[16] Van Jacobson, Craig Leres, Steve McCanne, “tcpdump,” available at http://www.tcpdump.org/.

[17] Mindcraft Inc., “WebStone: the Benchmark for Web Servers,” available at

http://www.mindcraft.com/benchmarks/webstone/.

[18] Ron Minnich, “LOBOS: (Linux OS Boots OS) Booting a Kernel in 32-bit Mode,” The Fourth Annual Linux Showcase and Conference, Atlanta GA, Oct. 2000.

[19] Netscape, “Navigator Proxy Auto-Config File Format,” available at http://wp.netscape.com/eng/mozilla/2.0/relnotes/demo/proxy-live.html, Mar. 1996.

[20] D. Oppenheimer, D. A. Patterson, “Why do Internet services fail, and what can be done about it?” In Proceedings of the 10th ACM SIGOPS European Workshop, Saint-Emilion, France, Sep. 2002.

[21] Andy Oram, “Peer-to-Peer: Harnessing the Benefits of a Disruptive Technology,”

O’Reilly, Mar. 2001.

[22] Alex C. Snoeren, Hari Balakrishnan, “An End-to-End Approach to Host Mobility,” In Proceedings of the 6th Annual ACM/IEEE International Conference on Mobile Computing and Networking, pp. 155–166, Boston, Massachusetts, Aug. 2000.

[23] Alex C. Snoeren, David G. Andersen, Hari Balakrishnan, “Fine-Grained Failover Using Connection Migration,” In Proceedings of the 3rd USENIX Symposium on Internet Technologies and Systems (USITS '01), Mar. 2001.

[24] K. Srinivasan, “M-TCP: Transport Layer Support for Highly Available Network Services,” Technical Report DCS-TR459, Rutgers University, Oct. 2001.

[25] D. Wessels, “Squid Web Proxy Cache,” available at http://www. squid-cache.org/.

[26] C. S. Yang, M. Y. Luo, “Realizing Fault Resilience in Web-Server Cluster”, In Proceedings of the 2000 ACM/IEEE Conf. on Supercomputing (CDROM), p.21-es, Nov.

2000.

[27] C. S. Yang, M. Y. Luo, “Constructing Zero-Loss Web Services”, In Proceedings of IEEE INFOCOM 2001, pp. 1781-1790, Apr. 2001.

[28] Dmitrii Zagorodnov, Keith Marzullo, Lorenzo Alvisi, Thomas C. Bressoud,

“Engineering fault-tolerant TCP/IP servers using FT-TCP,” In Proceedings of IEEE Intl.

Conf. on Dependable Systems and Networks (DSN), pp. 22-26, Apr. 2003.