Other approaches

Autonomic computing [32] proposes self-healing techniques that can automatically detect, diagnose, and repair software and hardware problems. Recovery-oriented computing [2,33] also proposed new

techniques to deal with hardware faults, software bugs, and operator errors. The basic principle of these two projects is similar to ours, i.e. systems should deal with faults instead of preventing them.

Checkpointing [12, 34–38] is a common technique for system recovery. It saves the state of a running program periodically to a stable storage. When the system crashes, the last checkpointed state can be reloaded to recover the system. This approach has two drawbacks. First, it cannot solve the software aging problem since the checkpoint state ages rather than being refreshed, so even if the system can be recovered the software may fail again immediately. Second, checkpointing usually results in a large performance overhead due to the large volume of states that need to be stored.

Recursive Restart (RR) [18,19] and Scalable Network Services (SNS) [20] both allow a fine-grained component-based service to restart a component or a set of components (instead of the whole service) once the component fails, and thus reduce the service restart time. However, the inter-component communication degrades the system performance, which is not suitable for performance-critical Internet services. Moreover, they require an Internet service to be composed of fine-grained components, which requires redesigning the legacy Internet service programs.

Backdoors system [39] can recover a service session state in a cluster environment. When a service node fails, the session state on the failed node is transferred to a backup node via the remote direct memory access (RDMA) mechanism. Similar to our work, the system detects OS failures. However, it requires service applications to checkpoint their state periodically, which requires moderate application modifications. Moreover, it requires special/programmable network cards with a RDMA capability.

Zap [40] allows a group of processes with online connections to be migrated to a new host by using the following two mechanisms. First, it checkpoints the connection state right before the migration so that the state can be restored on the new host. Second, it utilized an end-to-end VNAT (Virtual Network Address Translation) [41] approach and a proxy to direct the traffic to the new host. In contrast with our work, Zap does not address application and OS failures. Moreover, an extra proxy is required for client connection migration.

SSM [42] is a highly-available storage system for the user session state. In the system, stateless application servers access the session state via connection with the storage nodes. SSM improves the availability of the session state by distributing the state to multiple storage nodes. Moreover, each storage node can be restarted and recovered independently if faults have occurred on that node.

In contrast with our work, SSM assumes that application servers are stateless, and requires the application services to access the session state on the storage nodes. In contrast, our work does not have such a requirement and is able to recover stateful service applications.

7. DISCUSSIONS

In this section, we provide discussions about the memory overhead and the fault-detection techniques.

As shown by the experimental results, the log buffer does not cause memory pressure on the system in normal cases. However, there exist some pathological cases in which the buffer may overwhelm the system. For example, uploading a huge file to our fault-tolerant FTP server causes memory pressure on the server as a result of the logging of the client request (i.e. the content of the uploaded file).

For another example, if the consumption rate of the response data is far behind the production rate in a Web proxy system (e.g. maybe owing to the congestion of the network path between the proxy and the client), the not-yet-consumed response data will cause memory pressure to the proxy system.

window of the server connection temporarily. In the future, we will incorporate these approaches into our framework.

As described previously, we use techniques to detect application and OS faults. However, there are still a number of faults that cannot be detected by these techniques. For example, an infinite loop in a service application cannot be detected since it does not cause the abnormal termination of the application. We can detect such faults by sending test requests to the application periodically and checking the results. Some other OS faults can be detected by catching kernel exceptions, or by tracking the numbers of served interrupts and context switches, as proposed by Sultan et al. [39]. In the future, we will incorporate more fault-detection techniques into our framework.

8. CONCLUSIONS

In this paper, we proposed a framework for providing a zero-loss Internet service recovery and upgrade on a single host. Based on Xen and FT-TCP, the framework can recover transient application and OS faults. In addition, it allows online service upgrades without stopping the service. In order to make the application of FT-TCP to VMMs more feasible, we proposed techniques to reduce the inter-VM switches and communication. Moreover, we proposed service-specific optimizations to reduce the recovery time of FT-TCP. Finally, the framework was shown to provide an interface for service designers to implement more service-specific optimizations. We evaluated the effectiveness and efficiency of our framework on two popular service programs, Squid and Proftpd. According to the experimental results, our approach incurs little performance overhead (i.e. ranging from 1 to 4%) and memory overhead (i.e. less than 750 KB).

In the current implementation, the framework integrates optimizations for FTP and proxy services.

In the future we will investigate more possible optimizations for other services and integrate them into our framework. In addition, we will provide a better interface for the service designers so that they can specify their optimizations, instead of implementing the optimizations by themselves.

REFERENCES

1. Performance Technologies Inc. The Effects of Network Downtime on Profits and Productivity—a White Paper Analysis on the Importance of Non-stop Networking.

http://searchtechtarget.techtarget.com/0,293857,sid7 gci827871,00.html [January 2007].

2. Patterson D et al. Recovery-oriented computing (ROC): Motivation, definition, techniques, and case studies. Computer Science Technical Report UCB//CSD-02-1175, UC Berkeley, March 2002.

3. Intel Corporation. Intel Networking Technology—Load Balancing.

http://www.intel.com/network/connectivity/resources/technologies/load balancing.htm [2003].

4. Jann J, Browning LM, Burugula RS. Dynamic reconfiguration: Basic building blocks for autonomic computing on IBM pSeries servers. IBM Systems Journal 2003; 42(1):29–37.

5. Patterson D, Chen P, Gibson G, Katz RH. Introduction to redundant arrays of inexpensive disks (RAID). Digest of Papers for 34th IEEE Computer Society International Conference (COMPCON Spring ’89). IEEE Computer Society Press:

Los Alamitos, CA, 1989; 112–117.

6. Brown A, Patterson DA. To err is human. Proceedings of the 2001 Workshop on Evaluating and Architecting System Dependability, Sweden, July 2001.

7. Oppenheimer D, Ganapathi A, Patterson DA. Why do Internet services fail, and what can be done about it? Proceedings of the 4th USENIX Symposium on Internet Technologies and Systems (USITS’03), Seattle, WA, March 2003.

8. Huang Y, Kintala C, Kolettis N, Fulton ND. Software rejuvenation: Analysis, module and applications. Proceedings of the 25th International Symposium on Fault Tolerant Computing, Pasadena, CA, June 1995; 381–390.

9. Parnas DL. Software aging. Proceeding of the 16th International Conference on Software Engineering, Sorrento, Italy, 1994; 279–287.

10. Chou A, Yang J, Chelf B, Hallem S, Engler D. An empirical study of operating system errors. Proceedings of the 18th ACM Symposium on Operating Systems Principles. ACM Press: New York, 2001; 73–88.

11. HP NonStop Group. Personal communication, 1998.

12. Plank JS. An overview of checkpointing in uniprocessor and distributed systems, focusing on implementation and performance. Technical Report UTCS-97-372, University of Tennessee, July 1997.

13. Alvisi L, Bressoud TC, El-Khashab A, Marzullo K, Zagorodnov D. Wrapping server-side TCP to mask connection failures.

Proceedings of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM ’01), Anchorage, AK, April 2001; 329–337.

14. Snoeren AC, Andersen DG, Balakrishnan H. Fine-grained failover using connection migration. Proceedings of the 3rd USENIX Symposium on Internet Technologies and Systems (USITS ’01), San Francisco, CA, March 2001.

15. Snoeren AC, Balakrishnan H. An end-to-end approach to host mobility. Proceedings of the 6th Annual ACM/IEEE International Conference on Mobile Computing and Networking, Boston, MA, August 2000; 155–166.

16. Sultan F, Srinivasan K, Iyer D, Iftode L. Migratory TCP: Connection migration for service continuity in the Internet.

Proceedings of the 22nd International Conference on Distributed Computing Systems (ICDCS’02), Washington, DC, July 2002; 469–470.

17. Zagorodnov D, Marzullo K, Alvisi L, Bressoud TC. Engineering fault-tolerant TCP/IP servers using FT-TCP. Proceedings of IEEE International Conference on Dependable Systems and Networks (DSN). IEEE Computer Society Press:

Los Alamitos, CA, 2003; 22–26.

18. Candea G, Cutler J, Fox A. Improving availability with recursive microreboots: A soft-state system case study. Performance Evaluation Journal 2004; 56(1–4):213–248.

19. Candea G, Fox A. Crash-only software. Proceedings of the 9th Workshop on Hot Topics in Operating Systems (HotOS-IX), Lihue, Hawaii, June 2003; 67–72.

20. Chawathe Y, Brewer EA. System support for scalable and fault tolerant Internet service. Proceedings of the 1998 IFIP International Conference on Distributed Systems Platforms and Open Distributed Processing (Middleware ’98), The Lake District, U.K., September 1998.

21. Barham P, Dragovic B, Fraser K, Hand S, Harris T, Ho A, Neugebauer R, Pratt I, Warfield A. Xen and the art of virtualization. Proceedings of the 19th ACM Symposium on Operating Systems Principles, October 2003. ACM Press:

New York, 2003; 164–177.

22. ACME Laboratories. http load—Multiprocessing HTTP Test Client.

http://www.acme.com/software/http load/ [October 2006].

23. Kegel D. The dkftpbench Benchmark. http://www.kegel.com/dkftpbench/ [January 2007].

24. Wessels D. Squid Web Proxy Cache. http://www.squid-cache.org/ [January 2007].

25. Morrissey J, Renner M, Roesen D, Saunders TJ. The Proftpd Software. http://www.proftpd.org/ [January 2007].

26. Mindcraft Inc. WebStone: The Benchmark for Web Servers.

http://www.mindcraft.com/benchmarks/webstone/ [January 2007].

27. Maltz D, Bhagwat P. TCP splicing for application layer proxy performance. IBM Research Report 21139, Computer Science/Mathematics, IBM Research Division, March 1998.

28. Microsoft Corporation. Windows 2000 clustering: Performing a rolling upgrade. Windows 2000 Technical Resources, 2000.

Available at:

http://www.microsoft.com/technet/prodtechnol/windows2000serv/deploy/rollout/rollupgr.mspx [January 2007].

29. Lowell DE, Saito Y, Samberg EJ. Devirtualizable virtual machines enabling general, single-node, online maintenance.

Proceedings of the 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2004), Boston, MA, October 2004; 211–223.

30. Clark C, Fraser K, Hand S, Hansen JG, Jul E, Limpach C, Pratt I, Warfield A. Live migration of virtual machines.

Proceedings of the 2nd USENIX Symposium on Networked Systems Design and Implementation (NSD ’05), Boston, MA, May 2005.

31. Nelson M, Lim BH, Hutchins G. Fast transparent migration for virtual machines. Proceedings of USENIX 2005 Annual Technical Conference (USENIX ’05), Marriott Anaheim, CA, April 2005; 391–394.

32. Kephart JO, Chess DM. The vision of autonomic computing. Computer Journal 2003; 36(1):41–50.

33. Brown A, Patterson DA. Undo for operators: Building an undoable e-mail store. Proceedings of USENIX Annual Technical Conference, June 2003; 1–14.

34. Hsu ST, Chang RC. Continuous checkpointing: Joining the checkpointing with virtual memory paging. Software: Practice and Experience 1997; 27(9):1103–1120.

in Operating Systems, Dourdan, France, September 1992; 86–91.

36. Li CCJ, Fuchs WK. CATCH-compiler-assisted techniques for checkpointing. Proceedings of the 20th Annual International Symposium on Fault-Tolerant Computing, June 1990; 74–81.

37. Long J, Fuchs WK, Abraham JA. Compiler-assisted static checkpoint insertion. Proceedings of the 22nd Annual International Symposium on Fault-Tolerant Computing, Boston, MA, July 1992; 58–65.

38. Plank JS, Beck M, Kingsley G, Li K. Libckpt: Transparent checkpointing under UNIX. Proceedings of USENIX Winter 1995 Technical Conference, New Orleans, LA, January 1995; 213–223.

39. Sultan F, Bohra A, Smaldone S, Pan Y, Gallard P, Neamtiu I, Iftode L. Recovering Internet service sessions from operating system failures. IEEE Internet Computing 2005; 9(2):17–27.

40. Osman S, Subhraveti D, Su G, Nieh J. The design and implementation of Zap: A system for migrating computing environments. Proceedings of 5th USENIX/ACM Symposium on Operating Systems Design and Implementation (OSDI

’02), Boston, MA, December 2002; 361–376.

41. Su G, Nieh J. Mobile communication with virtual network address translation. Technical Report CUCS-003-02, Columbia University, February 2002.

42. Ling BC, Kcman E, Fox A. Session state: Beyond soft state. Proceedings of the 1st USENIX/ACM Symposium on Networked Systems Design and Implementation (NSDI ’04), San Francisco, CA, 2004; 295–308.