5.1 Conclusion
This thesis discusses Web automation applications and the common needs of a Web automation creation solution. On the basis of the common needs, this thesis proposes a general platform to create Web automation applications, without being limited to any specific application.
A general Web automation creation solution should consider the follow issues:
z Interoperability: the core concept of Web automation applications is to reuse existing Web resources. From the interoperation aspect, Web automation applications can adopt standard interoperation protocols (such as OAI, Z39.50), or rely only on the common HTTP protocol. Standard interoperation protocols are not supported by many Web resources; on the other hand, using the common HTTP protocol gives access to all Web resources, but problems such as complexity of Web sites and volatility of the interfaces need to be solved.
z Parallel processing: Web automation applications usually need to interact with several Web resources at the same time.
z Server-side or client-side Web automation execution: execution in server side has the advantage of easier deployment, maintenance and management, but puts limit to performance scalability. Execution in client side gives the contrary.
z Flexible presentation: a flexible interface is necessary so that the tool can fulfill different design needs.
z Integration with corporate systems: Web automation applications may need to work with enterprise servers.
z Scheduling: Web automation tasks are usually repetitive and thus may need scheduled execution.
z Integrated development environment: modern application creation tools provide integrated debugging, wizards and others.
z Intelligent tools: Web automation applications are to replace human labor, so intelligence is a desirable characteristic of an agent.
Feature Support by WIS
Interoperability Needs HTML and HTTP only; data extraction facility Parallel processing Multiple session, multiple context
Server-side or client-side
Mix the advantages of both server and client approaches Flexible
presentation
HTML interfaces; round-trip free by using DHTML Integration with
Table 5-1: Design considerations and WIS support
5.2 Future Work
Web automation applications are useful both for common users and enterprises. For common users, Web automation applications can save time and make the use of the Internet more efficient and pleasant. For the enterprise, Web automation can reduce administration costs and human mistakes. The variety of Web automation applications seems to be unlimited, and the applications that can be created by WIS are only limited by the creativity of developers.
Most Web automation applications are built from scratch, which is the main
obstacle to its popularity. WIS aims to turn the creation of Web automation applications easier, but there is still much more to do.
An integrated development environment is a good target for the next step, which gives the possibility of massive creation of Web automation applications. The software industry has benefited much from IDEs, and so can be the field of Web automation applications, pushing us to the era of “service reuse”. There are several issues that can be considered in an IDE for Web automation application creation:
z Integrated debugger for easier troubleshoot of applications.
z Authoring tool: editors and GUI tools with drag-and-drop capability.
z Wizards: there are some related works mentioned in Chapter 2 that emphasizes the creation of Web automation tasks by learning the surfing steps from the user’s interaction with the source. But it is doubtful that the computer can realize everything that the user has done because the complexities of Web applications, which may contain scripts with dynamic content that are not so easy to be detected. So implementing this facility as a wizard is a viable solution, which serves as a starting point for the creation phase. The developer can then make changes to the generated code to fix the incorrect parts.
Web automation applications are related with agents to work in place of human interaction, which in many cases need some sort of intelligence. For example, intelligent metasearchers should analyze returned results and organize them by reranking and summarizing. These intelligent features are usually designed for specific situations, a characteristic that keeps them out of the API set of WIS. With application specific intelligence, WIS will be no more a common client; having different flavors of WIS is unfavorable for wide deployment, so it is better implement
them using component technologies such as Java which can be downloaded by the application when needed. Intelligent tools of common purpose for Web automation applications are a matter of future research.
BIBLIOGRAPHY
[1] V. Anupam, F. Juliana, K. Bharat, L. Daniel, “Automating Web Navigation with the WebVCR”, Computer Networks, Volume: 33, Issue: 1-6, pp. 503-517, June 2000.
[2] A. Banerjee, A. Corera, Z. Greenvoss, A. Krowczyk, C. Nagel, C. Peiris, T.
Thangarathinam, B. Maiani, C# Web Services: Building Web Services with .NET Remoting and ASP.NET, Wrox, 2001.
[3] D. Box, Essential COM, Addison Wesley Longman, 1998.
[4] J.E.F. Friedl, Mastering Regular Expressions, O’Reilly, 2002
[5] E. Harmon, Delphi COM Programming, Macmillan Technical Publishing, 2000.
[6] E. Selberg, O. Etzioni, “The Metacrawler Architecture for Resource Aggregation on the Web”, IEEE Expert, pp.11-14, Jan.-Feb. 1997.
[7] M.W. Spalti, “Finding and Managing Web Content with Copernic 2000”, Library Computing, Westport, pp. 217-221, Volume 18, no. 3, September 2000.
[8] A. Sugiura, K. Yoshiyuki, “Internet Scrapbook: Automating Web Browsing Tasks by Programming-by-Demonstration”, Computer Networks and ISDN Systems, Volume: 30, Issue: 1-7, pp. 688-690, April 1998.
[9] B. Krulwich, “Automating the Internet – Agent as User Surrogates,” IEEE Internet Computing, Volume: 1, Issue: 4, pp 34-38, July-Aug. 1997.
[10] M.G.. Wales, “WIDL: Interface Definition for the Web”, IEEE Internet Computing, Volume 3, Issue 1, Jan.-Feb. 1999.
[11] W.H. Yen, M.J. Hwang, H.R. Ke, “Integrated Search of Digital Library”, Proceedings of 2000 Taiwan Area Network Conference, pp.484-491, October 2000.
[12] WIDL, http://www.w3.org/TR/NOTE-widl-970922
[13] SOAP, http://www.w3.org/TR/SOAP/.
[14] Web Services Description Language, http://www.w3.org/TR/wsdl [15] Web Services Activity, http://www.w3.org/2002/ws/
[16] Microsoft Scripting Technologies, http://msdn.microsoft.com/scripting/.
[17] Readerware, http://www.readerware.com/rwreviews.html [18] Meta-Search Engines,
http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/MetaSearch.html [19] BestBookDeal, http://www.bestbookdeal.com
[20] BestWebBuys, http://www.bestWebbuys.com [21] Readerware, http://www.readerware.com
[22] Metalib, http://www.exlibrisgroup.com/metalib.htm
[23] A Gentle Introduction to XML, http://www.tei-c.org/P4X/SG.html [24] Scripting Technologies, http://msdn.microsoft.com/scripting/default.asp [25] “Document Object Model (Core) Level 1”, World Wide Web Consortium,
http://www.w3.org/TR/2000/WD-DOM-Level-1-20000929/
[26] “Document Object Model (Core) Level 2”, World Wide Web Consortium, http://www.w3.org/TR/2000/REC-DOM-Level-2-Core-20001113/
[27] MSDN Home Page, http://msdn.microsoft.com [28] Consortium on Core Electronic Resources in Taiwan,
http://www.stic.gov.tw/fdb/index.html