Chapter 5 Evolutional Knowledge Acquisition
5.4 Adjusting Certainty Factor of Collaborative Dynamic Knowledge
Where g(t), the Continuous Events Accumulating Function given in formula (5.4), is used to record the number of consecutive “1” or consecutive “0” received at time t.
5.4 Adjusting Certainty Factor of Collaborative Dynamic Knowledge
Since Dynamic EMCUD can be extended to the collaborative framework, each discovered dynamic knowledge learned from different local KBSs should be integrated to further applied. Three cases are used to assist experts in adjusting the CF values of the discovered knowledge of the new evolved objects from the collection of inference logs. Assume there are n local KBSs and each new evolved object may be discovered in p local KBSs, different CF values of a given embedded rule could be generated in each KBS. For p > 0, the CF Adjusting Function shown in formula (5.4) is proposed to help experts obtain the average of different CF value of a given embedded rule in each local KBS and adjust the scale of the CF increment or decrement (∆CF) according to the discover of the new object in the collaborative KBS.
For each new embedded rule Ri, let the CF value be CF(Ri) and let the CF(Rij)
Depending on whether the new objects are discovered in the collaborative KBS or not, the coefficient δ can be defined as follows.
Case 1: the new object can be discovered in the collaborative KBS.
δ is set to p/n.
Case 2: the new object can not be discovered in the collaborative KBS.
δ is set to (p-n)/n.
For p = 0, since the new object can not be discovered in any local KBS, the new object could be discovered in the collaborative KBS according to the correlations of profiles. Therefore, the CF Adjusting Function could be reduced to formula (5.5),
where the CF(Ric) is the CF value of the new discovered rule in the collaborative KBS due to different configurations of profile.
. CF(Ric)=CF(Ric)+δ×∆CF (5.5) δ is set to -2.
5.5 Experiments
Up to now, many antivirus products have been developed to discover worms, virus or Trojan horse in a computer system. However, these products are hard to
automatically discover the variants of worms because the signature based approach fails when the signatures are changed. To overcome the weakness, we propose a worm detection prototype system, which has neo-learning module, to enhance the ability of commercial antivirus products by the collaborative framework. The case of worm detection is given to illustrate the idea of TEA. First, the domain ontology construction flow will be described and then Nimda worm is used as an example.
5.5.1 Computer Worm Ontology Construction
One of the purposes of applying ontology is to provide domain of discourse that is understandable by human and computers. Since ontology can be represented by machine readable markup languages such as RDF, the knowledge can be shared for different knowledge bases automatically through computers processing. Moreover, the reusability of ontology has become increasingly important to developers of intelligent systems.
Figure 5.3 Worm Ontology Construction Flow
In this experiment, the ontology is not only reusable but also adaptive to the current environment. Also, we construct the ontology based upon a concept tree consisting of several prior knowledge including skeletal model and real cases provided by knowledge engineers and domain experts in TEA. In Figure 5.3, the flow of constructing worm ontology is illustrated to help experts construct the ontology more easily, the following four Steps are proposed:
Step 1: Skeletal model construction.
Create the skeletal worm model by identifying each worm with six general attributes including the basic information, the service, the exploitation, the carrier, the symptoms and the defense instruction.
Step 2: Concept tree construction.
Since it is often easier and more accurate for experts to provide critical cases rather than domain ontology, the power of critical cases described in terms of relevant objects and attributes to build domain ontology is remarkable. Therefore, after case diagnosing a concept tree is created based upon the skeletal model in Step 1.
Step 3: Concept tree transformation.
After concept tree is created, it is transformed into AOT, and attribute ordering will be next acquired from experts. Then the original EMCUD can be processed to generate the initial rules. However, because it is not easy to identify some attribute ordering values precisely, the attribute which is uncertain to identify the ordering value should be traced and analyzed with time by constructing an AST. Each attribute signal is recorded in each time interval, when the attribute appears important the
signal equals one and when it appears unimportant the signal equals zero. Therefore, the attribute ordering table will be reconstructed according to the attribute signals collected with time.
Step 4: Merge procedure.
Two relations “has” and “is” are used for constructing worm ontology during the merge procedure in this paper. The relation of “has” includes attribute ordering value, for example, when the attribute ordering value equals 3 then the relation should be
“HAS:3”. Therefore, from Step 3, the ordering value would be retrieved from the reconstructed AOT by AST. Hence, the ontology can be easily transformed into AOT with updated value in Dynamic EMCUD whenever the variants are discovered.
5.5.2 Example of Nimda Worm Detection
Nimda, an incredibly sophisticated worm that made headlines worldwide, is taken as an example. Nimda is the first worm to modify existing web sites to start offering infected files for download by using Unicode exploit to infect IIS web server. It is the first worm to use normal end user machines to scan the vulnerable web sites. This technique enables Nimda to easily infect intranet web sites located behind firewalls.
Assume a simple Nimda concept tree is created in Figure 5.4 after series of Nidma cases diagnosis, and it can be transformed into a worm AT like Table 5.1. The following attributes are considered: the name of the e-mail attachment used by worms, the medium used by worms to upload, and the name of the file used by worms to start execution on servers. After constructing the worm AT, we construct the initial AOT shown in Table 5.2.
Nimda
Mail_Attachment Upload_Medium Excuted_File_Name
Symptoms Carrier
Readme.exe puta!!.scr Admin.dll cool.dll
Figure 5.4 Example of Initial Nimda Concept Tree
Table 5.1 An Example of Original Nimda AT Onject
Attribute
Nimda
Mail_Attachment Readme.exe Upload_Medium Admin.dll Executed_File_Name Riched20.dll
Table 5.2 An Example of Original Nimda AOT Object
Attribute
Nimda
Mail_Attachment 2
Upload_Medium 3
Executed_File_Name 4
With both AT and AOT, the EMCUD can be processed to generate eight embedded rules and some of them have low CF value such as rule R1: “IF Not Mail_Attachment = Readme.exe and Upload_Medium = Admin.dll and Executed_File_Name = Riched20.dll Then Nimda” with CF value = 0.67. Therefore, suppose that in the inference process, the rule R1 above is learned by neo-learning module almost all the time during a period, and suppose in the last two time points the embedded rule R2: “IF Not Mail_Attachment = Readme.exe and Not Upload_Medium
= Admin.dll and Not Executed_File_Name = Riched20.dll Then Nimda” with CF value = 0.4 is fired, the AST in Table 5.3 to record the evolutional trend can be obtained.
Suppose that Nimda is the latest worm occurred in the world, its ordering value of each attribute can not be easily determined because its variants may soon be broken out. The expert may define an AST with several time points, and then assign 0 in the first attribute, N1, at first time point in Table 5.3. The attribute event N2 at the second time point is set to zero. For simplified discussion, we use gracefully accumulating function to adjust the AOT value of each attribute to each object according to the AST.
Table 5.3 An Example of Nimda AST Object
Attribute
N1 N2 N3 N4 N5 N6 N7
Mail_Attachment 0 0 0 0 0 0 0
Upload_Medium 0 1 1 1 0 0 0
Executed_File_Name 1 0 0 0 1 0 0
In Table 5.3, the Mail_Attachment attribute is calculated by Function 5.2, and the attribute is assigned a new ordering value = 1 since it is very possible to be changed again, subsequently, ordering value = 3 are assigned for both attributes Upload_Medium and Excuted_File_Name according to the AST. Therefore, the CF value of the rule R1 is leveled up from 0.67 to 0.74. Moreover, several new attribute-values are learned by neo-learning module with Mail_Attachment = puta!!.scr in R1, a new worm variant Nimda.B shown in Table 5.4 can be integrated into Table 5.5, and also an AOT is updated as shown in Table 5.6. Moreover, the Nimda ontology after discovering Nimda.B is updated as Figure 5.5.
Table 5.4 An Example of Updated Nimda AT After Discovering Nimda.B Object
Attribute
Nimda.A Nimda.B
Mail_Attachment Readme.exe puta!!.scr Upload_Medium Admin.dll Admin.dll Executed_File_Name Riched20.dll Riched20.dll
Table 5.5 An Example of Integrated Nimda AT Object
Table 5.6 An Example of Updated Nimda AOT After Discovering Nimda.B Object
Figure 5.5 The Updated Nimda Ontology after Discovering Nimda.B
Therefore, with the accumulated inference logs from distributed sensors, the TEA can also update the knowledge frequently. Assume VODKA learns another new attribute values including Mail_Attachment = sample.exe, Upload_Medium = cool.dll, and Executed_File_Name = httpodbc.dll in R while the rule R has always been fired
in each time point in a short period, then a new variant Nimda.E is found. Finally, based upon the updated tables shown in Table 5.7 and Table 5.8, the built system will give a whole picture of worms to guide the users who are not familiar in the domain for preventing or removing the malicious worms. Finally, the updated tables are shown in Tables 5.7 and 5.8, and the detailed of ontology of Nimda could be also updated as Figure 5.6.
Table 5.7 An Example of Integrated Nimda AT After Discovering Nimda.E Object
Attribute
Nimda
Mail_Attachment {Readme.exe; puta!!.scr; sample.exe}
Upload_Medium {Admin.dll; cool.dll}
Executed_File_Name {Riched20.dll; httpodbc.dll }
Table 5.8 An Example of Updated Nimda AOT After Discovering Nimda.E Object
Figure 5.6 The Updated Nimda Ontology after Discovering Nimda.E
Owing to the different background and dynamic knowledge which can change with the times, the domain knowledge constructed at a time may become degraded in the near future. In this chapter, we propose a new knowledge acquisition method, called TEA, which traces information with times by interacting with human experts and supported by the learning strategy of VODKA to efficiently update the time-related domain knowledge according to the current environment. Therefore, we enrich the knowledge base and ease the effort of constructing the domain knowledge which is changing with the times and environment. Three cases will be used in collaborative framework to assist experts in adjusting the CF values of the discovered knowledge of the new evolved objects from the collection of inference logs. A worm detection system is illustrated to ease the experts’ efforts from analyzing and learning and to help retrieving meaningful information for making proper decisions since the knowledge bases become more adaptive for a changing environment by using TEA.
Chapter 6
Application in Worms and DDoS Detection
A Worm Immune Service Expert system (WISE) with Dynamic EMCUD and a worm classification embedded rule base is implemented to discover the new variant worms generated by the attacking traffic generator in the experimental environment to evaluate the performance of our proposed method. A DDoS intrusion tolerance system is also implemented.
6.1 The Background of Worms and DDoS Attack
6.1.1 The Introduction of Computer Worms
In recent years, computer worms are grown dramatically to influence the wide computer networks due to the property of easily modifying the source code of original computer worms to create new variant for escaping the detection of related systems, e.g., Symantec Norton [72], Network Viruswall [74], etc. Generally speaking, computer worm usually self-propagates through the following four stages: Target selection, Exploitation, Infection, and Propagation [80]. In Target Selection Stage, a worm performs reconnaissance and simply probes potential victim to see if it's running a service on a particular port. If the service is running, the worm goes to Exploitation Stage, in which a worm compromises the target by exploiting a particular vulnerability and published exploits. If success, the worm goes to Infection Stage, in which the worm sets up on the newly infected machine. Finally, in Propagation Stage,
the worm starts to spread by choosing new targets. And another victim will enter the next four Stages cycle.
6.1.2 The Introduction of DDoS Intrusions
As mentioned above, many different DDoS attacking tools and defending methods [17] to help mitigate the malicious traffic developed result in the rapid growth of complicated characteristics of DDoS intrusion tolerance in recent years.
The introduction of DDoS is given in Appendix A.
As we know, there are two different types of attack technique in DDoS attacks:
bandwidth consumption and resource consumption. The bandwidth consumption means that the attacking traffic launched by the compromised hosts, which are controlled by attackers, is aggregated to a single huge flood and overwhelms the victim. The resource consumption means that attackers make use of the leak of the network protocol or the system security such as the techniques of SYN flood, land and Teardrop, resulting in the starvation of system resources [16].
As the DDoS attack tools have become more complicated in recent years, the maintenance of the characteristics of DDoS attacks is becoming more difficult despite the previously known common characteristics of each category of discovered DDoS attacks. Therefore, we will propose a knowledge base to store the characteristics of DDoS attacks, which may be obtained by analyzing the traffic behaviors of the DDoS attacking tools, for DDoS intrusion tolerance. Besides, two criteria considering the difference between two types of DDoS attacks will be proposed to evaluate the degree of intrusion tolerance.
Intrusion tolerance is the ability of a system to continue providing (possibly degraded but) adequate services after a penetration [70]. As mentioned above, it is very hard to detect and prevent the DDoS attacks. Therefore, the intrusion tolerance of DDoS attacks is an important issue to mitigate the damage during DDoS attacks for providing the critical services continuously on Internet. Although a variety of methods, which are given in Appendix A, have been proposed to mitigate the damage during DDoS attacks for providing the critical services continuously, it is still very difficult to keep up with the rapid growth of DDoS expertise in their studies. To solve this problem, a DDoS ontology is proposed to provide a common vocabulary among domain experts and an integrated knowledge acquisition framework is then proposed to assist in quickly accumulating their expertise. We also use the behaviors of access control list to evaluate the performance of the DDoS models.
6.2 The Framework Worm Immune Service Expert System
As we know, many antivirus products have been proposed to discover worms, virus or Trojan horse in a computer system. Although these antivirus softwares are developed to protect our system well, it is hard to automatically discover the variant worms without updating their signature database because the signature the worm signatures may change over times. To overcome the weakness, the worm detection prototype system, namely WISE, is proposed to enhance the commercial antivirus products instead of replacing them. WISE is a knowledge-based system. Unlike pattern matching system, it does not need to write the program again, and therefore is suitable for worm, which is usually variant quickly that updates knowledge base frequently. By only updating the knowledge base, WISE can modify the defense me
chanism for the variants of worm; as a result, the system can be easily maintained.
Besides, WISE contains embedded meanings of knowledge, so it can easily capture some variant worms that in order to avoid signature-based detection system to modify characteristic less. Since the growth of the knowledge of worms is very fast, we propose a collaborative architecture for the adaptive worm detecting problem.
Figure 6.1 The Collaborative Framework for Worm Detection
Figure 6.1 shows the collaborative framework for worm detection. In the architecture, each worm sensor provides a web interface to collect or discover all the symptoms of worm cases by user and scanning tools. The NEO-learning module helps each worm sensor constructing AST to reconstruct AOT increment and update main acquisition table using AT increment (monitoring the frequent inference logs of weak embedded rules of worms with the times), where each sensor has its own Worm KB.
For example, when worm infects a victim system, the user can scan the host computer by some general antivirus software or can call for help from the Internet. The system
collects all the information and infers the information based upon the worm knowledge with embedded meaning constructed by EMCUD. Consequently, the result of inferring will be passed to the users to teach the way of recovering the system.
Moreover, the statuses which satisfy certain embedded rules will be considered to learn the new knowledge of new variant worms by neo-learning module. By collecting the new worms knowledge and infrequent inference logs and consulting the Profile, the collaborative framework can integrate the new worm knowledge.
In our WISE system, the knowledge of computer worms can be divided into several KCs, including the service provided by host may be infected by certain worms and then produced some symptoms in host or network. Some worm lifecycle, Profile model, and dynamic behaviors knowledge classes are also created. The Log Collecting Stage will be encoded by four meta-rules in DRAMA; the Knowledge Learning Stage and Dynamic EMCUD are implemented using the JSP to make a communication channel using the API provided by DRAMA. The related attributes of various computer worms can be collected by some probe tools and used to evaluate the ability of Dynamic EMCUD, which deployed in the prototype system.
Figure 6.2 The Experimental Environment for Detecting Computer Worms
In order to evaluate the WISE, an experimental environment shown in Figure 6.2 for detecting various computer worms is built. In this environment, the victim is received both the normal traffic and the attacking traffic (various worm behaviors).
All received traffic can be treated as normal or attacking behavior, which can be transformed as attribute-value pairs. The network traffic collected from Internet is assumed as normal traffic since most attacking behaviors with significant signatures will be filtered by firewall. The attacking traffic generator is designed to randomly generate various worms attacking traffic to infect the victim. Besides the attacking traffic, some signatures, e.g., the system status, host vulnerability information, and large e-mailing behavior, of the victim infected by worms can be also collected. The probe, such as Nessus [73], is also used to automatically collect these worm related attributes (symptoms). Then, these attributes is used to trigger the corresponding classification rules in worm KB. If variant worms occurred frequently in a period, some candidate worm variants may be discovered by Dynamic EMCUD. Finally, the corresponding embedded rules of variant worms confirmed by experts will be generated to update the worm KB.
6.3 DDoS Intrusion Tolerance
As we know, the traditional methods for detecting and filtering DDoS attacks [27]
are monitoring the status of network and system, specifying the alert thresholds,
are monitoring the status of network and system, specifying the alert thresholds,