Simulation Results and Analysis

In this section, each performance of the proposed mechanism in two main cases (e.g. honest and dishonest environment) is presented and analyzed in the following.

(1) Honest Environment (Case 1)

There are two performance metrics will be analyzed in such case. They are trust value and detection accuracy and the results of them are in the following.

 Trust value

In the proposed mechanism, with fixing the multiple parameters ( =0.5, =0.01, =3, =1) and multi values of , the trust values for other simulators in three different types of nodes (OSSEC, snort and bro) are illustrated as follows.

Figure 10. Average trust values for three types of simulators in OSSEC ( =1)

0 0.2 0.4 0.6 0.8 1

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96

trust value

amount of test messages

OSSEC snort bro

33

Figure 11. Average trust values for three types of simulators in snort ( =1)

Figure 12. Average trust values for three types of simulators in bro ( =1)

Figure 13. Average trust values for three types of simulators in OSSEC ( =0.1)

0

34

Figure 14. Average trust values for three types of simulators in snort ( =0.1)

Figure 15. Average trust values for three types of simulators in bro ( =0.1)

After observing the curve of the average trust values among 100 test messages, it shows that with the decreasing in , trust values for three types of simulators are gathering together at 0.8. But decreasing represents that it mitigates penalty when a node replying “do not know.” In addition, the same causes different penalties about the trust value in different mechanism (the proposed one and Fung et al.), shown in Figure 16 and Figure 17. With ensuring the same and enough severity of penalty about replying “do not know,” the values of are chosen to be 0.1 and 1 in this mechanism and Fung et al. separately. Both conditions show that the trust values are approaching to 0.5 from 1 linearly.

35

Compared to the results in Fung et al. in the following, the trust values for different types of simulators are clustered into two groups. The group of same type with the observing node gains higher trust values than the different ones. On the contrary, the proposed mechanism eliminates the difference of trustworthiness between each node. Thus all simulators will not only consult the same types of nodes but also the different types. It can avoid forming some separate groups and make different types of simulators assist each other. In such way, collaboration can be applied among all simulators.

Figure 16. T and AW convergence curve with “do not know” responses

Figure 17. T convergence curve with “do not know” responses (Fung et al.)

-0.1

36

Figure 18. Trust values for others in OSSEC ( =1, Fung et al.)

Figure 19. Trust values for others in snort ( =1, Fung et al.)

Figure 20. Trust values for others in bro ( =1, Fung et al.)

0

37

If keep decreasing the value of , the trust values in both mechanisms will gather together. But it nearly relinquishes the adoption on the trust value when a node replying “do not know” responses. E.g. the trust value for a node will not be lower than 0.8 until the percentage of replying “do not know” is over 90% when =0.01 in this mechanism or =0.1 in Fung et al. Thus a node can maintain its high trust value without any contribution to others by keeping replying “do not know.” So it is not proposed to set the value of too low, even though this can help eliminating the difference of trustworthiness between each node.

 Detection Accuracy

In the proposed mechanism, with fixing the multiple parameters ( =0.1, =0.1, =0.2) and the multi values of and , detection accuracy is observed in the three different types of nodes, after the nodes detecting 5,000 events in each. In addition all nodes passed through learning phase, which represents that the nodes send 100 test messages to each other before detecting events. The results in both the proposed mechanism and Fung et al. are illustrated in the following figures.

Figure 21. Detection accuracy under different after learning ( =6)

p=0.2 p=0.5

p=0.8 70.000%

80.000%

90.000%

100.000%

OSSEC

snort

bro

detection accuracy

38

Figure 22. Detection accuracy under different after learning ( =6, Fung et al.)

Figure 23. Detection accuracy under different after learning ( =3, Fung et al.)

Figure 24. Detection accuracy under different after learning ( =1, Fung et al.)

p=0.2

39

Figure 25. Detection accuracy under different ( =6, non-cooperation)

Figure 26. Detection accuracy under different ( =3, non-cooperation)

Figure 27. Detection accuracy under different ( =1, non-cooperation)

p=0.2

40

Generally malicious attacks occur very often, so the nodes with lower have higher probability to detect those attacks. Similarly the nodes detect more attacks in the environment with higher .

Compared to Fung et al. after learning phase, the proposed mechanism performs much better than it. Even though the value of is set to be 6 in the proposed mechanism, it outperforms Fung et al. when the value of is set to be 1. The reason for such phenomenon is that each node has sufficient time to construct the symptom table in the proposed mechanism when passing through learning phase. A node has ability on detecting an attack without contacting it before. In addition, the nodes in both collaborative mechanisms outperform the non-cooperation conditions.

And the following figures show that the detection accuracy in the proposed mechanism without passing through learning phase. Although the detection accuracy in the nodes without learning is not as high as the one after learning phase, it still outperforms others (Fung et al. and non-cooperation) mentioned above.

Figure 28. Detection accuracy under different without learning ( =6)

p=0.2 p=0.5

p=0.8 70.000%

80.000%

90.000%

100.000%

OSSEC

snort

bro

detection accuracy

41

Figure 29. Detection accuracy under different without learning ( =3)

Figure 30. Detection accuracy under different without learning ( =1)

(2) Dishonest Environment (Case 2)

Similarly there are two performance metrics will be analyzed in such case. They are trust value and detection accuracy and the results of them are in the following.

 Trust Value

In this proposed mechanism, with fixing the multiple parameters ( =0.5, =3, =0.1), and the multi values of and , the trust values for three different

42

illustrated in the following figures. All dishonest nodes passed through 100 times test messages while being honest. Moreover, the trust value when the dishonest nodes replies fake response in second times is presented in particular. Thus it is simple to perceive the efficiency and effectiveness on the penalties about a node replying fake responses.

43

44

45

Figure 39. Trust values for three dishonest nodes in bro ( =0.2, =0.1)

After observing the curve of trust values for three dishonest nodes among 30 fake responses, it shows that with the decreasing in value of from 1 to 0.2, the lowest trust values decrease from 0.2 to 0. And with the increasing in from 0.01 to 0.1, it shows that the trust values decrease little rapider. Totally the trust values for three dishonest nodes decrease to the values lower than 0.5 in 5 times.

Compared with results in Fung et al. in the following, they give different penalties to different types of nodes. For the dishonest node of the same type, it obtains server penalty than the different type.

Figure 40. Trust values for three dishonest nodes in OSSEC (Fung et al.)

0 0.2 0.4 0.6 0.8 1

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

trust value

amount of fake responses

OSSEC snort bro OSSEC:

snort:

bro:

0.6667 0.6528 0.6348

46

Figure 41. Trust values for three dishonest nodes in snort (Fung et al.)

Figure 42. Trust values for three dishonest nodes in bro (Fung et al.)

The reason for the trust values for dishonest nodes in both mechanisms do not decrease to the lowest value immediately is that they do not only consider one satisfaction value each time. Other honest feedback before can comprise the dishonest feedback in the beginning. Moreover, in the proposed mechanism a node considers the feedback of other nodes about the dishonest nodes. Other nodes probably do not receive the fake responses yet, so they provide high trust values about the dishonest nodes. Thus the trust values for dishonest nodes decrease more slowly than Fung et al., but it can be improved by increasing .

47

 Detection Accuracy

In the proposed mechanism, with fixing multiple parameters ( =0.2, =0.1, =0.1) and multi values of , and amount of dishonest nodes, detection accuracy in the three different types of honest nodes is observed after each node receiving 1,000 events. Moreover, all the nodes do not pass through learning phase in such case. The results are illustrated in the following.

Figure 43. Detection accuracy under different with 3 dishonest nodes ( =6)

Figure 44. Detection accuracy under different with 15 dishonest nodes ( =6)

p=0.2

48

Figure 45. Detection accuracy under different with 27 dishonest nodes ( =6)

Figure 46. Detection accuracy under different with 3 dishonest nodes ( =3)

Figure 47. Detection accuracy under different with 15 dishonest nodes ( =3)

p=0.2

49

Figure 48. Detection accuracy under different with 27 dishonest nodes ( =3)

Although with the dishonest nodes the detection accuracy is decreased, totally the proposed mechanism in the dishonest environment still outperforms Fung et al.

and non-cooperation in the honest environment even 90% of nodes are dishonest.

These results prove that it is a robust mechanism. But this performance is based on the rest of honest nodes are different types, since only different types of nodes can share different information with each other. If the rest of honest nodes are in the same type, it is helpless to increase the performance. Because of that all the nodes have the same little information.