1This kind of attack is called a snowblind attack.
I
NTRODUCTION
During the last several years, malicious traffic detection has been an active area of network security because the Internet has witnessed a surge in malicious traffic generated by network attacks, such as denial of service (DoS), and propagation of botnets, viruses, worms, trojan
horses, spyware, and so on. Moreover, malicious traffic makes network performance inefficient and troubles users. For example, distributed DoS (DDoS) attacks increase Domain Name Service (DNS) latencies by 230 percent and web latencies by 30 percent [1]. During July–August 2001, 395,000 computers were infected world-wide with the CodeRed worm, which resulted in approximately $2.6 billion in damages [1].
There are a multitude of malicious traffic detection techniques, and thus, vulnerabilities in common security components, such as firewalls, are unavoidable. Intrusion detection systems (IDSs) and intrusion prevention systems (IPSs) are commonly used today. They are used to detect different types of malicious traffic, net-work communications, and computer system usage with the mission of preserving systems from widespread damage; that is because other detection and prevention techniques, such as firewalls, access control, skepticism, and encryp-tion have failed to fully protect networks and computer systems from increasingly sophisticat-ed attacks and malware [2, 3].
An IDS/IPS monitors the activities of a given environment and decides whether these activities are malicious or normal based on system integri-ty, confidentiality and the availability of informa-tion resources. As soon as a malicious or an intrusive event is detected, the IDS produces a relative alert and passes it to the network admin-istrator promptly while the IPS not only executes what the IDS does but also blocks network traf-fic from the suspected malicious source. Howev-er, there is no “perfect” detection approach, which can always correctly distinguish between malicious and normal activities. In other words, IDSs/IPSs can identify a normal activity as a malicious one, causing a false positive (FP), or malicious traffic as normal, causing a false nega-tive (FN). FPs and FNs cause several problems. For example, FNs generate unauthorized or abnormal activities on the Internet or in comput-er systems. On the othcomput-er hand, a lot of FPs may easily conceal real attacks1and thus overwhelm
the security operator. When real attacks occur,
A
BSTRACT
False positives and false negatives happen to every intrusion detection and intrusion preven-tion system. This work proposes a mechanism for false positive/negative assessment with
multi-ple IDSs/IPSs to collect FP and FN cases from
real-world traffic and statistically analyze these cases. Over a period of 16 months, more than 2000 FPs and FNs have been collected and ana-lyzed. From the statistical analysis results, we obtain three interesting findings. First, more than 92.85 percent of false cases are FPs even if the numbers of attack types for FP and FN are similar. That is mainly because the behavior of applications or the format of the application content is self-defined; that is, there is not com-plete conformance to the specifications of RFCs. Accordingly, when this application meets an IDS/IPS with strict detection rules, its traffic will be regarded as malicious traffic, resulting in a lot of FPs. Second, about 91 percent of FP alerts, equal to about 85 percent of false cases, are not related to security issues, but to management
pol-icy. For example, some companies and campuses
limit or forbid their employees and students from using peer-to-peer applications; therefore, in order to easily detect P2P traffic, an IDS/IPS is configured to be sensitive to it. Hence, this causes alerts to be triggered easily regardless of whether the P2P application has malicious traffic or not. The last finding shows that buffer over-flow, SQL server attacks, and worm slammer attacks account for 93 percent of FNs, even though they are aged attacks. This indicates that these attacks always have new variations to evade IDS/IPS detection.
T
OPICS IN
N
ETWORK
T
ESTING
Cheng-Yuan Ho, National Chiao Tung University
Yuan-Cheng Lai, National Taiwan University of Science and Technology
I-Wei Chen, Fu-Yu Wang, and Wei-Hsuan Tai, National Chiao Tung University
Statistical Analysis of False Positives and
False Negatives from Real Traffic with
true positives (real alerts) are deeply buried within FPs, so it is easy for the security operator to miss them [4].
Accordingly, a variety of commercial prod-ucts, open source, and research into IDSs were proposed. Wu and Banzhaf [2] provided an overview of different IDS algorithms, such as artificial neural networks, swarm intelligence, evolutionary computation, artificial immune sys-tems, fuzzy sets and soft computing, and their problems. A collaborative intelligent IDS and a fuzzy inference system were proposed to reduce FPs through fuzzy alert correlation in [3] and [5], respectively, while Sourour et al. in [4] reduced both FPs and FNs with their environ-mental awareness intrusion detection and pre-vention system. A system of Attack Session Extraction (ASE) was proposed in [6] to create a pool of traffic traces causing possible FPs and FNs to IDSs. One to two years later, the ASE was expanded into a bigger system, called the
PCAPLib system [7]. The PCAPLib system not
only extracted and classified the real-world traf-fic captured from Campus BetaSite [8] into prop-er categories by levprop-eraging multiple IDSs, but also anonymized users’ privacy in these FP and FN traffic traces out of security considerations. However, previous work only focused on study-ing how to reduce FPs and/or FNs in IDSs or how to collect and extract the FP and FN traffic traces from real-world traffic.
This work collects more than two thousand cases of FPs and FNs from the real-world traffic of Campus BetaSite by the PCAPLib system, in order to observe what kinds of FPs or FNs hap-pen easily in which protocols and in what kind of attacks, and investigate their frequencies across all FPs and FNs. Also, the reasons behind these FP and FN cases for network forensics and trends in malicious traffic attacks are conjec-tured in this work. From statistical analysis results, we find that
• There are 13 times more FPs than there are FNs although the number of attack types in FP and FN are similar
• About 91 percent of FP alerts are not relat-ed to security issues
• Buffer overflow, SQL server attacks and worm slammer attacks account for 93 per-cent of FNs
With this work, application users and developers can understand why the traffic of an application is sometimes blocked by the IPS while the devel-opers of IDS/IPS can pay attention to the men-tioned FN cases, protocols and so on.
The remainder of this article is organized as follows. The effects of FPs and FNs are detailed. The methodology of how to collect and assess FPs and FNs from real-world traffic is described. The experimental environment in this work and statistical analysis are shown. Finally, the last sec-tion concludes this work and outlines future work.
FP
S AND
FN
S
FPs and FNs of the IDS/IPS are mystery terms that describe a situation where the IDS/IPS makes a mistake. The former means that the IDS/IPS triggers an alert when there is no mali-cious activity in the traffic while the latter means
that there is no alert raised by the IDS/IPS when malicious traffic passes through it. FP and FN rates are two metrics important in measuring the accuracy of the IDS/IPS [9].
An FP of the IDS/IPS will not result in an intrusion and it may be caused by two reasons: the detection mechanism of the IDS/IPS may be faulty or the IDS/IPS detects an anomaly that turns out to be benign. Therefore, an FP may cause security analysts to expend unnecessary effort. Moreover, if a hacker launches a
snow-blind attack, the challenge for security analysts is
to somehow identify the real attack amidst the chaff caused by the hacker. This may create a potential vulnerability for the IDS. On the other hand, when an IPS has an FP, the primary con-cern is that legitimate traffic might be blocked. Most organizations consider blocking legitimate traffic as a much more serious problem than generating a false alert. Consequently, an FP of the IPS is a much more serious matter than that of the IDS. If the IPS blocks legitimate traffic a few times, it will be yanked out of the network.
An FN is simply a missed attack, which may put networks or computer systems in danger. Clearly an FN is undesirable, and every vendor strives to provide the most complete coverage possible. However, there is no silver bullet: no product detects all attacks. Hence, the goal becomes providing coverage against high priority attacks. Aside from lack of coverage, several other reasons may also cause an FN. For exam-ple, in order to evade the IDS or IPS, the attack may incorporate obfuscation techniques. Anoth-er possibility is ovAnoth-erwhelming the IDS with traf-fic beyond its processing capacity, so the IDS will drop the packets needed to detect the attack. For an IPS, overwhelming it has a different effect: it causes traffic to be dropped. The attack doesn’t succeed because attack packets are dropped, but it is also not detected. Accordingly, the attack can be tried again.
In practice, for a vendor of IDSs/IPSs, an FN is much more serious than an FP because of neg-ative effects of an FN including reduced trust in the IDS/IPS, and because of damage caused by the intrusion. However, from a user’s point of view, an FP may be more serious than an FN because an FP may cause the IPS to block the user’s benign traffic. In addition, the user may allow some FNs as long as they’re not too fre-quent. Therefore, it is necessary to investigate and analyze FPs and FNs with IDSs/IPSs in detail.
M
ETHODOLOGY
This section first takes a look at the Campus BetaSite and the PCAPLib system (which is the traffic source), and then details how to identify and assess 2000 cases of FPs and FNs for net-work forensics on a set of IDSs/IPSs. Herein, the method of assessing FPs/FNs is called false posi-tive/negative assessment (FPNA).
T
HEC
AMPUSB
ETAS
ITE AND THEPCAPL
IBS
YSTEMAs shown in Fig. 1, the traffic source for the PCAPLib system comes from the Campus Beta-Site deployed at National Chiao Tung
Universi-from a user’s point of view, an FP may be more serious than an FN because an FP may cause the IPS to
block the user’s benign traffic. In addition, the user may allow some FNs
as long as they’re not too frequent. Therefore, it is necessary to investi-gate and analyze FPs
and FNs with IDSs/IPSs in detail.
ty, Hsinchu, Taiwan. The Campus BetaSite is used by developers to test and debug products while maintaining network quality for network users. Moreover, it is an operational network on campus and records network traffic from net-work users into packet capture (PCAP) files. The volume of network traffic on/through the BetaSite is roughly 100 Gbytes/h.
The goal of trace sharing is to preserve real-world traffic behavior in packet traces so that it can be replicated and picked up easily by researchers for network forensics.2To achieve
this goal, the PCAPLib system consists of front-end and back-front-end systems. The front-front-end system not only extracts and classifies valuable packet traces from real-world traffic but also precisely and deeply protects the sensitive information in the packets. This is because recording the entire real-world traffic consumes storage space and searching for specific events within the huge traces is time-consuming. Therefore, recording only traffic associated with specific/special events would be better. Besides, packet anonymization protects privacy from leakage in trace sharing. On the other hand, the back-end system is responsible for storing the extracted PCAP files, whether anonymous or otherwise, and for demonstrating the usefulness of the PCAPLib system in network forensics when used in con-junction with other applications, such as FPNA.
The preprocessing component of the front-end system uses a traffic replay tool (e.g., tcpre-play) to replay captured raw traffic to multiple devices under test (DUTs) to leverage their domain knowledge. If a DUT detects abnormal behavior in the traffic, it will trigger an alert. For the core processing component of the front-end system, there are two mechanisms, Active Trace
Collection (ATC) and Deep Packet Anonymiza-tion (DPA). Based on DUT logs, the ATC finds out the anchor packets that trigger the logs, pro-cesses packets and connection associations to extract each specific/special session into packet traces, and uses supervised classification to cate-gorize the extracted packet traces. On the other hand, the DPA parses application-level protocol identities and anonymizes sensitive fields for pri-vacy protection of packet traces, while still main-taining their usefulness for research.
F
ALSEP
OSITIVE/N
EGATIVEA
SSESSMENT FP and FN rates are two important metrics in measuring the accuracy of a network security system, such as an IDS or IPS. It has been demonstrated that even a small rate (1 in 10,000) of FPs could generate an unacceptable number of FPs in practical detections [7]. The assess-ment is important to IDS/IPS developers trying to optimize the accuracy of detection by reduc-ing both FPs and FNs, because the FP/FN rate limits the performance of network security sys-tems due to the base-rate fallacy phenomenon. The statistical analyses in this work can elucidate the causes and rankings of FPs and FNs, thus allowing developers to avoid similar pitfalls dur-ing their product development.As in previous work [6, 7], the ATC leverages the domain knowledge of the DUTs of intrusion detection/prevention, antivirus, anti-spam and application classifier to collect real-world pack-ets. The detection of DUTs may be incorrect, resulting in FPs or FNs. As a demonstration of network forensics using real-world traffic, this work assesses FP/FN cases using the FPNA mechanism as shown in Fig. 2a. FPNA has the following three procedures, majority voting, trace Figure 1. Architecture and block diagram of the PCAPLib system.
Core-processing Pre-processing Active trace collection Replay area Log/alert collection server Multiple DUTs Replay tools Raw PCAP Beta site Extraction module Database Extracted PCAP Extracted PCAP Raw PCAP Anonymous extracted PCAP FP FN
Replayer Check device Device under text N
ILLT Website FP/FN assessment Application Back-end system Front-end system Classification module Anonymization module Deep packet anonymization
2It captures, records, and analyzes network events in order to discover the source of security attacks or other problem inci-dents.
verification and manual analysis. First, majority
voting is a decision which has a majority, that is, more than half of the votes. It is a binary sion voting used most often in influential deci-sion-making bodies, including the legislatures of democratic nations. In this work, the voters are all DUTs and potential FPs/FNs are detected under the definition of majority voting. In other words, if only one or a few DUTs generate a detection log for some specific packet trace, this trace appears as an FN or a true negative (TN) case. On the other hand, when more than half of the DUTs have alerts for this trace, the trace is likely to be an FP or a true positive (TP). Major-ity voting’s flow chart is described in Fig. 2b.
Second, after detecting the potential FPs/FNs/TPs/TNs, this work replays the extract-ed packet trace according to the log to the DUTs again. This step is called trace verification because it verifies whether this case is repro-ducible to the original DUTs. This case is a reproducible FP/FN/TP/TN when it meets the following two conditions.
• For any DUT, it must produce an alert if it did last time
• The two alerts must be the same when one came from some DUT last time and the other is produced by the same DUT this time
Otherwise, this case is un-reproducible. For example, there are one traffic flow and three DUTs, A, B and C. After this traffic flow passes through the PCAPLib system, we get an extract-ed packet trace from this traffic and two alerts from A and C. Two alerts are named A1 and C1, respectively. Then, we replay this extracted packet to A, B and C again. If A and C produce alerts, called A2 and C2, and the content of A2 and C2 are same as that of A1 and C1, respec-tively, this extracted packet trace is reproducible. In order to show these two conditions in Fig. 2c, we use “are all alerts same as before?” to repre-sent them. Late, in order to know whether the reproducible traffic trace is a publicly malicious case, the step of manual analysis manually inves-tigates the causes of the reproducible traffic trace and compares these causes with Common Vulnerabilities and Exposures (CVE), a dictio-nary of publicly known information security vul-nerabilities and exposures. After this step, an FP/FN or a TP/TN is identified and the occur-rences of frequent cases are also counted. Fig-ures 2c and 2d respectively describe the flow charts of the second and third steps.
S
TATISTICAL
A
NALYSIS
This section reviews the experimental environ-ment and DUTs’ information of the FPNA is first overviewed. Then, statistical analysis of FPs and FNs, and some interesting observations and summarization are detailed.
E
XPERIMENTALE
NVIRONMENTThe PCAP files were captured real-world traffic at the BetaSite, as shown in Fig. 1, during the period Oct. 1, 2009 to Feb. 1, 2011. As men-tioned in Section 3, the FPNA has majority vot-ing, trace verification and manual analysis mechanisms. Majority voting can be executed in
a computer because it only counts the number of logs/alerts for a PCAP file and marks “poten-tial FP, FN, TP or TN” on this PCAP file. An experimental environment for trace verification for replaying PCAP files is shown in Fig. 3a. Here, seven DUTs are used and their detailed information, such as vendor, device, name, etc., is described in Fig. 3b. Depending on where the IDS/IPS locates, an IDS/IPS can be either
net-work-based or host-based. A netnet-work-based
IDS/IPS is an independent platform, while a host-based one consists of an agent on a host. According to the detection method, an IDS/IPS
Figure 2. Details of the false positive/negative assessment mechanism: a) whole
flow chart of FPNA mechanism; b) flow chart of majority voting; c) flow chart of trace verification; and d) flow chart of manual analysis
TP FP Are all alerts same as before? Yes Yes No No Majority voting Potential FP/TP Potential FN/TN Potential FP/TP Reproducible FP/TP Manual analysis (reference: CVE) TN FN Reproducible FN/TN Manual analysis (reference: CVE) Manual analysis Reproducible FP/TP Un-reproducible FP/TP Are all alerts same as before? Yes No Trace verification Reproducible FN/TN Un-reproducible FN/TN Potential FN/TN
Replay to all DUTs Replay to all DUTs
Over half DUTs alert? Manual analysis FP FN TP TN Trace verification FPNA (a) (b) (c) (d) Majority voting Extracted PCAP and related alerts Extracted PCAP and related alerts
can be placed into one of two categories, name-ly signature-based and anomaname-ly-based. A signa-ture-based IDS/IPS compares packets with preconfigured and predetermined attack pat-terns or predefined descriptions of intrusive behavior known as signatures. On the other hand, an anomaly-based one tries to build mod-els for normal behaviors and detects anomalies in observed data by noticing deviations from these models. From Fig. 3b, we observe that only Trend Micro TDA2 is an IDS while the other six DUTs are IPSs. In this work, all DUTs are network-based security detection systems due to the PCAPLib system’s architecture whereas they are all signature-based because a signature-based IDS/IPS is more easily imple-mented than an anomaly-based one. During replay, all functions, like antivirus, anti-spam,
P2P, instant messenger (IM), streaming scan, and system logs of DUTs are enabled if possi-ble. After trace verification, reproducible FPs/FNs/TPs/TNs will be passed to the manual analysis step, where all alerts are compared to the CVE in order to check whether they are FPs, FNs, TPs, or TNs.
S
TATISTICALR
ESULTSThis subsection analyzes what kinds of FPs or FNs happen easily to IDS/IPS with real-world traffic and investigates their frequencies across all FPs and FNs. There are two hierarchies of classification in this work. One is by protocols, such as HTTP, FTP, NetBIOS and IRC and the other is by IDS policy types (also called “attack types”), like DDoS, buffer overflow, Web attack, scan, and so on.
Figure 3. a) Experimental environment; b) detailed information of seven DUTs. Vendor IDS/IPS Location Method AntiVirus AntiSpam P2P IM Streaming Device name Fortinet IPS Network Signature Yes Yes Yes Yes No FortiGate-110c McAfee IPS Network Signature No No Yes Yes No M-1250 ZyXEL IPS Network Signature Yes No Yes Yes No ZyWALL USG1000 Trend Micro IDS Network Signature Yes No Yes Yes Yes TDA2 D-Link IPS Network Signature No No Yes Yes No DFL-1600 TippingPoint IPS Network Signature No No Yes Yes No 5000E BroadWeb IPS Network Signature No No Yes Yes No NetKeeper7K (a) (b) McAfee M-1250 Inline mode BroadWeb NetKeeper7K Inline mode D-Link DEL-1600 Inline mode Trend Micro TDA2 Sniffer mode TippingPoint 5000E Inline mode ZyXEL Zy-WALL USG 1000 Inline mode Fortinet FortiGate-110c Inline mode Replay module Traffic 1 Traffic 1 Traffic 2 Traffic 2 Switch Switch Syslog Regeneration tap
FP Cases Taking the Most Percentage of False Cases — Figure 4a depicts the numbers
and ratios of FP and FN cases, while Fig. 4b shows those of FP and FN types. From Fig. 4a, we can observe that the number of FPs is 13 times that of FNs. In other words, more than 92.85 percent of false cases are FPs. However, when we calculate how many kinds of attack there are in FPs and FNs, as shown in Fig. 4b, we find that the number of kinds of attack in FN cases, 27, is close to that in FP cases, 35. Accord-ing to Figs. 4a and 4b, we guess that FP cases have many cases with traffic similarity, meaning that network traffic of a certain protocol happens to exhibit some characteristics belonging to other protocols [7]. To prove this guess, the number of each type of attack is calculated. For instance, the information of protocol, attack message and the number of each attack type in FP cases is depict-ed in Fig. 4c. There are dozens or hundrdepict-eds of FP cases as compared to only a few FN cases.
From the information in Fig. 4c, we can see that about 91 percent of FP alerts, equal to about 85 percent of false cases, are not related to security issues, but to management policy. Policy here means some configuration argu-ments are artificially constructed for some rea-son. For instance, some companies and campuses limit or forbid their employees and students from using peer to peer (P2P) applications, and therefore, thresholds of P2P traffic in an IDS/IPS will be configured very low. Hence, this causes alerts to be easily triggered regardless of whether the P2P application has malicious traffic or not.
Policy and Self-Defined Formats Causing FPs — Figure 5a shows the most frequent attack
types of FPs from the sample traces in our inves-tigation.
• The “HTTP-Inspection” alert results from application clients using their self-defined formats, not defined by RFCs, and the traf-fic happens to be similar to an ASCII-encoding attack, apache-whitespace attack, and so on.
• The “SQL Injection comment attempt” alert results from BitTorrent clients who happen to bind port 80, and the traffic hap-pens to be similar to an injection attempt. • Then “TCP port scan” alert results from
applications which test how many free ports there are in order to establish many con-nections at the same time.
• The “FTP wu-ftp bad file completion attempt” alert results from the “[“ charac-ter which appears often in FTP transfer data.
• The “Veritas Backup Agent DoS attempt” alert results from BitTorrent clients who bind port 10000 (the port monitored by the rule), and the traffic happens to be similar to a DoS attempt.
Figures 5b, 5c, and 5d present the propor-tions of four protocols of FPs, those of five attack types under HTTP and those of two attack types under unknown applications using TCP (UAT), respectively. From Fig. 5b, HTTP accounts for 90 percent of FPs, and from Fig. 5c, 67 percent of HTTP is web attacks.
Figure 4. Statistics and comparisons between FPs and FNs: a) FP/FN count ratio; b) FP/FN type ratio; and
c) examples of FP types (protocol, attack message and number). Protocol HTTP HTTP HTTP HTTP HTTP HTTP TCP (scan) FTP Attack message
HTTP-Inspection (8119001) ASCII-ENCODING ATTACK HTTP-Inspection (8119012) APACHE-WHITESPACE ATTACK HTTP-Inspection (8119013) NON-RFC-HTTP-DELIMITER ATTACK
Sig (8009136) SQL injection SQL command attempt-2 Sig (8009000) SQL injection comment attempt Sig (8009004) SQL injection SQL command attempt
Scan-detection (8122001) TCP port scan Sig (8000392) FTP wu-ftp bad file completion attempt
No. 460 399 276 191 176 116 106 76 FN 2026, 93% 152, 7% FP/FN count ratio (a) (c) FP FN 35, 56% 27, 44% FP/FN type ratio (b) FP Some companies and campuses limit
or forbid their employees and
stu-dents from using peer to peer (P2P)
applications, and therefore, thresholds
of P2P traffic in an IDS/IPS will be con-figured very low. Hence, this causes
alerts to be easily triggered regardless of whether the P2P application has mali-cious traffic or not.
Many Aged Attacks Having New Variations
— Figure 6a shows the most frequent attack types of FNs.
• The “Buffer Overflow” alert results from Windows being vulnerable to buffer over-flow when handling certain types of Remote Procedure Call (RPC) traffic, and this flaw occurs within the ‘netapi32.dll’ component of the Server service with NetPathCanoni-calize requests.
• The “SQL Server Attack” alert results from a login that fails for user ‘sa’.
• The “MS-SQL Worm Slammer” alert is caused by DoS on some Internet hosts.
In sum, the buffer overflow and the MS-SQL worm slammer, totaling 103 FN cases, are aimed at Microsoft products because Microsoft is estimated to make up nearly 90 percent of the OS market-share [10]. Moreover, although buffer overflow, SQL server attacks and worm slammer attacks are aged attacks, they still account for 93 percent of FNs. This may indicate that these attacks always have new variations to avoid IDS/IPS detection.
Figures 6b–d, and 6e present the proportions of four protocols of FNs, those of two attack types under NetBIOS, that of one attack type under UAT and that of one attack type under an unknown application using UDP (UAU), respec-tively. From Fig. 6b, NetBIOS accounts for 68 percent of FNs, and from Fig. 6c, 90 percent of NetBIOS is buffer overflow attacks.
C
ONCLUSION
This work proposes the FPNA mechanism in the PCAPLib system to provide statistical analysis of FP and FN cases. The FPNA collected more than two thousand FPs and FNs during sixteen months. 92.85 percent of false cases were FPs and 7.15 percent were FNs. Out of numerous FPs, about 91 percent of FP alerts occur because of IDS’s or IPS’s policy, not due to security issues. The distribution of the collected FPs shows that 90 percent are using HTTP and 57 percent of FPs are thought to be HTTP inspec-tion attacks. NetBIOS accounts for 68 percent of FNs and about 67 percent of FN cases are aimed at Microsoft products. From the statistical analy-sis, we also observe that traffic similarity is the main cause of FP cases, and missing attack sig-natures in the signature design is the cause of FN cases.
Although there are thousands of FP and FN cases in this work, these FPs/FNs are detected by the signature-based IDSs/IPSs. Maybe some FPs or FNs occur in the anomaly-based IDSs/IPSs, and accordingly, new DUTs, includ-ing anomaly-based or online/cloud IDSs/IPSs, will join the experimental environment in the future. Furthermore, the FPNA will continue to trace whether statistical results change when the DUTs update their engines and virus patterns. In summary, FPs/FNs are still the key issues for Figure 5. FP attack ranking and detailed analysis: a) top five frequent attack types; b) proportion of four
protocols of FP; c) five attack types under HTTP; d) only two attack types under UAT.
(a) (b)
(c) (d)
Frequent IDS policy types under unknown (TCP) Frequent IDS policy types
under HTTP Frequent protocol of FP Attack ranking of FP HTTP 1172, 57% 484, 24% 112, 6% 76, 4% 56, 3% 126, 6% 112, 94% 1828, 90% 119, 6% 76, 4% 3, 0% 1212, 67% 8, 0% 7, 6% 19, 1% 81, 4% 508, 28% Unknown (TCP) Scan Exploit FTP SNMP HTTP-inspection SQL injection TCP port scan FTP attack
Veritas backup agent DoS 6-12th Web attack Exploit DoS Buffer overflow Others Although buffer overflow, SQL server attacks and worm slammer attacks are
aged attacks, they still account for 93 percent of FNs. This
may indicate that these attacks always have new variations
to avoid IDS/IPS detection.
IDSs/IPSs which are less reliable today because of the limitations of the signature-based methodology.
R
EFERENCES[1] K.-C. Lan, A. Hussain, and D. Dutta, “Effect of Malicious Traffic on The Network,” Proc. Passive and Active
Mea-surement Wksp. (PAM), San Diego, CA, Apr. 2003.
[2] S.-X. Wu and W. Banzhaf, “The Use of Computational Intelligence in Intrusion Detection Systems: A Review,”
Elsevier Applied Soft Computing, vol. 10, issue 1, Jan.
2010, pp. 1–35.
[3] H. T. Elshoush and I. M. Osman, “Reducing False Positives through Fuzzy Alert Correlation in Collaborative Intelligent Intrusion Detection Systems — A Review,” Prof. IEEE Int’l.
Conf. Fuzzy Systems, July 2000, pp. 1–8.
[4] M. Sourour, B. Adel, and A. Tarek, “Environmental Awareness Intrusion Detection and Prevention System toward Reducing False Positives and False Negatives,”
Proc. IEEE Symp. Computational Intelligence in Cyber Security, Apr. 2009.
[5] G. P. Spathoulas and S. K. Katsikas, “Using a Fuzzy Inference System to Reduce False Positives in Intrusion Figure 6. FN attack ranking and detailed analysis: a) top four frequent attack types; b) proportion of four
protocols of FN; c) only two attack types under NetBIOS; d) only one attack type under UAT; e) only one attack type under UAU.
97, 63% 94, 90% 39, 100% (d) (e) (c) (b) (a) 6, 100% 10, 10% Frequent IDS policy types
under NetBIOS
Frequent IDS policy types under unknown (TCP)
Frequent IDS policy types under unknown (UDP)
104, 68% 39, 26% 6, 4% 6, 4% 3, 2% 10, 7%
Attach ranking of FN Frequent protocols of FN
39, 26% NetBIOS Unknown (TCP) Unknown (UDP) IRC Buffer overflow SQL server attack Buffer overflow Exploit Worm Access control Access control MS-SQL worm slammer
The FPNA will continue to trace whether statistical results change when
the DUTs update their engines and virus patterns. In summary, FPs/FNs are still the key issues
for IDSs/IPSs which are less reliable today
because of the limitations of the
signature-based methodology.
Detection,” Proc. 16th Int’l. Conf. Systems, Signals and
Image Processing, June 2009.
[6] I.-W. Chen et al., “Extracting Attack Sessions from Real Traffic with Intrusion Prevention Systems,” Proc. IEEE
ICC, June 2009.
[7] S.-H. Wang, “Extracting, Classifying and Anonymizing Packet Traces with Case Studies on False Positives/Neg-atives Assessment,” M.S. thesis, Dept. Comp. Sci., Nat’l. Chiao Tung Univ., Taiwan, 2010.
[8] Y.-D. Lin et al., “On Campus Beta Site: Architecture Designs, Operational Experience, and Top Product Defects,” IEEE Commun. Mag., vol. 48, no. 12, Dec. 2010, pp. 83–91.
[9] TippingPoint Technologies, “IPS vs. IDS: Similar on the Surface, Polar Opposites Underneath,” Whitepaper, http://rovingplanet.net/resources_whitepapers.html. [10] “Global Market Share Statistics and News,”
http://mar-ketshare.hitslink.com/os-market-share.aspx?qprid=9.
B
IOGRAPHIESCHENG-YUANHO([email protected]) is an assistant research fellow in the Information and Communications Technology Laboratory of Microelectronics and Information Systems Research Center, Network Benchmarking Lab, and D-Link NCTU Joint Research Center at National Chiao Tung University, Hsinchu, Taiwan. His research interests include the design, analysis, and modeling of the congestion con-trol algorithms, real-flow test, network security, cloud com-puting, high-speed networking, embedded hardware-software co-design, quality of service, and mobile and
wireless networks. Ho received a Ph.D. in computer science from National Chiao Tung University, Taiwan, in 2008. YUAN-CHENGLAI([email protected]) received a Ph.D. degree in computer science from National Chiao Tung University in 1997. He joined the faculty of the Department of Information Management at National Taiwan University of Science and Technology in 2001 and has been a professor since 2008. His research interests include wireless networks, network perfor-mance evaluation, network security, and content networking. I-WEICHEN([email protected]) received an M.S. degree in computer science and information engineering from National Chiao Tung University in 2003. He is the Director of the Network Benchmarking Laboratory at National Chiao Tung University. His research interests include network benchmark, network security, high-speed networking, real-flow test, and wire-speed switching and routing.
FU-YUWANG([email protected]) received an M.S. degree in computer science and information engineering from Chung Hua University, Taiwan, in 2007. He joined the Network Benchmarking Laboratory at National Chiao Tung University in 2008. His research interests include network security, botnet, real-flow certification, and quality of service. WEI-HSUANTAI([email protected]) received an M.S. degree in computer science from National Chiao Tung Uni-versity in 2011. His research interests include network secu-rity, intrusion detection, and false positive and false negative analysis. His mentor is Dr. Cheng-Yuan Ho.
