a smaller Σ’ and mapping table2 maps the σ’w into σ’’. The two tables combine into a single a mapping table and the input stream is translated according to this table.
4.8 Combine Single Symbol Blocks
The expected block size determines the factor of speedup, so when the con-secutive single symbol blocks combine into one block, the factor of speedup rises.
Paper [10] proposed the above idea and a scheme for string matching. Combination Rule 1 applied to the input data stream can be used directly here, but Combination Rule 1 applied to patterns has to be extended to pattern matching.
There are possible multiple following blocks of one block in a variable NFA, and each single symbol block tries to combine with its following single symbol block.
However, when one of following blocks is multi-symbol, the preceding block stays in the state machine.
Figure 4.6 shows an example how combined single symbol block, and the max-imum size of combined block is 3. C1,C2, and C3 are single symbol block in the original variable stride NFA. The process of combination starts at C1, and then C1
Figure 4.6: Combine single symbol block into one block
and C2 combine to form C1C2 which is from q1 to q3 in variable stride NFA, because C1 and C2 are single symbol block. C1C2 does not reach the maximum size of block, so C1C2 and C3 can still combine together. However, block C4C5 is from q3 to q5, so C1C2 has to stay in automata after combining block so that the input stream can still reach q5 from q1. The result of the combination is shown at the bottom of Figure 4.6.
For pattern matching, the pattern must start at the beginning of the file so that the ambiguity at the beginning of the pattern does not exist, but the ambiguity at the end of the pattern still happens.
Chapter 5
Implementation and Result
5.1 Rule Set and Traffic
The rules in experiment are from Snort [1],and are released in 2011/06/02, and are represented as Perl Compactable Regular Expression. The rules which match the header{tcp, $HOME_NET, any, $EXT ERNAL_NET, any} is found, and the regular expressions after the keyword “pcre”in these rules are extracted. There are 1395 rules which include regular expression in the rule set, and the number of the extracted rules is 209; the rules converted into NFA are chosen from these rules randomly. The extracted rules include most features defined in PCRE, except the Unicode and back reference. The chosen traffic is from the darpa98.
One of the rules of matching requirement is shown at the top of Figure 5.1 . This rule is a tcp[The regular expression is the only part used in the experiment, so the regular expression is extracted at the bottom of Figure 5.1.
Figure 5.1: An example shows that a regular expression is extracted from a rule.
5.2 Implementation of NFA
Regular expressions are parsed by a self-made parser generated by Flex and Bison. Some features of Perl regular expression about back reference are not im-plemented, because these functions are not used in the chosen regular expressions.
Each rule is converted into NFA with ϵ individually and rules which are in the same group connect to the same start point of NFA that is suggested in [9] to avoid state reputation for the anchored pattern with m-modifier. ϵ in NFA can be removed by the algorithm in [9] and this algorithm can reduced the NFA state and transition.
However, there are some states that cannot reach any accepting state in the NFA after reducing, so an algorithm to remove these redundant states is added after re-ducing NFA. These NFAs are converted into variable stride NFAs and double stride NFAs which are used to compare with variable NFAs.
After generating variable-stride NFA, this machine can be used to scan the input stream by the detector. The detector uses Libpcap [15] to capture the packet from tcpdump file. Unlike Snort, the packet decoder is not implemented, so the detector can only detect the payload of a packet. The payload also has to be divided by Winnowing algorithm (with or without combination rule) as mentioned before, and
Table 5.1: Average alphabet size, number of state, number of transition, and speedup
#RE per NFA 1 3
type NFA 2-NFA VSNFA BC NFA 2-NFA VSNFA BC
|Σ| 17 67 41 81 31 202 230 360
#state 1102 1102 825 773 996 996 714 685
#tx 4590 10392 10734 15067 4725 15688 21534 29458
Speedup 1 2.00 1.31 3.25 1 1.98 1.41 3.56
two mapping tables generated by alphabet reduction are also applied to the input stream.
5.3 Result
Table 5.1 shows the comparisons between normal NFA, double stride NFA, vari-able stride NFA and varivari-able stride NFA with single symbol block combination. The window size in variable stride method is 3. The speedup of normal variable stride is smaller than double stride and the speedup of variable stride with block combi-nation is larger than double stride. The speedup of variable stride is determined by the average block length. The theoretical value of the average block length is 2, but its real value is about only 1.45, so the speedup of variable stride is smaller than the predicted value. The speedup of variable stride with block combination is significantly larger than normal variable stride, because there are many single symbol blocks in the normal variable stride NFA, and when blocks are combined, the average length of a block becomes larger. This proves that block combination scheme can work well.
The usage of memory of normal variable stride is more than double stride because the variable stride NFA needs more transitions to present regular expression .* and [∧]*; this cancels out the self synchronized benefit of variable stride. When blocks are combined, new blocks which have never been used in normal variable stride appear
Table 5.2: Mix double stride NFA and variable stride NFA.
DSNFA VSNFA Total
#rules 12 18 30
#state 501 435 936
#tx 6750 4108 10858
in variable stride with block combination, so the size of the alphabet increases, and variable stride NFA needs more transitions to present regular expressions .* and [∧]*.
When one regular expression is converted into one NFA, not all of the variable stride NFAs need more memory than double stride. Some variable stride NFAs only need 60%-75% memory space relative to double stride NFAs. That suggests that some rules which use less memory in variable stride method can be converted into variable stride NFA and other rules can be converted into double stride NFA. It is easy to keep two kinds of NFA in the same system because the difference between variable stride method and double stride method is the division of input stream except generating process. Another good side effect is that variable stride NFA has better throughput when those variable stride NFAs are chosen. Table 5.2 shows the result of mixing the two kind of NFAs. The number of rules is 30 ,and the rules are the same as Table 5.1. The result of this mix can reduce 5% memory space relative to original double stride NFA.
Figure 5.2 shows the distribution of the number of outgoing transitions per state.
The states without transition are accepting states. For both kinds of VSNFA, the 90% states have fewer than 16 outgoing transitions and 60% states have fewer than 7 transitions. This suggests that variable stride NFA can be properly encoded to distinguish between the state with fewer transitions and the state with more transitions and make NFA more efficient.
Figure 5.2: shows the distribution of the number of outgoing transitions per state
Chapter 6 Conclusion
Winnowing algorithm was extended to pattern matching in this paper. For improving its efficiency: the notation of limit was introduced to speed up the gener-ating process of variable stride NFA; alphabet reduction was employed to decrease the usage of memory; block combination was utilized to increase the throughput of variable stride NFA.
As a result of the experiment shown, variable stride and block combination scheme are shown to improve scanning time relative to both the original method and the double stride method. However, this also increases the usage of memory.
In addition, the time requirements for generating variable stride NFA remains a problem. The future practicality of this whole algorithm will rest on developing improvements within the generating process.
References
[1] Snort. [Online]. Available: http://www.snort.org/
[2] Bro. [Online]. Available: http://bro-ids.org/
[3] Cisco Adaptive Security Appliance. [Online]. Available: http://www.cisco.com/
[4] Clamav. [Online]. Available: http://www.clamav.net/
[5] M. Becchi and P. Crowley, “An improved algorithm to accelerate regular ex-pression evaluation,” in Proc. of ACM ANCS’07, 2007, pp. 145–154.
[6] D. Ficara, S. Giodano, G. Procissi, F. Vitucci, G. Antichi, and A. D. Pietro,
“An improved DFA for fast regular expression matching,” ACM SIGCOMM’08 Computer Communication Review, vol. 38, Issue 5, pp. 29–40, Oct. 2008.
[7] L. Yang, R. Karim, V. Ganapathy, and R. Smith, “Improving NFA-based sig-nature matching using ordered binary decision diagrams,” in Proc. of RAID’10, 2010, pp. 58–78.
[8] B. Brodie, R. Cytron, and D. Taylor, “A scalable architecture for high-throughput regular-expression pattern matching,” in Proc. of ISCA’06, 2006, pp. 191–202.
[9] M. Becchi and P. Crowley, “Efficient regular expression evaluation: Theory to practice,” in Proc. of ACM/IEEE ANCS’08, 2008, pp. 50–59.
[10] N. Hua, H. Song, and T. Lakshman, “Variable-stride multi-pattern matching for scalable deep packet inspection,” in Proc. of IEEE INFOCOM’09, 2009, pp.
415–423.
[11] S. Schleimer, D. S. Wilkerson, and A. Aiken, “Winnowing: Local algorithms for document fingerprinting,” in Proc. of ACM SIGMOD’03 on Management of data, 2003, pp. 76–85.
[12] K. Thompson, “Regular expression searching algorithm,” Communication of ACM, vol. 11, Issue 6, pp. 419–422, Jun. 1968.
[13] J. E. Hopcroft, R. Motwani, and J. D. Ullman, Inroduction to Automata Theory, Languages,and Computation. Addison Wesly, 1979.
[14] A. V. Aho and M. J. Corasick, “Efficient string matching: An aid to biblio-graphic search,” Commucations of the ACM, vol. 18, Issue 6, pp. 333–340, Jun.
1975.
[15] Libpcap. [Online]. Available: http://www.tcpdump.org/
[16] S. Kumar, B. Chandrasekaran, J. Turner, and G. Varghese, “Curing regular expressions matching algorithms from insomnia, amnesia, and acalculia,” in Proc. of ACM/IEEE ANCS’07, 2007, pp. 155–164.
[17] R. Smith, C. Estan, S. Jha, and S. Kong, “Deflating the big bang: Fast and scalable deep packet inspection with extended finite automata,” in Proc. of ACM SIGCOMM’08 conference on Data communication, 2008, pp. 207–218.
[18] M. Becchi and P. Crowley, “A hybrid finite automaton for practical deep packet inspection,” in Proc. of ACM CoNEXT’07, 2007.
[19] S. Kumar, S. Dharmapurikar, F. Yu, P. Crowley, and J. Turner, “Algorithms to accelerate multiple regular expressions matching for deep packet inspection,”
in Proc. of ACM SIGCOMM’06, 2006, pp. 339–350.
[20] Y. Sun, H. Liu, V. C. Valgenti, and M. S. Kim, “Hybrid regular expression matching for deep packet inspection on multi-core architecture,” in Proc. of 19th International Conference on Computer Communications and Networks, Aug. 2010, pp. 1–7.
[21] S. Kumar, J. Turner, and J. Williams, “Advanced algorithms for fast and scal-able deep packet inspection,” in Proc. of ACM/IEEE ANCS’06, 2006, pp. 81–
92.
[22] M. Becchi and S. Cadambi, “Memory-efficient regular expression search using state merging,” in Proc. of IEEE INFOCOM’07, May 2007, pp. 1064–1072.
[23] N. Cascarano, P. Rolando, F. Risso, and R. Sisto, “iNFAnt: NFA pattern matching on GPGPU devices,” ACM SIGCOMM’10 Computer Communication Review, vol. 40, Issue 5, pp. 20–26, Oct. 2010.
[24] R. Smith, N. Goyal, J. Ormont, K. Sankaralingam, and C. Estan, “Evaluating GPUs for network packet signature matching,” in Proc. of IEEE International Symposium on Performance Analysis of Systems and Software, Apr. 2009, pp.
175–184.
[25] G. Vasiliadis and S. Ioannidis, “GrAVity: a massively parallel antivirus engine,”
in Proc. of Proceedings of the 13th international conference on Recent advances in intrusion detection, 2010, pp. 79–96.
[26] Y. Zu, M. Yang, Z. Xu, L. Wang, X. Tian, K. Peng, and Q. Dong, “Gpu-based nfa implementation for memory efficient high speed regular expression
matching,” in Proc. of the 17th ACM SIGPLAN symposium on Principles and Practice of Parallel Programming, 2012, pp. 129–140.
[27] J. van Lunteren, “High-performance pattern-matching for intrusion detection,”
in Proc. of INFOCOM’06, Apr. 2006, pp. 1–13.
[28] W. Lin and B. Liu, “Pipelined parallel AC-based approach for multi-string matching,” in Proc. of 14th IEEE International Conference on Parallel and Distributed Systems, Dec. 2008, pp. 665–672.
[29] I. Bonesana, M. Paolieri, and M. D. Santambrogio, “An adaptable FPGA-based system for regular expression matching,” in Proc. of Design, Automation and Test in Europe, Mar. 2008, pp. 1262–1267.
[30] N. Yamagaki and R. S. S. Kamiya, “High-speed regular expression match-ing engine usmatch-ing multi-character nfa,” International Conference on Field Pro-grammable Logic and Applications, pp. 131–136, Sep. 2008.
[31] A. Mitra, W. Najar, and L. Bhuyan, “Compiling PCRE to FPGA for acceler-ating SNORT IDS,” in Proc. of ANCS’07, Dec. 2007, pp. 127–136.
[32] Mansoor and M. M. V. Kumar, “High speed pattern matching for network IDS/
IPS,” in Proc. of the 2006 14th IEEE International Conference on Network Protocols, Nov. 2006, pp. 187–196.
[33] A. Bremler-Barr, D. Hay, and Y. Koral, “Compactdfa: Generic state machine compression for scalable pattern matching,” in Proc. of IEEE INFOCOM’10, Mar. 2010, pp. 1–9.
[34] F. Yu, R. H. Katz, and T. V. Laksman, “Gigabit rate packet pattern-matching using TCAM,” in Proc. of the 12th IEEE International Conference on Network Protocols, Oct. 2004, pp. 174–183.
[35] Y. Sun, V. C. Valgenti, and M. S. Kim, “NFA-based pattern matching for deep packet inspection,” in Proc. of 20th International Conference on Computer Communications and Networks, Jul. 2011, pp. 1–6.