Overheads in a storage-based mail proxy
MTA MTA
Port 25 Port 10026
AMaViS
Port 10025
Mail Mail Mail
MIME paser and decoder
Part Part
File File Decompression
program Virus Scanner
Sender SMTP server
proxy
Fig.1 Storage-based proxy - AMaViS
Since AMaViS is a widely used storage-based mail proxy, we choose it to observe the mechanism and overheads in a storage-based mail proxy. Figure 1 shows the typical composition and the dataflow of AMaViS. AMaViS acts as an “interface”
daemon connecting two MTA (mail transport agent) daemons. An MTA daemon receives mails from port 25. The AMaViS daemon scans the mail from the MTA. If the mail doesn’t contain virus, the AMaViS daemon transmits it to another MTA daemon which responds of sending the mail to the real target. The reasons of this complicated three daemon architecture are: (1) The historical problem, the original version of AMaViS is a script program called by MTA. It became a daemon for performance issue. (2) To protect against mail loss. AMaVis is not a full-featured SMTP server, it needs MTAs which respond of sending and receiving respectively to prevent the unpredictable things.
There exist three distinct overheads: file access, inter-process communication and process forking in AMaViS. These overheads are also examined with the
performance results in Chapter 5.
A storage-based mail proxy receives an entire file before starting to process it. It often stores the file to a disk through the file system. Any processing could be slowed down by the lengthy file system access and disk access. This overhead increases as there are other processing stages like decompression and virus scanning, all involving heavy file access. In Figure 1, the file access overhead is in all three daemons, especially in AMaViS. AMaViS receives the mail and decodes attachments into files.
If the file needs to be decompressed, AMaViS calls the external program to decompress it into another file. Finally, AMaViS calls virus scanner to scan those files.
Lots of file system access overhead in this processing.
Since mails need to be transferred between the three daemons, there are several inter-process communications. When AMaViS calls the external program to decompress and scan viruses, the inter-process communications also occur because the data need to be transferred between different processes.
AMaViS and most MTAs use multiple processes to achieve the concurrency.
When there are many clients, per-client processes are forked in the three daemons.
Lots of the memory is occupied by these processes. The fork system call also brings heavy overheads.
Requirements of a stream-based mail proxy
The most essential requirement of a stream-based mail proxy is that each component in the proxy should be stream-based. The processing in a mail proxy contains MIME parsing, decoding, decompressing, virus scanning, and encoding. The proxy receives a part of a mail in a memory buffer, and then processes the buffer according to its content. Some intermediate buffers may be required. For example, decompressing and decoding need extra buffers. The processing is on the buffers rather than on the entire file.
Concurrency strategy
The per-connection multi-process architecture uses too much memory. The multi-threaded architecture is more feasible. A thread can be allocated either to execute a specific function or to serve a connection. In the former strategy, each processing function in the proxy has a corresponding thread which handles many connections concurrently and also needs to synchronize with other threads. In the latter, a thread is allocated for each connection instead. Each thread handles the mail step by step, from protocol handling to virus scanning.
However, our implementation platform is on Perl. The creation of threads in Perl uses as much memory as forking processes[11] and the Perl interpreter is also duplicated, so the name of thread in Perl is “ithread” which means interpreter-level thread. Finally we choose the single-process architecture with socket I/O multiplexing to handle concurrency. Although the single-process architecture could not take advantage over the multi-processor system and is more complicated to maintain the code, it has the most economical memory usage and eliminates the context-switching overheads. There is also no thread synchronizing and inter-process communication.
These could render high scalability in terms of the number of connections.
On-the-fly decompression
Storage-based systems need to store the decompressed files, which may be much larger than the original files. A denial-of-service attack could send a file that is over 100 times larger after decompression. Storage-based systems thus often bypass or block the file whose size might exceed a threshold after the decompression.
Lossless data compression methods are often the “adaptive dictionary”
algorithms, such as LZ77[21], LZ78[22] and LZW[23]. A word is added to a dictionary when it appears for the first time. When the same word appears again, the encoder substitutes a short code for it. The file can be later decompressed by indexing
on the dictionary. This sequential compression/decompression mechanism makes it possible to decompress the portion of data in order. As long as the dictionary is located at the beginning of the file and the proxy receives segments in order, the stream-based decompressing by indexing should be feasible. Table 1 presents the common compression formats. The BWT[24] algorithm is block-based since it processes a block of data which is 900KB by default. The proxy need to queue the data until the entire block is received. The self-extract file contain the decompress program and the compressed data. The proxy need to identify the self-extract file and decompress it on-the-fly. Some compressed file may be encrypted, the proxy can’t process the encrypted file.
Format Program Algorithm File extent Stream?
unix compress compress LZW .Z Yes
gzip gzip Deflate
(LZ77+Huffman)
bzip2 bzip2 BWT .bz2 Block-based
lha lha LZ78+Huffman .lha .lzh Yes
self-extract itself Depends on format .exe Yes * Table 1 Compression formats
The original design objectives of file compression are not for the streaming purpose. The ready-made programs and libraries all process an entire file. It makes stream-based systems not so popular in the market. To do the on-the-fly decompression, the system needs to modify low-level decompression libraries and call the low-level API directly. For example, for the files with the “.gz” extension, the deflate function in Zlib[] is called instead of executing the gzip[12] program. The detailed implementation is addressed in Chapter 4.
A file can be compressed more than once, i.e. recursively, and a compressed
archive may contain multiple compressed files. On-the-fly decompressing is complicate to handle the recursive compression, because it needs to parse the decompressed content continuously to check if another compressed file is there. A compressed file creates a decompressing process and a parsing process which might find another compressed file. When the archive contains multiple compressed files with recursive compression, several decompressing and parsing process at the same time and the data transfer between them is complicated. By contrast, the storage-based system can simply solve this problem by recursive decompression using external program sequentially.
Virus patterns across segment boundaries
The stream-based system scans individual buffers where segments of file content are processed, but virus patterns may be across the segment boundaries. There are two solutions to this problem. The system can keep the state of the virus scanner, i.e.
which signature has its head matching the tail of last segment, through the entire scanning. This solution needs to modify the virus scanner. Another solution uses a mechanism called cushioned scanning[13]. A cushioned scan extends the buffer with sufficiently large data from the tail of the previous scan buffer on the head side. That is, data in the cushion buffer is scanned twice. The size of a cushion buffer should not be shorter than the longest pattern in the virus database. The same problem also occurs on decompressing, the decompression engine need to keep the decompressing status of the file throughout the entire decompressing.