Chapter 3 Design Issues
3.3 Other Issues
3.3.3 Improve operation speed and bandwidth consuming
Because of the uncertainty and preciousness of the mobile networks, we are trying to accomplish most of operations in the local file system without going through Internet. When accessing Internet, we want to use it more efficiently. In addition, comparing with local operations, communicating with the server is time-consuming. There are four circumstances that AshFS consumes a large amount of network bandwidth:
z Retrieving metadata: especially after reading an uncached directory.
z Cache validation: when checking time stamps.
z Time stamp synchronization: After uploading a file.
z Fetching and storing files.
When a user lists an uncached directory, AshFS library will first try to link and call
“readdir” function of AshFS process to find all entries under this directory. Then for each entry, the library will ask “stat” function for the metadata. After knowing what type of this entry is (file or directory), it finally shows the contents of the requested directory on the screen. During this process, stat function was called many times. Because each entry was uncached, the function needed to communicate with the server when it was called. Therefore a massive of send-receive pairs occurred here, and consumed bandwidth much. In order to be
more efficient, we decided not to request metadata one by one. Instead, we asked the server for all entries’ metadata in one time. So our readdir function is also responsible for filling all entries’ metadata. And our stat function will not contact to the server, just search AshFS’s cache.
The 2nd bandwidth-consuming case is cache validation. Look Figure 5 again. After thread 2 checks time stamp of request directory successfully, every entries’ time stamps will also be check. To avoid too many requests and replies happening, we decided to do it at once.
Similarly, thread 2 will request all entries’ time stamp at once.
The 3rd bandwidth-consuming case is time stamp synchronization. As we talked in Part B, upload manager will make time stamp of a server’s newly-uploaded file equal one in client’s cache. It is unnecessary to change time stamp of server’s file one by one, we can also change them all at once. When upload manager finishes propagating a file, it record its time stamp. After upload manager completes a part of its work in write queue, it creates a shell script file, and appends commands to it. Then upload the shell script to the server and ask server to execute it. This method effectively reduces the communication times.
To sum up the first three cases, network accessing is unavoidable, and it usually slows down the speed of a network file system. We want that AshFS’s network accessing can be more efficient, so we request as more objects as possible in one time. This principle effectively decreases communications with the server, and reduces the bandwidth usage.
The 4th bandwidth-consuming circumstance is the most important one: fetching and storing files. When a user tries to download or upload a new file from/to the server, a large amount of bandwidth will be used to transfer file contents. Except doing some optimizations to cancel out unnecessary operations, there is nothing we can do to preserve our constrained bandwidth in tradition. However, if the user is updating an old file instead of a whole-new one, it is not the case. Since the updated file may be similar as the old one in server or client’s cache, we can avoid transferring the same parts of contents over the network but only the
different ones. By exploiting the similarities between the new and old files, the actual amounts of transferred packets will be much less than total file size. It indeed preserves the bandwidth consumption and makes our bandwidth-usage more efficient.
Chapter 4 Implementation
In Chapter 3, we discussed several design issues, and these mechanisms have been built in our AshFS file system. Now we are going to account for how they were implemented.
4.1 System Architecture
File systems have been implemented in the OS kernel traditionally, in order to obtain the fastest speed and the best integration with OS. Although computing power has a great improvement recently, the performance bottlenecks of the most network file systems are at the network transmission speed. So the speed improvement of a pure kernel-based network file system is no longer significant. When developing a mobile file system, we decided to create it in user space to achieve a good portability. In addition, such a user-space file system can corporate with other applications easily to enhance its own features.
Figure 6 depicts AshFS's system architecture, where an ellipse represents a system call, a rounded rectangle represents an AshFS filesystem component, a rectangle represents a kernel interface, and a diamond represents an if-else statement. A complete AshFS system consists of a kernel module, a userspace library, a user application, a secure client and a secure server.
The figure also depicts the path of a file system request in AshFS. At the first time, the system call (request) enters the virtual file system (VFS) in the kernel, when it realizes that the request is for AshFS file system, the call is handled to the AshFS kernel module. Then AshFS kernel module passes the call to user-space AshFS library. Finally, the library communicates with the user-space AshFS process to know how to answer it. If the requested
file object is already in the cache and is fresh, the process will make a new request to the local file system. If the requested file object is uncached, AshFS process will ask the secure client for answers. Since mobile users usually roam around in less-reliable networks, we thought that our server and client should communicate with each other in a secure channel.
Figure 6: AshFS system architecture
4.2 Tools
To build a complete AshFS system, there are five components needed to be implemented (Figure 6). We planed not to do all of them by ourselves, but tried to combine with some powerful utilities in existence. We are going to introduce them and explain why they are adopted here.
4.2.1 FUSE
Conventional file systems which were implemented in the OS kernel take advantage of low overhead of accessing internal kernel services. However the speed improvement of a pure kernel-based network file system is no longer significant, because the network equipments are falling far behind the CPU and memory in speed. Recently efforts have made it become feasible to create a file system in user space, FUSE is one of it.
FUSE is a free filesystem framework and its name comes from “File system in user space”. It aims to provide the means by which file systems can be written in user space and exposed via the kernel to rest of the OS in a transparent way [1]. The infrastructure consists of an OS-dependent kernel module, portable user space libraries and a mount utility. Data and metadata of file system based on FUSE are provided by userspace process, and it makes this kind of file system can be more portable, since less functionality is bundled to the kernel.
Figure 7: Path of stat request in FUSE
The list (stat) call enters the VFS in the kernel, and when it identifies that the request is for FUSE file system, it passes the request to the FUSE kernel module. Then FUSE kernel module will pass the request to user-space FUSE libraries, by allowing these libraries to read a special device /dev/fuse. Finally, these libraries will communicate with the user space processes to know how to answer it.
4.2.2 SSH and OpenSSH
SSH (Secure Shell) original means a network protocol that allows data to be exchanged using a secure channel between two computers [8]. Another meaning of SSH is a very popular secure communication application. Users can remotely login to the SSH server with security protection [7]. With the support of SSH File Transfer Protocol and its application SFTP, user can store and download files to/from the server. Because SSH encrypts all data in transmission, it provides confidentiality and integrity of data over an insecure network. For mobile users, SSH can make their communications more secure. OpenSSH is one of the most widely used SSH implementations, and many Linux distributions have made it become one of the default installations. OpenSSH includes a set of applications for both server and client.
Since its popularity and simplicity, we decided to use it as our secure client and secure server.
4.2.3 Rsync
Rsync is an open source utility that provides fast incremental file transfer [27-28]. It synchronizes files and directories from one location to another while minimizing data transfer using delta encoding when appropriate. Since it copies only the diffs of files that have actually changed, much bandwidth consumption can be preserved. Rsync can synchronize files and directories from/to a remote rsync daemon or via a remote shell such as RSH or SSH, which meets our security concerns. In addition, it supports some others functions like compression and recursion to facilitate synchronization processes. Because its flexibility, security and
efficiency in transferring files, we chose it to become one of AshFS’s synchronization utilities.
4.3 Current Status and System Requirements
AshFS is now implemented in C on Linux. The server is a standard OpenSSH server, and the client is a user-level process who will fork other process in background. Figure 8 shows the basic flow of current AshFS when a user carries out one file system operation.
AshFS uses OpenSSH client applications to communicate with the server: “ssh” for passing control messages, and “sftp” for retrieving and updating files. AshFS will create two processes to individually execute these programs at initial and then return to the main procedure.
To realize AshFS, we implemented a set of functions like read(), write(), getattr(), readdir(), etc. FUSE uses a special structure which contains function pointers to this set of functions, and these functions will overlay a normal file system function [12]. These functions will access the special cache structure we designed for AshFS, and create other process or thread like Upload manager to start some mechanisms.
In order to run AshFS, besides AshFS client program we wrote, the client-side computers should have been installed the FUSE framework (both the FUSE kernel module and library) and the OpenSSH client suite which at least contains ssh and sftp command.
Surely, you also need to prepare a SSH server.
Figure 8: Path of stat request in AshFS
Chapter 5
Experiments and Performance Evaluation
In this chapter, some experiments were set to test if AshFS is suitable for our proposed environment. For an objective comparison, we added some of our related works into the experiments. Since MAFS is implemented on FreeBSD and RSC filesystem is designed for the Symbian OS, they were not tested in these experiments. Here we listed the results and analyzed these data. In the end of each experiment, we tried to interpret the relationship between results and our design principles.
5.1 Experiment Environment and Tools
To explore how our file system performs, some experiments have been made. We connected two computers into a Gigabit network; one acted as our client and the other acted as our server. The client computer had an Athlon XP 2600+ CPU and 2GB ram. The server was a Pentium R computer with 3.0GHz clock rate and 1.5GB ram. Both client and server were running Linux with 2.6.17 kernel above. In our experiments, the server shared some parts of its local file system; the client mounted it and performed file system some operations under the mount point. Each experiment was repetitive running for 10 times and the data presented here were the average. Here are some tools and what they were doing for.
z time-1.7-29-fc7: It times a simple command or gives resource usage [21]. We used it to measure the finishing time of each experiment.
z bonnie++-1.03: Bonnie++ is an industry-standard file system benchmark used to benchmark ideal performance in a uniform and repeatable way [19-20]. We used it to evaluate write throughput of a single file.
z CBQ.init-v0.7.3: Cbq.init is a Linux shell script file which uses CBQ (class-based queuing) mechanism to control network traffic [22]. We used it to constrain the bandwidth between the client and the server.
z iftop-0.17-6.fc7: It listens to network traffic on a named interface and displays current bandwidth usage by pairs of hosts [23]. We used it to measure client’s bandwidth usage.
5.2 Frequencies of Filesystem Operations
In this section, we used several methods to test operation speed of our target file systems.
Theses results should be close to how users really feel when using them.
5.2.1 List frequencies
In these experiments we executed “list” operation and recorded execution time of each target file system. Because some file systems have permanent/temporary cache and the cache state affects its performance, we made measurements for both cold and hot cache. In the first experiment, the client listed every sub-directory under the mounted directory and measured the latencies. For easing comparison, we used frequency (1/second) instead of time (second).
The result was shown in the Figure 9.
It is apparent that AshFS and Coda were slower than others when they listed the uncached directory. We thought that it is because of the cache overhead. But even under this circumstance, AshFS were 10 times faster than Coda. Nevertheless, if the client reads the directory again, Coda and AshFS will faster than others because they answer the request from cache directly without contacting to the server.
5.2.2 Frequencies of a sequence of operations
In this experiment, we measured the latencies of a series of file system operations:
create, list (stat), and remove. First the client created a list of files, and then it listed these files, finally it deleted them in sequential order. Each kind of latency was measured and Figure 10 shows the result. We also performed the same operations on local file system (ext3), and the result was denotes as “Local” in Figure 10.
The operation speed of AshFS was faster than all other network file systems. Coda also did well in this experiment. Because in most of modifications, AshFS and Coda will write to local disk first without waiting responses from the server, and communications often take time.
Figure 9: Frequencies of List operations
Figure 10: Frequencies of a sequence of operations
5.3 Write Throughput
We were going to evaluate write throughput of each file system. To explore how bandwidth affects these file systems, we divided environments into two different network bandwidth. Furthermore, consumed bandwidth of each operation was measured for comparing protocol overheads of these file systems.
5.3.1 Write throughput under unlimited bandwidth
There are two kinds of write throughput benchmark, one is writing a single large file, and the other is writing separate small files. For testing the anterior one, we used Bonnie++ to benchmark our target file systems. For the latter one, we copied a directory with varied total size manually from the local file system to the target filesystem, and then measured its execution time. Each directory contains many small files with different sizes from 100B to 1MB.
Figure 11 and 12 show the write throughput benchmarks. Obviously, sshfs had a lower performance in both two benchmarks; it is because of its encryption. It stands to reason that client’s local file system has the best throughput. Coda and AshFS led others because they wrote to local file system first, and didn’t communicate with the server much at that time.
When writing separate small files, all network file systems showed here are evidently slower than local, because much more time was spent on allocating space to files and file system itself (many stat and create actions were executed). Note that AshFS performed a little slower than Coda, and the gap was almost a constant in single file throughput, but it became smaller in separate files throughput. It could because that AshFS always transacts communications in batch (not individual and not synchronous for every file operation). Hence AshFS performed well in small files.
Figure 11: Single file write throughput
Figure 12: Separate files write throughput
Figure 13 shows the write overheads. We measured total transferred and received flows.
Then subtracted it from original size of the written file and took this value as the overhead. In both benchmarks, Coda’s overhead was much higher than others. It maybe means that Coda’s protocols are more talkative, so it needs more network resources for write operations. Sshfs took 2nd high overhead, because the encryption of SSH2. Nevertheless, AshFS also uses SSH2, but its overhead was 52-77% less than Coda and 32-34% less than sshfs. Here shows that our
design in reducing communications between client and server works.
Figure 13: Write overhead
5.3.2 Write throughput under lower bandwidth
To simulate mobile networks with usual transfer rate, we confined the experiment bandwidth to 4096kbps (512KB/s) for both downloading and uploading.
Figure 14 and 15 are our experiment results. NFS, SMB and sshfs performed notably worse than others, their throughputs were almost all equal to 0.46 MB/s (We have enlarged their data to make our figures more clear; original data displayed three overlapped lines lying on the x-axis). It shows that their performances were severely constrained by the available network bandwidth. It is because of they don’t have a persistent cache, and they usually need a steady, strong connection to the server. AshFS and Coda both take asynchronously write policy which means that files are written to local disk first, so their throughputs were similar to ones under unlimited bandwidth.
Figure 14: Single file throughput in 512KBps
Figure 15: Separate files throughput in 512KBps
Figure 16 are the write overheads in 512KBps bandwidth. There are no big differences except Coda’s overhead. Comparing with the overhead in unlimited bandwidth, even though coda’s overhead became less, it still used 51-52% more network resources than AshFS.
Figure 16: Write overhead in 512 KBps
Tables below are numeric benchmark data for reference.
AshFS Coda NFS SMB sshfs Local(ext3)
thruput-256M 37.314 42.007 28.086 30.660 10.446 47.482 thruput-384M 37.701 41.034 24.769 27.105 10.019 46.078 thruput-512M 37.703 40.251 21.453 23.549 10.257 44.299 thruput-640M 37.584 39.987 23.002 21.406 9.931 46.431 thruput-768M 36.753 39.840 22.138 19.738 9.181 43.542 overhead (%) 6.497 28.126 6.449 5.750 9.938 0.000
Table 3: Single file throughput under high BW
AshFS Coda NFS SMB sshfs Local(ext3)
thruput-256M 27.236 34.839 19.460 17.676 8.628 40.962 thruput-384M 28.197 31.457 18.941 17.844 8.615 42.025 thruput-512M 27.065 27.137 18.768 17.740 8.751 41.524 thruput-640M 27.356 28.938 19.468 17.776 8.745 42.875 thruput-768M 28.927 30.373 17.429 17.114 7.997 42.381 overhead (%) 6.483 13.467 6.820 8.275 9.648 0.000
Table 4: Separate files throughput under high BW
AshFS Coda NFS SMB sshfs Local(ext3) thruput-256M 33.066 40.751 0.4656 0.4679 0.4601 58.084
thruput-384M 35.071 39.795 0.4664 0.4675 0.4610 46.748 thruput-512M 35.664 38.294 0.4659 0.4688 0.4601 46.361 thruput-640M 34.229 39.587 0.4658 0.4676 0.4611 45.767 thruput-768M 35.026 40.243 0.4656 0.4672 0.4613 46.748 thruput-896M 35.256 40.687 0.4654 0.4671 0.4602 45.384 overhead (%) 6.297 12.825 6.406 5.719 9.516 0.000
Table 5: Single file throughput under 512KBps
AshFS Coda NFS SMB sshfs Local(ext3)
thruput-256M 24.267 33.810 0.4648 0.4655 0.4603 53.076 thruput-384M 25.120 33.550 0.4642 0.4651 0.4605 47.206 thruput-512M 24.700 29.922 0.4644 0.4660 0.4608 46.166 thruput-640M 23.917 28.693 0.4649 0.4656 0.4605 43.304 thruput-768M 23.358 24.710 0.4647 0.4653 0.4605 41.776 thruput-896M 23.210 26.963 0.4644 0.4659 0.4600 39.282 overhead (%) 6.375 13.375 6.750 6.219 9.766 0.000
Table 6: Separate files throughput under 512KBps
5.4 Synchronization Performance
5.4.1 Synchronization time
How long does it take to propagate all changed files to the server? We were going to measure it in this experiment. First we disconnected the network connection, and then wrote several files to our target file systems. As a result of lacking support of disconnected operations, just Coda and AshFS remained as our target file systems. After writing has finished, we reconnected the network. Once the target file system started to synchronize, we timed it. Figure 17 is our result; the horizontal coordinate denotes the total size of all written
files. Obviously, AshFS can finish synchronizing faster. The reason is because AshFS has a lower protocol overhead, and it transfers less data when synchronizing. In fact, we also recorded their transfer rates: The average transfer rate of AshFS was about 15MB/s, and one
files. Obviously, AshFS can finish synchronizing faster. The reason is because AshFS has a lower protocol overhead, and it transfers less data when synchronizing. In fact, we also recorded their transfer rates: The average transfer rate of AshFS was about 15MB/s, and one