Hybrid Caching Mechanism
4.2.2 File size
When we apply BFC mechanism to a file, extra overhead is necessary to be added into the file, and we also need extra workload on handling BFC mechanism, including the BFC
initialization, recognition, transmission and execution. As a result, if a small size file apply BFC, not only it cannot obtain better performance, but also increase the device's workload even if the format is fit to use BFC mechanism. So, we suggest that if any file size smaller or equal to a constant, says τ , we use WFC mechanism directly. Otherwise, we will determine it by the next factor.
4.2.3 Templates
After determine the selection of WFC and BFC, as every format (or application) has it's own characteristic, we build some templates to assign different cache setting based on different file format and application. A brief description for the supported templates will be introduced in the following:
BFC Template
For files which size are larger than τ and formats are suitable for BFC's application. Here we define three BFC templates for several different situations.
• BFCFIX is appropriate for the file that we seldom or even never modify, but we can get
benefit on fetching, such as the video and audio file, .avi, .mp3, etc. BFCFIX divides file into fix blocks.
• BFCVAR aimed to the ASCII text-based file, which user make insertion and deletion commonly to the file. File format such as .c, .cpp, .txt, .jpg and the calendar file, etc, can
have greater performance by using the BFC. However, if this kind of files is set to unable to be written, we use WHOLEFILE template.
• BFCSPEC particularly focus on the files that metadata locates both at the beginning and
the end, for example, pdf files. Moreover, pdf file builds file data page by page, so we divide the file blocks to match the pages. In addition pdf file data also include control messages which exist in pair, we need to keep any of a pair into the same block.
WFC's Template
• WHOLEFILE is appropriate to small files which size is smaller than τ, and which
formats belong to Mircosoft Office or other unmentioned format, or a text-based file and its file mode(in inode) is set unable to write. WHOLEFILE template treats file without dividing the file and adding any overhead to inside.
Fille type Cache mechanism Template Remark
All WFC WHOLEFILE Size <τ
avi, mp3, flv BFC BFCFIX Block size fix
txt, c, cpp,
BFC BFCVAR Block size variable
calendar, jpg
pdf BFC BFCSPEC Divided page by page
Mircosoft office series
WFC WHOLEFILE
& OTHERS
Table 4.1: Templates
Dynamic Regulation
In some special case, the WFC file will exceed the size of τ after user's modification.
Under this situation, upload will still apply WFC. Then both server and client apply the divide function to the target file and produce snapshot for each blocks. After that, the cache strategy will be changed to BFC. Besides if a big BFC file size decrease to lower than τ , we take the similar way to handle and change to WFC mechanism.
4.2.4 Strategy
We introduced two criteria to determine template for WFC and BFC mechanism, Fig. 4.2 shows the cache mechanism selection strategy.
File
In the following we talk about when and where HCM will be started and who takes it.
The initialization is mainly took charge by HCM server. Firstly, users prepare some files in server, and then server runs the initialization to identify the BFC-suit file, and build
snapshot and super snapshot. In some case, for example, user may create a new BFC-suit file with some content, if the file size finally exceeds τ , HCM runs initialization in both client side and server side after client upload the file to server. Afterwards, all the file modification will be handled in the HCM BFC mechanism.
In read operation, HCM client first check the target file existence in local, if it is located in one of the local caches, then it is cache hit and we read it directly and execute according to the
``TEMPLATE" term in super snapshot. Otherwise, we connect to server to ask for file if it is cache miss. HCM client first gets the metadata of target file from HCM server. Next client recognize the cache mechanism according to the file format and file size. Finally HCM client sends relative request based on the mechanism.
In write operation, we suggest taking the write-on-close scheme. When we close a file, we check the dirty bit to recognize the modification was happened or not. If the target file is WFC mechanism, HCM client writes the file entirely to HCM server; else if it is BFC mechanism, HCM client first check the ``CODE" of every blocks to collect the dirty blocks, and then send a BFC-based request to HCM server.
4.3 Initialization
As shown in Fig 2.1, the structure of a file for applying WFC is simple, just include the file header (metadata) and also a single data block without any extra overhead, thus it is unnecessary to take any preprocess on the target files. On the other hand, Fig.4.3 depicts the structure of file to apply BFC does. We can see that except for the header and the data blocks, we also define three more components ``breakpoint", ``snapshot" and ``super snapshot" to approach the BFC function. Therefore, an initialization preprocess is necessary to build up a
BFC-suit file. As a result, after HCM determined the template of a file, if the template is
belongs to BFC template, HCM then executes the initialization process. As discuss before, files to apply BFC mechanism need to add extra information.
The brief description of BFC Initialization preprocess is shown in Fig. 4.4. There are several parts in the initialization flow: Segmentation, Digest, Identification, Snapshot and Super snapshot.
The first thing we do is to divide up files into multiple blocks. If the template is
``BFCVAR", then we use we use the Rabin Fingerprint algorithm [13] [14] to get a variable size boundary, with the suggested average block size of 8KB (probability 2−13), and the range
of block size from 2KB to 64KB setting. Rabin Fingerprint is a polynomial representation of data modulo a predetermined irreducible polynomial. The reason we choose Rabin Fingerprint is that it provide an effectively computation on a sliding window on block segmentation, and we can easily have random size non-overlapping blocks. Furthermore, we claim that the block boundary should be variable, the boundary is determined based on file contents rather than by certain position, therefore insertions/deletions only affect the surrounding blocks, avoid
causing the problem of large file contents shift even only single bytes is change at the start. On the other hand, if the template is ``BFCFIX", we suppose that this kind of file will never be modified, we determine the breakpoint directly by the position, with block size 8KB. Finally if the template is ``BFCSPEC", we determined the breakpoint based on certain file content, take pdf file as an example, we set the breakpoint at the end of one page.
4.3.2 Digest
After dividing a file into blocks, we then give a digest to each block by the SHA-1 hash function [7] [15], in order to simplify the comparison of two blocks. Sha-1 hash function take a input data which size limit to 264− 1 bits, and then output a 160 bits hash value.
Theoretically, only equal contents can obtain same hash value. We can compare the equality of two blocks easily through the respectively digest. (In 2005, Wang, X. and Yin, Y.L. and Yu, H.
announced an attack method on breaking SHA-1 [16]. They indicate that they can find SHA-1 collision in less than 263operations, where a brute-force search would require280operations.
Here we still use SHA-1 because there only contain at most dozens thousand blocks in a file, and we do not use the hash code into security way, we think that SHA-1 is enough for our design. Additionally, the output of SHA-1 is smaller than upper SHA family [17]).
4.3.3 Identification
Now we divide file into block, and give each block a digest for easily compare, in the next stage we assign identification information to each blocks for searching the blocks more
effectively. In LBFS, they use a term ``offset" to find out the target block. Nevertheless, we suppose that this method would be suffer from the issue of block insertion/deletion, any insertion/deletion happen at the start may cause a large scale of ``offset" shift, then we need to modified the shifted blocks' offset record. In order to avoid the shift-offset issue, we assign a term ``BLOCK ID" to instead, then search block by block with ``BLOCK ID". However, this method may avoid the shift-offset issue, but simultaneously produce another issue--longer search time. When we search in a unsequential link-list, the BIG O is O(n) (n:the number of blocks), compare with the ``offset" method, the search efficiency is significant decrease from O(1) to O(n).
To overcome the issue, we propose to assign one more term ``GROUP ID" joining
``BLOCK ID". We arrange all the blocks in to groups, each group has an id and contain 10 blocks at default. The group size is variable, any block insertion/deletion only affect that group itself. At this situation, according to the condition of ``GROUP ID" and ``BLOCK ID", the search time decrease dramatically from O(n) to O(g). (g:average group size, approximately 10 at default). If a certain group experienced a series of insertion and group size increase to more than 30 blocks, then we can execute a regrouping process, allotting blocks to the surrounding groups until group size reduce to 20.
To maintain this method in memory, we store the blocks in a composite data structure combining an array and lists, we call it aList. Utilize the array as the group id indexer, and simultaneously have a pointer point to a link-list that store the grouping blocks. The initial process of aList is illustrated in Fig. 4.5. First we open a file from disk then store it in memory
with a char * array buffer, finally allot the blocks to aList by the ``GROUP ID" and ``BLOCK ID". Now we can find out the target file effectively with ``GROUP ID" for array index and
``BLOCK ID" for link-list search.
Fi/B11
File (Fi)
1 2 3
．．．
Disk
Fi/B12 Fi/B13 Fi/B1N
．．．
Fi/B21 Fi/B22 Fi/B23 Fi/B24 ．．． Fi/B2M
．．．．．
K Fi/BK1 Fi/BK2 Fi/BK3 ．．． Fi/BKO
Open file Fi & Read Fi to buffer
Transform to block cache Array
Buffer
F1 F2 ．．． Fn
．．．
Fa Fb ．．． Fm
Figure 4.5:Using Array and Link-listed on BFC mechanism
4.3.4 Snapshot
After dividing the file, assign digest and ID to each blocks, we build up a snapshot to store this information, and then save it in front of every block at the breakpoint.
BLOCK SIZE BLOCK ID PRECIOUS BLOCK ID
NEXT BLOCK ID
DIGEST (20 BYTES)
LAST MODIFIED TIMESTAMP (8 BYTES)
CODE GROUP ID
0 23 31
35
7
Figure 4.6: Snapshot composition
In our snapshot contents (Fig.4.6), except the SHA-1 hash digest with 20 bytes, ``BLOCK ID" and ``GROUP ID", we also define several terms for recording the block state to enhance the efficiency of BFC mechanism, they are described in the following:
• BLOCK SIZE: Record the size of this block. Use to allocate memory.
• PRECIOUS BLOCK ID: Record the precious block id of current block.
• NEXT BLOCK ID: Record the next block id of current block.
• LAST MODIFIED TIMESTAMP: Record the last modified time.
• CODE: a code stand for ``CLEAN'', ``MODIFIED", ``CREATED", ``DELETED",
``REGROUP" to express the block state. ``CLEAN" means no any modification is done on the block since last update; ``MODIFIED" means the block content is changed and
wait for update to server; ``CREATED" means we create a new block, simultaneously to indicate server to run an insert progress after receive this kind of snapshot;
``REGROUP" means that block has happened to be regrouped. Moreover, as we need a way to indicate to server that we have done a deletion, we also define a term of
``DELETED" stands for an empty block which data is cleared by user. Here we still hold the snapshot in our system even the block content is cleared, then we send the
``DELETED" snapshot to server according to the update progress, after the server receive the snapshot and detects the ``DELETED" code, server searches the certain block and then delete the block and also the snapshot to finish the action of deletion.
When the ``MODIFIED" and ``CREATED" and blocks are updated to server, then we turn them to ``CLEAN"; On the other hand, after the ``DELETED" block(``DELETED" always means empty block) is updated, we will delete the snapshot.
Totally, each snapshot is 48 bytes (with 1 byte empty for option data), which is similar to the LBFS recommended.