Evaluation of Software Obfuscation

Software obfuscation increases difficulty in reverse engineering by transform-ing an original program into an obfuscated one which thwarts reverse engi-neering but preserves the original functionality [17]. Despite the theoretic proof of the impossibility of omnipotent obfuscation [36], obfuscation is still able to reach positive results in specific situations [37] and implementation of obfuscation have been widely discussed [17, 18, 19, 20, 21]. According to [17], obfuscation is classified into three types: control-flow obfuscation, data obfuscation and layout obfuscation. Control-flow obfuscation disguises the real execution under scrambled control flow of a program to make re-verse engineering difficult. Various implementations have been introduced to accomplish control-flow obfuscation [18, 38, 39, 40]. Data obfuscation transforms data and data structures in a program without modifying the original functionality. [21] and [41] presented obfuscating transformations by extending the concept of data obfuscation. Layout obfuscation removes the information that an attacker can seize from the program. Most of the commercial obfuscators such as Dotfuscator [42], DashO [43], Zelix [44] and ProGuard [45] adopt the basic idea of layout obfuscation.

Each type of obfuscation provides effective though limited resistance against malicious reverse engineering. In recent years, many researchers proposed various evaluation methods to assess the effectiveness of an obfuscating

trans-formation. The methods are mostly based on empirical analysis, which eval-uates an obfuscating transformation by running practical experiments to ob-serve how much the obfuscated program resists against deobfuscators or how much time a human subject takes to interpret it [46, 47, 48, 49, 50, 51].

Udupa et al. [52] examined control flow flattening, a control-flow ob-fuscating transformation, by measuring the time required by automatic de-obfuscation. Anckaert et al. [47] introduced a framework to evaluate an obfuscating transformation based on software complexity metrics, which cal-culate the complexity with respect to instructions, control flow, data flow and data. The authors implemented three obfuscating transformations (con-trol flow flattening, static disassembly thwarting and binary opaque predi-cates) and applied the transformations to eleven C programs of the SPECint 2000 benchmark suite, and the obfuscated programs were produced from the benchmark suite. The results of the complexity analysis show that all of the three transformations can provide non-negative effects, but the transforma-tion, binary opaque predicates, is less potent than two others. Majumdar et al. [48, 53] considered a specific reverse-engineering technique, slicing, and developed metrics to evaluate the capability of obfuscation against that technique. [48] and [53] presented three obfuscating transformations (bogus predicate, adding to a while loop, and variable encoding) and applied them to five example programs to derive the values by the defined metrics. The metric values imply that these transformations significantly make reverse engineering difficult.

Ceccato et al. [49, 50] assessed the difficulty an attacker would encounter in examining identifier renaming, one of the obfuscation techniques, by

ques-tionnaires. The authors asked human subjects to interpret the original and obfuscated programs and to fulfill a comprehension task. The subjects were also asked to fill in a post-task survey questionnaire to describe their behav-ior during the task and the confidence about it. Certain types of statistical tests, such as the Mann-Whitney test and the Wilcoxon test, were adopted to analyze the task results and the questionnaires. The analysis results point out that identifier renaming effectively reduces the capability of the subjects to understand the source code.

The existing work [47, 48, 49, 50, 52] evaluated the effectiveness of obfus-cation by empirical studies. Practical experiments were performed to mea-sure individual obfuscating transformations according to the defined met-rics or the perception of human subjects. These experiment results indicate the relation specifically between a designated original program and a sin-gle obfuscating transformation. While the same obfuscating transformation is intended to be applied to another program, the experiment results may not be applicable to determine the capability of that transformation in the case. In addition, the existing experiment results of evaluating individual transformations cannot help determine the effectiveness of a compound ob-fuscating transformation, which comprises several separate transformations.

It thus requires a formal method for evaluating obfuscating transformations in a high-level of abstraction.

Preda and Giacobazzi [54] proposed a formal method for analyzing the effect of a control-flow obfuscating transformation based on program seman-tics. They considered a specific control-flow obfuscating transformation, which obscures the control flow by inserting opaque predicates. They

eval-uated the transformation by analyzing the effects of the opaque predicates inserted. They also modeled attackers for comparing obfuscating transfor-mations. Their method is the closest to ours, which evaluates control-flow obfuscating transformations based on formal analysis as well. However, our method can formalize and evaluate more types of control-flow obfuscating transformations, not limited to the type of inserting opaque predicates.

2.3 Summary

We reviewed and analyzed the attack graph-based and the AHP-based meth-ods for network risk assessment in terms of external attacks. The analysis showed that there is a need to propose a new risk assessment method, which can represent the risk in the real-world and is capable of addressing the dy-namics of a network. We also discussed the existing methods of evaluating obfuscation. Most of them are empirical-based and examine the effects of obfuscating transformations by experiments; however, a formal method is necessary to help system administrators systematically and effectively assess the capability of obfuscation.

Chapter 3

Risk Assessment of Wireless