Testing is a time consuming process and hard to automate. Its main purpose is to make
sure every functionality works. The simplest suggestion is to “show it wrong or right”. If we
do the testing manually, we could lose in the labyrinthine program execution states. To
automate the testing processes, it is essentially important to direct the testing program. We
divide ALERT into three stages: instrumenter, symbolic evaluator, and logic resolver. Each
stage will generate inputs for next stage. A control top module exchanges messages between
every stage. It is a better architecture for testing duty.
Program states consist of thousands of variables and their combinations are
unpredictable. We know it is impossible to enumerate program states through the values of
variables. So we try an alternative way to enumerate program logic. It is much feasible that a
program behavioral representation can be considered as the combination of its logic
constraints. There are many related work that take advantage of program logic. ALERT is the
first application for dynamic analysis.
We discover that program logic is reversible under some circumstances. The reversible
states enable ALERT to solve constraints and produce feasible inputs that lead program to
perform a different execution path. By recording constraints of each running instance, we can
enforce the program to perform all of its possible paths.
6.1. Future Work
ALERT is just a beginning. For the different testing strategy, such as integration testing,
we can enhance ALERT in the component interaction aspect, which enable ALERT to resolve
more complicated program logic. In addition to simulating the manual input operation,
ALERT could be extended to generate specified environments for exhibitions of security
issues by modification of the messages transmitting between the logic resolver and symbolic
evaluator layers.
6.1.1. Feasible Input Generation
We desire a tool that tells when and how a defect or fault could occur, especially when
we don’t have enough information to prove the existence of the feature. ALERT is just a
beginning of that. It automatically resolves specified program logic (or called constraint rules)
and produces feasible inputs to render program in the way we want. Not only single function,
but integration constrains can be evaluated as a collection of symbols. We look forward to a
way of understanding the program semantics automatically. This method would become a
new trend in the field of automatic testing to deal with every logic feature in program.
6.1.2. Delta Debugging
In the former software debugging process, we are familiar with getting in the reference
between source codes and runtime interaction. To find out what is going wrong behind
programs, it would be convenient for programmers to automate the process that traces
programs without manual assistance. Delta debugging[2] is a latest debugging method that
supports automatic software fault localization. Generally, to report buggy sites, Delta
debugging requires the comparison of a failure program run with a successful one. Moreover,
the failing program execution characteristics do not actually differ much from the successful
program execution. Delta debugging eliminates the common characteristics through
comparison of value transaction and cause chain of two runs. Nevertheless, in real cases, we
only have program failing runs. That is a tough work to determine the similarity of failing run
and successful runs with which we can apply Delta debugging method.
References
[1] "gcov: a Test Coverage Program http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html,"
[2] H. Cleve and A. Zeller, "Locating causes of program failures," in ICSE '05: Proceedings of the 27th International Conference on Software Engineering, 2005, pp. 342-351.
[3] W. R. Bush, J. D. Pincus and D. J. Sielaff, "A static analyzer for finding dynamic programming errors," SPE, vol. 30, pp. 775-802, 2000.
[4] J. Whaley, M. C. Martin and M. S. Lam, "Automatic extraction of object-oriented
component interfaces," in ISSTA '02: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, 2002, pp. 218-228.
[5] D. Jackson, "Aspect: detecting bugs with abstract dependences," ACM Trans. Softw. Eng.
Methodol., vol. 4, pp. 109-145, 1995.
[6] D. Engler, D. Y. Chen, S. Hallem, A. Chou and B. Chelf, "Bugs as deviant behavior: A general approach to inferring errors in systems code," in SOSP '01: Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, 2001, pp. 57-72.
[7] S. Hallem, B. Chelf, Y. Xie and D. Engler, "A system and language for building system-specific, static analyses," in PLDI '02: Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation, 2002, pp. 69-82.
[8] H. R. and J. B., "Purify: Fast Detection of. Memory Leaks and Access Errors." Winter USENIX. Conference, 1992.
[9] P. Godefroid, N. Klarlund and K. Sen, "DART: Directed automated random testing," in PLDI '05: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, 2005, pp. 213-223.
[10] B. Demsky and M. Rinard, "Data structure repair using goal-directed reasoning," in ICSE '05: Proceedings of the 27th International Conference on Software Engineering, 2005, pp.
176-185.
[11] H. Agrawal and E. H. Spafford, "Bibliography on debugging and backtracking,"
SIGSOFT Softw. Eng. Notes, vol. 14, pp. 49-56, 1989.
[12] T. Wang and A. Roychoudhury, "Automated path generation for software fault
localization," in ASE '05: Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, 2005, pp. 347-351.
[13] C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe and R. Stata,
"Extended static checking for java," in PLDI '02: Proceedings of the ACM SIGPLAN 2002 Conference on Programming Language Design and Implementation, 2002, pp. 234-245.
[14] lp_solve http://lpsolve.sourceforge.net/
[15] K. Sen, D. Marinov and G. Agha, "CUTE: A concolic unit testing engine for C," in ESEC/FSE-13: Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, pp. 263-272.
[16] SWI-Prolog -- an LGPL comprehensive portable Prolog compiler.
http://www.swi-prolog.org/
[17] zlib -- A Massively Spiffy Yet Delicately Unobtrusive Compression Library.
http://www.zlib.net/