Simulation-based functional validation is still one of popular means to verify a digital hardware design described in a HDL. In simulation-based validation, the circuit behavior of an implementation described in a HDL can only be compared against the expected behavior or the specification on Observation Points (OPs). Even if some design errors are executed and activated, the erroneous effects caused by the design errors are still required to be propagated to the OPs for error detection.
In this thesis, we have implemented a probabilistic observability measure for HDL descriptions. Unlike tag-based approaches, which can provide only two levels of measurement, our fine-grained observability measures have less possibility of overestimating the extent of validation with reasonable computation time. Even when multiple errors occur, we still can provide some meaningful values for users' reference to reduce the risk of misleading the verification results. This is better than using binary decisions only.
The proposed probabilistic observability measures can be used to replace tags for the application of observability-based code coverage metric. In addition, since hard-to-observe points can be identified using our observability measure, designers can insert assertions in those locations to find hard-to-observe bugs more easily. This observability-driven assertion insertion is simple, but should be very effective.
The proposed observability measures for HDL descriptions can also be applied to assist debugging faulty HDL designs when a discrepancy between the simulation values of the OPs and the expected values occurs. The probabilistic observability measures can be used as a new probabilistic confidence score, which has better capability of estimating the likelihood of correctness for error candidates in error space. The experimental results shown in section 4.5 confirm that the proposed PCS measurement is indeed accurate in estimating the likelihood of correctness such that accurate debugging priority can be obtained. As a result, debugging priority sorted with the proposed PCS can effectively speed up error searching process in the input error space. As compared to CS in [46], the PCS-based debugging priority can save more than half of the efforts (or time) needed for error searching process in an error space in our experiments, at the cost of little extra computation time. The time saving contributed by the proposed PCS method should usually be much larger than the extra computation time the PCS calculation needs. Therefore, the gain of the proposed PCS can often outweigh the cost of extra computation time the PCS needs.
One possible future research direction is to generate a test vector set that creates some highly transparent sensitized paths to propagate potential incorrect values of the exercised statements to OPs for higher observability-based coverage. Other possible future improvements may include 1) more accurate observability estimation approaches for multiple paths, 2) a more accurate probabilistic observability measure by considering the probability distribution of each signal, and 3) integrating our dump-file based observability and PCS-based HDL debugging approach with commercial HDL simulator to form an efficient verification/debugging framework.
These future directions may provide a comprehensive solution for the observability issue during simulation-based functional validation.
Bibliography
[1] S.Tasiran, K. Keutzer, “Coverage Metrics for Functional Validation of Hardware Designs,” IEEE Design and Test of Computers, vol. 18, no. 4, pp. 36-45, July-August 2001.
[2] Collett International Research, Inc., http://www.collett.com.
[3] V. Bhagwati and S. Devadas, “Automatic Verification of Pipelined Micro -processors”, Proceedings of IEEE/ACM Design Automation Conference, pp.603-608, June 1994.
[4] R. B. Jones, D. L. Dill, and J. R. Burch, “Efficient Validity Checking for Processor Verification”, Proceedings of IEEE/ACM International Conference on Computer-Aided Design, pp.2-6, November 1995.
[5] R. E. Bryant, and Y.-A. Chen, “Verification of Arithmetic Functions with Binary Moment Diagrams”, Proceedings of IEEE/ACM Design Automation Conference, pp.535-541, June 1995.
[6] J. R. Burch, E. M. Clarke, K. L. McMillan, and D. L. Dill, “Symbolic Model Checking: 1020 States and Beyond,” Information and Computation, pp. 428-439, August 1992.
[7] A. Biere, A. Cimatti, E. M. Clarke, and Y. Zhu, “Symbolic Model Checking without BDDs,” Proceedings of International Workshop on Tools and Algorithms for the Construction and Analysis of Systems, pp.201-206, May 1999.
[8] F. Copty, L. Fix, R. Fraer, E. Giunchiglia, G. Kamhi, A. Tacchella, and M. Y. Vardi,
“Benefits of Bounded Model Checking at an Industrial Setting,” Proceedings of International Conference on Computer Aided Verification, vol. 2102, pp. 436–453, July 2001.
[9] A. Biere, A. Cimatti, E. M. Clarke, M. Fujita, and Y. Zhu, “Symbolic Model Checking Using SAT Procedures instead of BDDs,” Proceedings of IEEE/ACM Design Automation Conference, pp. 317-320, November 1999.
[10] P. Bjesse, T. Leonard, and A. Mokkedem, “Finding Bugs in an Alpha Microprocessor Using Satisfiability Solvers,” Proceedings of International Conference on Computer Aided Verification, vol. 2102, pp. 454-464, July 2001.
[11] H. Foster, A. Krolnik, and D. Lacey, Assertion-Based Design, Kluwer Academic Publishers, 2003.
[12] B. Beizer, Software Testing Techniques, 2nd edition, New York: Van Nostrand, 1990.
[13] R.C. Ho and M.A. Horowitz, “Validation Coverage Analysis for Complex Digital Designs,” Proceedings of IEEE/ACM International Conference Computer-Aided Design, pp. 322-325, November 1996.
[14] M. Benjamin, D. Geist, A. Hartman, Y. Wolfsthal, G. Mas, and R. Smeets, “A Study in Coverage-Driven Test Generation,” Proceedings of IEEE/ACM 36th Design Automation Conference, pp. 970-975, June 1999.
[15] J. Shen, and J.A. Abraham, “A RTL Abstraction Technique for Processor Microarchitecture Validation and Test Generation,” Journal of Electronic Testing:
Theory and Application, vol. 16, nos. 1-2, pp. 67-81, February 1999.
[16] S. Ur and Y. Yadin, “Micro Architecture Coverage Directed Generation of Test
Program,” Proceedings of IEEE/ACM Design Automation Conference, pp. 175-180, June 1999.
[17] D. Moundanos, J.A. Abraham, and Y.V.Hoskote, “Abstraction Technique for Validation Coverage Analysis and Test Generation,” IEEE Transactions of Computers, vol. 47, no.1, pp. 2-13, January 1998.
[18] S. Devadas, A. Ghosh, and K. Keutzer, "An Observability-based Code Coverage Metric for Functional Simulation," Proceedings of International Conference on Computer-Aided Design, pp. 418-425, November 1996.
[19] F. Fallah, S. Devadas, and K. Keutzer, "OCCOM: Efficient Computation of Observability-based Code Coverage Metrics for Functional Simulation," IEEE Transactions on Computer-Aided Design, vol 20, pp. 1003-1015, August 2001.
[20] F. Fallah, I. Ghosh, and M. Fujita, "Event-driven Observability Enhanced Coverage Analysis of C Programs for Functional Validation," Proceedings of IEEE Asian and South Pacific Design Automation Conference, pp.123-128, January 2003.
[21] F. Fallah, S. Devadas, and K. Keutzer, "Functional Vector Generation for HDL Models Using Linear Programming and Boolean Satisfiability," IEEE Transactions on Computer-Aided Design, vol. 20, pp. 994-1002, August 2001.
[22] M. Abramovici, M. A. Breuer, and A. D. Friedman, “Digital Systems Testing and Testable Design,” Piscataway, NJ: IEEE Press, 1990.
[23] L. H. Goldstein, "Controllability/Observability Analysis," IEEE Transactions on Circuits and Systems, vol. 26, pp685-693, September 1979.
[24] C. H. Chen and D. G. Saab, "A Novel Behavioral Testability Measure," IEEE Transactions on Computer-Aided Design, vol. 12, pp. 1960-1970, December 1993.
[25] S. Bhattacharya, S. Dey, and F. Brglez, "RT-level Transformations for Gate-level Testability," Proceedings of European Conference on Design Automation, pp. 162-169, February 1993.
[26] P. A. Thaker, V. D. Agrawal, M. E. Zaghloul, "Validation Vector Grade (VVG): A New Coverage Metric for Validation and Test," Proceedings of IEEE VLSI Test Symposium, pp. 182-188, April 1999.
[27] B. Murray and J. P. Hayes, "Hierarchical Test Generation Using Precomputed Test for Modules," IEEE Transactions on Computer-Aided Design, vol. 9, pp.
594-602, June 1990.
[28] S. Ravi, G. Lakshminarayana, and N. K. Jha, "TAO: Regular Expression-based Register-transfer Level Testability Analysis and Optimization," IEEE Transactions on VLSI Systems, vol. 9, pp. 824-832, December 2001.
[29] J. Voas, "PIE: A Dynamic Failure-based Technique," IEEE Transactions on Software Engineering, vol 18, pp. 717-727, August 1992.
[30] S.Y. Huang, K. T. Cheng, K. C. Chen, and D. I. Cheng, “Error Tracer: A fault Simulation-based Approach to Design Error Diagnosis”, Proceedings of IEEE International Test Conference, pp. 974-981, October 1997.
[31] D.W. Hoffmann, and T. Kropf, "Efficient Design Error Correction of Digital Circuits", Proceedings of IEEE International Conference on Computer Design, pp.
465-472, September 2000.
[32] M. Tomita, and H. H. Jiang, "An Algorithm for Locating Logic Design Errors", Proceedings of IEEE/ACM Design Automation Conference, pp. 468-471, June 1990.
[33] P.Y. Chung, Y.M. Wang, and I. N. Hajj, "Diagnosis and Correction of Logic
Design Errors in Digital Circuits", Proceedings of IEEE/ACM Design Automation Conference, pp503-508, June 1993.
[34] A. Smith, A. Veneris, M.F. Ali, and A. Viglas, “Fault Diagnosis and Logic Debugging Using Boolean Satisfiability,” IEEE Transactions on Computer-Aided Design, vol. 24, pp1606-1621, October 2005.
[35] J. R. Lyle, and M. Weiser, "Automatic Bug Location by Program Slicing", Proceedings of the Second International Conference on Computers and Applications, Beijing, China, pp. 877-883, June, 1987.
[36] M. Weiser, "Programmers Use Slices When Debugging", Communications of ACM, vol. 25, No. 7, pp.446-452, 1982.
[37] B. Peischl and F. Wotawa, “Automated Source-level Error Localization in Hardware Designs,” IEEE Design and Test Computer, vol. 23, no. 1, pp.8-19, January-February 2006.
[38] M. Khalil, Y. Le Traon, and C. Robach, “Towards an Automatic Diagnosis for High-level Validation”, Proceedings of IEEE International Test Conference, pp.
1010-1018, October 1998.
[39] C. H. Shi, and J. Y. Jou, "An Efficient Approach for Error Diagnosis in HDL Designs", Proceedings of IEEE International Symposium on Circuits and Systems, pp.
IV-732- IV-735, May 2003.
[40] B.R. Huang, T.J. Tsai, and C.N. Liu, "On Debugging Assistance in Assertion-based Verification," Proceedings of the 12th Workshop on Synthesis and System Integration of Mixed Information Technologies, pp. 290-295, October 2004.
[41] Y.C. Hsu, B. Tabbara, Y.A. Chen, and F. Tsai, "Advanced Technique for RTL
Debugging," Proceedings of IEEE/ACM Design Automation Conference, pp.362-367, June 2003.
[42] J.P. Roth, “Diagnosis of Automata Failures: A Calculus and a Method,“ IBM, Journal on Research and Develop, vol. 10, pp. 278-291, June 1980.
[43] F. Fallah, “Coverage-Directed Validation of Hardware Models,” Dissertation of M.I.T, 1999.
[44] J. Voas and K. Miller, "Software Testability: The New Verification," IEEE Transactions on Software Engineering, vol. 12, pp. 17-28, May 1995.
[45] J. Voas, G. McGraw, L. Kassab, and L. Voas, "A “Crystal Ball” for Software Reliability," IEEE Computers, vol. 30, pp. 29-36, June 1997.
[46] T.Y. Jiang, C.N. Liu, and J. Y. Jou, “Effective Error Diagnosis for RTL Designs in HDLs”, Proceedings of IEEE 11th Asia Test Symposium, pp. 362-367, November 2002.
[47] A. Offutt and G. Rothermel, “An Experimental Evaluation of Selective Mutation,” Proceedings of International Conference on Software Engineering, pp.
100-107, May 1993.