Ensuring the Quality of Verification: Coverage vs. Qualification

To find a design bug, three things must occur during the execution of the verification environment:

1. The bug must be activated, i.e., the code containing the bug must be exercised.
2. The bug must be propagated to an observable point, e.g., the outputs of the design.
3. The bug must be detected, i.e., behavior checked and a failure indicated.

The verification environment must exercise a path through the design (from inputs to checkers) and these temporal inter-process relationships are not present in code coverage information. It is quite possible that code will be covered only as an unintended side effect of a test case. Code coverage can indicate that code has been exercised during simulation, but this does not tell the user if a bug in the same code would propagate to a checker and cause a test case to fail. Furthermore, code coverage does not indicate if the output behavior is being checked correctly.

Functional coverage, the latest generation of coverage technology, allows verification engineers to specify what design behavior they would like to see exercised. Functional coverage does, in theory, allow some of the temporal inter-process relationships of the design behavior to be measured, but it is quite possible that an individual functional coverage point is hit as a side effect, and the related functionality can't impact the result of the scenario being executed. This is perhaps even more likely to occur when the input sequence is pseudo-randomized. The huge number of possible paths through the design, the likelihood of coding mistakes in the functional coverage code, and its subjective nature, mean it is unrealistic as a measure of completeness. Like code coverage, functional coverage does not indicate if behavior checking is sufficient. There is also no way to know which of the possible paths from inputs to checkers actually have been verified.

Functional qualification, based on functional level fault simulation, is a fundamentally different approach that measures functional verification completeness. By inserting an artificial bug into the design, functional qualification measures the ability of the verification environment to exercise the design functionality and to propagate artificial bugs to checkers that can indicate a failure. An artificial bug that cannot be detected points to a verification weakness. If an artificial bug cannot be detected, then there is evidence that actual design bugs would not be detected by the verification environment. Functional qualification is the first technology to directly measure the bug detection ability of a verification environment. Functional qualification, which effectively wraps around an existing verification environment, is not a sub-category of verification, just as equivalence checking is not a sub-category of synthesis.

1. Functional Qualification wraps around Functional Verification.

Verification is all about measuring the functional quality of design code. Since the beginning of simulation-based verification, being unable to measure the quality of the verification code has been an anomaly, since we rely on verification code to tell us about the quality of the design code. For professional verification engineers, who are experts in quality control, functional qualification is a revolution. Finally they can have objective feedback to improve the verification process and they can control the quality of the verification code.

Designers have a number of objective measurements of their work, for example the area, timing and power consumption of the design. Verification has been lacking an objective and comprehensive measurement and has therefore remained an "art." Functional qualification provides verification engineers with the same type of objective measurement designers have long enjoyed. Verification engineers can now objectively demonstrate the impact of their skills and evolve verification from an art toward a science.

Electronic design automation (EDA) researchers also have a conundrum with regard to coverage technology. To demonstrate the effectiveness of new technologies, they need a measurement. Because coverage has been the measurement, the major innovations in verification technology have been focused on how to automatically generate input sequences for covering the design. But this is not what the industry needs. EDA should automate the detection of potential design bugs, ensuring that bugs can propagate and that the design's behavior will be checked.

The EDA industry has a typical cycle: at each manufacturing node, new challenges emerge; these are solved and EDA research moves on to looking at the challenges for future nodes. Coverage is a poor solution for measuring verification quality but the industry moved on, assuming the problem was solved. Massive amounts of energy and effort have been spent pushing EDA users toward CDV methodologies. Yet there was an elephant in the room -- coverage is not a sufficient measurement!

Functional qualification can provide an objective and complete measurement of a design's verification. This can be scary for verification teams and engineers because opinions and politics will be indirectly measured. For the verification professional, this is an exciting time: they can objectively demonstrate just how good they are. We can expect to see more engineers attracted to verification when the measurements are objective. We can also expect to see more constructive innovation in future EDA verification products, which will seek to automate the propagation and checking of potential bugs. The industry needs to recognize that the coverage of yesterday is not sufficient for today's level of functional quality. Functional qualification demands a fundamental rethink about what measuring verification means and the implications are profound.

About the Author:

Mark Hampton is CTO at Certess, Inc. Prior to Certess, he was an EDA user, having worked in IC design before specializing in verification. He is currently living in Japan. Contact him at Support@certess.com.