Software Has Defects - Hardware Fails - No Process Is Perfect
An article in the October, 2004 edition of Crosstalk, The Journal of Defense Software Engineering states that the average defect removal efficiency for US software projects is 85%, with successful projects (i.e. on-time, on budget, meeting requirements) attaining a 95% defect removal efficiency although this number will decrease as defects are reported from users.
Since the much publicized Pentium floating-point error in 1994, the vast majority of microprocessor suppliers have published known defects for each processor. Failures of circuit boards are frequently attributed to bad batches of components such as capacitors or resistors.
Formal processes can reduce defects and improve overall quality, but even stringently applied processes cannot achieve zero defects. This applies to steel manufacturing, concrete production, jet engines .. virtually every non-trivial product.
Deal With It - We Can Help
Despite the fact that defects may exist within the pumps, valves, tubing, software, et cetera within any given jet engine, air travel remains remarkably safe. Jet engine manufacturers address potential defects and failures through defensive design (e.g., backup systems, diagnostics, automatic shutdown sequences, et cetera). Commercial aircraft designers specify highly reliable jet engines, but still design aircraft to fly with an engine failure.
ProDesCon approaches every job with the following core assumptions:
All delivered software/firmware has defects
All hardware will fail given sufficient time
Testing can demonstrate the existence of faults and defects
Testing cannot demonstratethe absence of faults and defects
Elimination, or even identification, of all defects is essentially impossible. Moreover, the behavior of any given system subjected to abnormal conditions and events is a function of the design. Undesired system behavior during abnormal conditions does not necessarily indicate a defect.
ProDesCon focuses on the function(s) the system is providing, the consequences of failed or faulty operation, and conditions and mechanisms which can result in failed or faulty operation. Our investigations use both inductive and deductive techniques to provide better coverage.
Effective risk management requires careful analysis. Attempting to fix each specific condition in isolation will likely increase overall complexity, consume valuable resources, and may create new failure modes. In some cases, user training may be the most effective method for dealing with the situation, in cases involving commercial equipment a simple parameter change may invoke optional checking, and some conditions may be so unlikely that no action may be the most reasonable course.
When no credible conditions leading to undesired behavior are found, or when consequences are minimal, the technical examination provides a model for future systems.
Levels of Digital Construction Model
Traditional software quality assurance standards divide systems into hardware and software. However, the complexity of today's digital and computer-based systems cannot be described using blanket terms such as software or hardware.
Complex logic devices such as Application Specific Integrated Circuits (ASICs) burn dedicated logic onto a single chip. Graphical workstations allow engineers to define control logic by dragging function blocks onto a screen and then identifying inputs and outputs. Is the ASIC hardware or software? Is the construction of a graphical logic representation programming or configuration?
ProDesCon has developed an alternative model to address: digital complexity, system integrity, Commercial Off The Shelf (COTS) equipment, complex logic devices, drag and drop graphical programming/configuration, security, and identification of low-value high-cost activities.
Click to view chart in PDF format
Our model defines a structure and taxonomy which provides a logical means to address complexity. The model’s structure defines levels of construction. Each level constructs a product which provides some new functionality using products constructed by the level below. For example, the people who design and manufacture circuit boards (Module Internals level), assemble the final product using components designed and manufactured by others (Component level).
The term software may be applied to:
- micro code running in a microprocessors,
- assembly programs performing low level diagnostics,
- compilers,
- C++ programs,
- ladder logic executing in a PLC, and
- even graphical configuration
While simple, perhaps even obvious, our model provides a powerful framework. We have used our model to analyze training requirements at various levels, establish documentation architectures, conduct gap reviews, and system integrity reviews.