This has been quite an important question to Software Product Managers, who have had to deal with releasing software products on Time, on Budget and with High Quality. Now all these three metrics of Time, Budget & Quality are the classic metrics to measure the results of a product manager who is building software

applications, be it enterprise applications which have well defined use cases. We at 99test believe that this is changing, depending on the type of software product one is building.

Classic Software Engineering methods such as UML, Use Case Diagrams and detailed Requirement Documents do help product managers & software developers; when they are building software products, where the customer problems and flows have been well understood and have not changed over a period of time. Now applications such as eCommerce Stores, Consumer Facing

SaaS applications, Mobile Applications and Social Applications have a completely different set of critical metrics of quality.

We are moving away from the traditional Software Quality Assurance paradigm, to releasing software applications to large number of customers in a very short time.

So the traditional methods of Water Fall development, where the bulk of the resources and efforts were in Software Assurance have no longer the same relevance when the release cycles are in Days and hours and not months


* How to estimate an Acceptable Quality Level for each Sprint?

* What is the Cost of a Showstopper Software Bug in the Product?


Part of the solution to these questions would be too look at - Impact of a Showstopper Bug to your Customers and also the Cost of Fixing a Showstopper Bug. Since most teams are moving away from Software Assurance in the Water Fall Model to Agile Development Process. The Product Manager needs to understand the resources available to him and engage the software testers early in the sprint to find critical bugs. The estimation of the cost of a showstopper bug could be as follows.

* Measure the Impact of a Showstopper Bug. i.e. What will be the cost to the business due to one showstopper bug in dollars?

* What will be the cost to fix one Showstopper Bug? when reported by customers.

* How many such Showstopper Bugs can be tolerated in one Sprint?

* How much effort needs to put in exploratory testing of the application for finding Showstopper Bugs?

So the summary is that cost of a showstopper bug varies across different types of software products. We can no longer take just the engineering view of the Quality, but will have to take into account the type & number of users of the Software Product and the tolerance for bugs in a sprint. i.e. Impact of a Showstopper Bug needs to be estimated. How much of revenue loss will be caused by a showstopper bug? to both the users and customers is the Cost to the Customer, due to a Showstopper Bug. Next would be to calculate the Cost of fixing that bug when reported by customers.


Cost of Acceptable Quality in one Sprint = ( Cost of a failed Customer Transaction + Cost of fixing a Showstopper ) * Number of Showstoppers * Number of Users/Customers


Subtracting the cost of fixing the bug when reported by customers by the incremental cost of testing will give a good metric as to how much of resources need to deployed to ensure an Acceptable Quality Level in the software product for this Sprint. Assuming that a Software Tester will find one or two critical bugs in one exploratory testing session of 1 hour. Thus, by calculating the worst case, i.e. by making a table of the expected functionality and then running through the table to calculate the cost of failure or bug in that particular functionality, multiplied by the Impact will give a full picture of the maxim damage of the bug in software. The interesting part of calculating the impact of the bug is that, as bugs become hard and harder to find, the impact of hard to find bugs might be inversely proportional to the cost of finding them. i.e. Product Managers need to become highly skilled in estimating the impact of Showstopper Bugs and then planning to get highly skilled testers to address the cost of finding critical bugs through exploratory testing methods and crowd testing, along with getting developers to write unite test cases, to prevent simple functionality related bugs with in modules, as most of the bugs do lurk in the integration points and state changes in software applications.

​