QA City

   QA City >> Expert Column
Dont Miss Experts

Predicting Software Release Quality-Part A

Ajay Kumar Chhokra
Ajay Kumar Chhokra
Director - Testing Center of Excellence, UHG India - IT
Ajay is part of UHG India IT Leadership Team and is responsible for driving Testing Competency an... more>>

Every one wants to know or have control over the Future. The same holds true for the Project Manager Community. A number of techniques for Quality Prediction are known but most of them are just theoretical concepts based on statistical models and just a few are based on past data. Many projects have done pilots, but none of them have emerged as a single Model for predicting Software Release Quality. This paper talks about two simple empirical approaches which can help any PM to get confidence about the quality of the release.

The first technique is based on the past Defects data and its glide path for which one needs to have all the past defect data for both pre-release and post release. Any special causes may have influence on the quality prediction. The second technique is based on Development team confidence. The development team includes developers, Analysts, Designers, Testers and Project managers. But in order to get better results, one has to use both the techniques.

Keywords: Prediction Markets, Forecasting, Release Quality

Description of the Article:

This article describes the various Forecasting methods and how past data can help in forecasting the vital signs of any project delivery. It also talks about evolution of Forecasting methods to Prediction markets covering its methodology, various contracts terms, its principles, pitfalls and success Factor. At the end, there is a case study where both Prediction Markets and Forecasting concepts have been applied to a project in UnitedHealth Group Information Services and the learning and outcome came out of those pilots. Finally, the paper talks about Conclusion and Future prospects of Forecasting and Prediction markets techniques to the IT industry.

Forecasting Methods

Forecasts essentially provide future values of the time series on a specific variable such as sales volume or Number of Defects in a Release. Thus, Forecasting methods can be divided into two broad categories on the basis of availability of historical time:

Qualitative Forecasting Methods

Qualitative forecasting techniques generally employ the judgment of experts in the appropriate field to generate forecasts. A key advantage of this procedure is that they can be applied in situations where historical data are simply not available. The important qualitative forecasting methods are Delphi technique where consensus is reached by having number of discussions.

Quantitative Forecasting Methods

Quantitative methods come in two forms: time-series methods and explanatory methods. Time-series methods make forecasts based purely on historical patterns in the data. On the other hand, explanatory methods use other data as inputs into the forecasting data. Time-series methods are probably the simplest methods to deploy, cheap to run, and relatively easy to interpret. Most quantitative forecasting methods try to explain patterns in historical data as a means of using those patterns to forecast future patterns.

Applicability of Forecasting Techniques

In consumer goods marketing, for example, these types of forecasting techniques are often used to assess a brands baseline performance. In IT industry, more of Quantitative Techniques are applied. These are more applied in product industry, where based on past data, one can predict the future case. For example, if we know, in past few releases, what has been the pattern of Defects identification in Alpha and Beta phase of product testing, the product companies are using those patterns to find

• Effectiveness of System Testing teams in terms of finding defects on time
• Extending the Defect curves and its glide path will give indication on where this release quality is headed.

Examples of applying Forecasting Techniques to Release Quality

The above chart shows that within first 4 weeks of System Testing of a Quarterly Release.

The above chart shows the trend of contribution of Non Value Added Defects.

Difference between Forecasting and Prediction

Prediction Markets

The fundamental principle behind Prediction Market is to bring a group of participants together and let them trade contracts whose payoff depends on the outcome of uncertain future events such as political or social events. Prediction markets are thus structured as betting exchanges, without any risk for the bookmaker. People who buy low and sell high are rewarded for improving the market prediction (profits), while those who buy high and sell low are punished for degrading the market prediction (losses).

However, many prediction markets use imaginary money rather than real money. Still, evidence shows that prediction markets using fictional money can be successful at making accurate predictions. Few of the characteristics of Prediction markets are

- aggregate information from multiple disparate sources
- relatively immune to coercion and manipulation
- offer incentives and rewards for consistent good performance
- have a market maker that enables them to function in low participation settings

About a dozen exist and are open to the public, including the Popular Science Predictions Exchange, TradeSports, the Iowa Electronic Markets, NewsFutures, Bet2Give, Hollywood Stock Exchange, The simExchange, Intrade, and Betfair. They are generally centered around a general domain of predictions, such as politics, movies and films, and technology and futurism. Other names for prediction markets include decision markets, information markets, idea futures, event derivatives, and virtual markets.

Prediction Market Contracts

There are various types of contracts in Prediction market. The simpler one is "winner-takes-all" contract:

I4C Principles

There are five principles of Prediction Markets known as I4C
For the participants motivation, Incentive plays a vital role in getting accurate prediction. Under those principles, a prediction market works in part because the price of shares for different forecasts serves as a concise "indicator" of information for all participants. The pricing process conveys information to uninformed participations and therefore helps them "improve" their knowledge about demand. These work best when there is a large, "independent" "crowd" of participants who receive strong monetary "incentives.

These must have the following characteristics:

Diversity of opinion to draw on different points of view, from participants with access to different information;
Independence of members from one another to avoid thought leaders influencing the opinions of others;
Decentralization to ensure that local and specific needs are taken into account;
• A reliable method for aggregating opinions to ensure that all results are taken into account, but avoids the perils of decision by committee. This is the trickiest part.

Prediction Market Pitfalls

A few common pitfalls of prediction markets: participants who lack relevant information, too few participants or too little trading

Experts on QA
Swaid Qadir Bhat
Sr System Architect
Virtusa Corporation
Subhash  Motwani
Prasad Rao Pasam
Ayaskanta  Mohanty
Managing Director
TATWA Technologies
Rajesh  Dagar
Software Architect
Connect Icon Pvt Ltd
Yasar  Khuthub
Software QA Manager
Azure IT Solutions
Sunil  Bhat
Project Management
HCL Infosystems Limi
Sharad  Agarwal
Team Lead