Popular Posts

Sunday, June 24, 2007

QA in open source projects

Quality assurance is a traditional activity in software development life cycle which starts from the inception stage of a project and flows through design, execution, release and maintenance phases. However these type of clearly defined phases cannot be realized in most of the open source projects. Rather defining SQA activities in to different stages, agile QA approaches emerge to facilitate quicker development and release cycles.
It is always not possible to define a properly managed quality assurance process in most of the open source projects. Strictly defined deadlines, properly documented requirements and designs, well planned releases are essential supporting components for having a good QA process. However most of the above (Sometimes none of the above) tasks are not done in open source software development. Due to these circumstances QA cannot control the whole process.
Is it possible to maintain an effective QA process in open source projects? Is it possible to adhere to a schedule even if a development process is not defined? How does QA control the open source product releases? Any QA engineer who involves in open source projects may raise these question.
I tried to find out answers for the above questions.

Maintaining a QA process

Most of the QA processes consist of the following phases.

Test requirements identification
Test design
Test execution

Due to the poorly documented open source software requirements, QA test requirement identification may be difficult. However QA can start this activity through wikis, mailing lists and they can publish the identified test requirements in a QA test plan. QA test plan will be the primary source of the quality assurance test activities which are going to be executed throughout the project. Also, Live updates on QA test plan is possible if it is wiki based.

In commercial software projects, QA test designing is performed by referring to the requirement specification. The use cases are converted to a set of test scenarios and which are used to derive test cases. How are tests designed when requirement specifications are not available? QA Test designing is an endless activity in most open source projects. The initial set of test cases are designed using the basic project requirements which are captured from the discussions happening in mailing lists and wiki pages. These are refined when a build is released for verifying the basic functionalities of the product. Subsequently more scenarios are captured and test cases are derived using them. But it is not the end of test design. More and more test cases and scenarios are identified when more functional builds are released for testing.

Test execution is a two-fold activity in open source projects. Initially QA team does ad-hoc testing to check the basic functions of the AUT (Application under test). Then the execution of test cases is carried out. If a new scenario is captured while performing the ad-hoc tests, it is amended to the QA test case document. So that, QA team is equipped with a comprehensive set of test cases when the final builds are available for testing.

QA Scheduling

Following the defined activities and maintaining the schedule are important features of a good QA process. However in open source world, schedule is haphazard. Projects are not governed by strict deadlines and plans. Because of that, builds are received randomly for QA testing. There is no strictly defined 'First Build to QA' date. When a set of features are done, QA gets a build for verifying the implemented features. However these random QA builds are really important for QA team to capture more test scenarios while studying the product.
Usually release candidates (RCs) are delivered for QA testing 2-3 weeks before the final product release (This depends, of course :) . Release candidates are equivalent to the official QA builds which includes almost all the functionalities. QA team conducts thorough functional and regression tests in these RCs. The effort and duration for testing a particular RC will be subjected to change even if it is planned initially due to the randomly added features and changes in the final release date.

Controlling product releases

In most commercial projects, an approximation about the release can be obtained through the QA metrics. Defect severity index, defect density provide a better indication of the project status and product stability. However, this is not exactly applicable for open source projects. Last moment changes are happened and regression issues are introduced. However it is really important to set an acceptable release criteria for any opensource project. For example, release will not be accepted by QA team unless blockers and critical bug count are zero. Also QA has to convince developers to resolve all the open bugs and update the status in defect tracking system. This will provide a clear indication on the release status.


Iranga said...

Charitha it would be great if you could share the following on QA Open Source projects

1.)How the non functional requirements are identified?
2.)Estimation prepared?
3.)QA task tracking?

Charitha said...

1). We don't have customer defined non-functional requirements when a open source product development starts. However, the discussions about the product architecture is taken place in mailing lists. During these discussions, non-functional requirements can be captured.

2). It is really hard to estimate QA effort and schedules in open source projects. As I can see, the primary reason for this is QA can't take the control of release cycles as in commercial projects. However test execution effort is estimated by assessing the count of functional test cases and different platforms need to be tested.

3). There is no project plan update process in open source projects I'm working in. So that, it is the sole responsibility of the QA engineers to follow up the bug cycles to minimize the open defect count and release the product adhering to the release criteria.