Popular Posts

Friday, September 19, 2008

QA role in Hackathons and Risk based testing

A few days back, we had a hackathon at WSO2 in order to resolve a large number of open bugs and boost the schedule of Data Services 1.0 release. A set of developers and QA team sat together and went through the open issues and eventually got more than 100 defects fixed and verified.
If you are familiar with hackathons or if you are a tester, you may certainly raise the following questions.
- What is the role of QA in a Hackathon
- How to deal with the regression issues and frequent builds

It certainly is a challenge to involve in testing and quality assurance during a hackathon. There is a high risk of breaking the features in between builds. However if you manage it properly, Hackathons are good opportunities for QA and testing teams to get more familiar with the AUT (Application under test) hence uncovering more and more bugs, learn and share the knowledge of practical use cases of products etc..

Lets try to find out answers for the above questions from QA user point of view.

What is the role of QA in a hackathon?

As defined in the wikipedia, Hackathon is an event when programmers meet to do collaborative software development. Testers, BAs, PMs and other astake holders may also take part in it.

The primary role of QA/test teams in a hackaton is reproduce the issues which were reported by them. This involves functional issues, cross browser issues as well as cross platform related bugs. The reporter of the bug knows the way to recreate it than any other.
This will greatly reduce the time to reproduce bugs and help developers to get them fixed effectively.
In most occasions, the AUT is started in debug mode in testers machines and someone familiar with the code, debug the source remotely to get the issues fixed quickly.

Whenever a bug is marked as resolved, testers get an update from source repository (in our case SVN), build the source and verify and close or reopen the bug.

Hackathons are ideal places to discuss the usage scenarios/patterns of the product being developed. QA/testers can identify more and more scenarios and apply them in testing. This help to uncover a lot of hidden bugs.

It is an essential for QA/testers to have and set the source code up in their IDEs. If you are familiar with the code and know the root cause of the issue, you are free to go ahead and fix the bug by your self. We did it and worked well :)
One of the most important requirement to have source with QA during a hackathon is, QA can go through a module with some developer and try to find out the places where bugs can be introduced. This glass box testing approach is one of the most important feature in a hackathon.

During a hackathon, QA find bugs and report it immediately to the relevant developer and get it fixed. QA should make sure to track all issues using the bug tracking system though some of the issues are reported verbally.

Dealing with regression issues and frequent builds

While fixing more and more bugs it is essential to ensure that the regression issue are minimum. Regression testing is the most difficult task for QA in a hackathon. Whenever a set of bugs are fixed, you get an update from source repository and build in your machine. Then you verify the bug and close it.
When you do this, there is a high chance to missing more and more regression issues. So, as a QA tester, how do you deal with it?

Before starting a hackathon, you should have a proper idea about the most critical features of the AUT. You should identify the areas which can cause most damages to the recognition of the product as well as for your customers. Then you should keep in mind that these critical features are not regressed in any new build use for testing. In other wards, though you test 4-5 new builds per day, you must ensure that the highly risked features are verified and tested properly.
Risk based testing is the most suitable approach to ensure the critical features are not broken during a hackathon.

It is important to get a few labeled builds for testing during a hackathon. We could not do this in Data service 1.0 hackathon though. We cannot guarantee the quality of the builds taken from users local repositories. Therefore, to make sure the AUT is in a proper state, it will be essential to have a labeled QA release built from a pure build environment and release for testing.

After all, Data Service 1.0 hackathon went really well and helped to deliver the product on time with acceptable quality!

No comments: