Web Application Bug Analysis

I've gone through all our past bug reports with Bluto with the goal to find the impact automated testing with Selenium would have on bug reports and prevention. My raw data is attached, which you may peruse at your leisure. It's located in the attachments section at the bottom of this page.

Overview of Bug Reports

The break down of closed bug reports:

Fixed	Open	Duplicate	Won't Fix	Works for Me	Remind	Later	Invalid	Total
501	90	60	36	41	1	20	57	806

Effort was focused on the 501 fixed bugs, and categorized them as either valid ("It broke, please fix it"), a feature ("It would be better if..."), or invalid (doesn't pertain directly to Bluto).

70.5% of fixed bug reports were considered valid bugs.

Areas going Obsolete

ie: Product Evolution

Bug reports were categorized as either current, or obsolete. If the feature the bug targeted is not in the current version, then it was marked obsolete. Those marked current mean that the bug report was referring to a feature that is still in existence within the current version of Bluto.

For example, 24% of bug reports were feature requests. Of them, 17% went obsolete.

70% of reports were "valid" bugs, and only 11% were of areas that eventually went obsolete.

Although Bluto went through a significant evolution, it appears that 88% of the features and capabilities remained. We shouldn't then be too timid to create a test for a feature because it is likely the feature could remain for a long while.

Obviously, just about every single feature went through it's own evolution, and that will break tests. But that is OK, because:

• A broken test alerts the tester what areas have been changed, which will focus their attention on the areas that need to be retested
• The act of remaking the test will quickly bring forward any new bugs
• Important areas won't be forgotten (like the link farmer)

Areas of Focus

An initial pass was made to categorize which areas the fixed bug reports pertained to. Given the quantity of areas, a graph may be the best way to display the results.

Bluto Bug Analysis: Location of Bug Reports

As could have been guessed, the most complicated part of the application has the majority of bug reports.

The other two large areas are harvesting and exporting, which I believe may be the easiest to test. It seemed the bug reports pertained mostly to the harvesting or exporting either working or not working, rather than an idiosyncrasy within it.

Potentially, a deeper analysis could be had to find the key areas that both a) break frequently, and b) easy and quick to test. This could give developers a short targeted test suit to run after major revisions during further development for Bluto. For Popeye, it can help us predict areas of complication and potentially create it's own short targeted test suit.

Automated Bug Testing

Automated testing isn't going to be able to catch everything, so a testing analysis was also done to see what can be tested, and what can't.

What Bugs can have Automated Testing

All fixed bugs that weren't feature requests were analyzed for the ability for Selenium to test it. This is a rough pass, and as we get more familiar with Selenium, or utilize other testing suits, we may be able to have the ability to expand testing.

63% (223) of bug reports could have testing run on them. Of them, 18% (62) required a special-case-scenario, generally a specific calendar, or types of calendars.

31% (110) of bugs could not have testing. This was because:

• You need eyes to see the problem, ie graphical errors: 50% (55)
• The bug was outside the realm of a browser, ie email, or export contents: 18% (20)
• A human is needed to comprehend the problem, ie misspellings or repeated objects: 17% (19)
• The process to make a test is more complicated than I could figure out: 15% (16)

Prevention through Automated Testing

If a bug could have automated testing, it was analyzed for the possibility that it could have been preventable, or discovered pretty quickly.

104 (46.6%) of bugs could have been prevented or discovered quickly. Another analysis should be done to find which were areas that broke frequently. A report was marked as preventable if a bug had been previously filed for that feature.

13% (28) would have been discovered during the process of making a test. This was generally a feature that was broken in the first place, or a bug that would be found through thorough testing of the feature.

41% (91) of bugs could not be prevented. This was because:

• Special case scenario. Some quirk within a calendar caused subsequent problems: 64% (58)
• Someone wouldn't think to do that. It's an odd occurrence that isn't part of typical workflow: 30% (27)

Future with Testing

What will automated testing do for the future of Bluto and creation of Popeye?

We shouldn't be overly timid in creating tests, as our history with Bluto shows that our features do not go obsolete at a significant rate.

We should make a small targeted test suit that all developers can optionally run after major revisions, and testers could run after every upgrade. This will:

• Quickly alert developers to changed areas, so they can evaluate if those changes will impact their work
• Alert testers of changed areas so they can focus their attention to those areas
• Prevent critical areas from going unnoticed

Testers can run a full suit several times a week. In addition to the above bonuses the full suit will have, it will also free up the testers time to find those weird, special-case-scenario bugs and bugs that occur when a user breaks from predicted workflow.

New features should have tests written for them. The very act of writing the test can significantly help find bugs.

Implementing Testing

A few things probably need to be ironed out in regards to testing.

• Who will write tests? Will both developers and testers write and update tests, or will the testers primarily be responsible for it?
• How will we distribute tests? In terms of the full test suit, one set could be installed on the general test server. As for the targeted test suit, how could it be distributed to all developers?