An area that has not been looked at in much detail yet on this blog is what software or tools you can use to record and track bugs, issues or defects that you find when testing a website.
One of the first posts I wrote, which described a typical Web Testing process, mentioned that a bug tracking system ‘can be hugely beneficial, as you can set priorities for each bug and then understand how many bugs you have at each priority level. It gives you a much better understanding of how many bugs you have to fix before you can launch and calculate how long you expect that process to take.’
There are a good range of bug tracking applications out there, a lot of them web-based, which are vital when tracking bugs and associated information about each issue. Here we take a look at 4 of the main players in bug tracking software, which are also available in our Testing Tools Directory.
Once you start testing a website you can come up with all sorts of bugs and issues that need addressing. I’m talking a bit more than the odd typo or broken image. Perhaps you find problems with the site search returning strange results or the checkout process not working correctly or there are specific issues in different browsers that work fine in other browsers. There are several different reasons to use an application that tracks bugs and feature requests.
As bugs start to get a little more complex it is important to hand the developer all of the required details about each bug so that he or she can fix it in a reasonable amount of time, without chasing around trying to reproduce it or get a screenshot or URL showing where that bug was found. Bug tracking software gives you a structured format to report bugs so that you include the URL where you found the bug, provide a full description and a screenshot and what browser or operating system you were using. These details will shorten the amount of time that a developer needs in order to squash the bug and improve their productivity. Get more bugs squashed in less time.
Alongside the complicated bugs will still be the relatively quick and simple items to fix such as the odd layout problem, typo, broken image or link, etc. These bugs, whilst they do need to be fixed, are sometimes not as important as other bugs and, when presented in a list or on a spreadsheet, can often be completed ahead of more important items because the developer naturally follows the list through from start to finish. Bug tracking software allows you to prioritise bugs so that developers work on the most critical issues first and leave the less important bugs until later. Rearranging priorities in a list or on a spreadsheet gets too cumbersome when dealing with a large number of bugs but issue tracking software handles this aspect with ease.
With bug tracking software you can assign bugs to different developers so everyone knows what they are working on. Once those bugs are completed then the developer assigns them back to you for checking or verifying. If you are happy you can close the bug, if not then reopen it and give it back to the developer. Do this for each bug and you can understand exactly where each bug is in the process, how many are assigned, how many are to be verified, etc. It allows you to have a much better handle on exactly where your web project is at instead of continually poring over your list or spreadsheet and trying to make an educated guess.
This improved workflow eases pressure on you and your developers and improves productivity because developers can just get on with fixing bugs or working on feature requests. It also means that you are not constantly updating lists and spreadsheets with the latest status of all the bugs reported.
Once bugs are fixed, tested and closed they can be referred to at any time so that you can see when the bug was reported, when it was fixed, who fixed it and what the developer did to fix it (assuming they fill their comments in correctly). This provides an extremely useful archive if you ever need to refer to a particular bug again instead of trying to go back through emails or out of date spreadsheets to find details on that bug or feature request.
Some bug tracking systems allow time tracking so you can see how much time was spent on fixing a bug. Using bug tracking software ensures that developers become more accountable because their comments and times are recorded and can easily be referred to.
Now that we’ve established what bug tracking software offers and why web project managers and testers should incorporate it as part of most web projects (definitely on all large web projects and probably on most small to medium projects too) let’s have a look at some of the main bug tracking applications currently being used.
My specific experience is with using Bugzilla so let’s start with that one.
Bugzilla was originally released by Netscape in 1998 and so has been around for a long while now. This open source software is one of the first bug tracking applications that spring to mind and, although it takes some setting up and the interface is a bit tricky to use, it has everything you need to record and track bugs effectively.
It is possible to sidestep the web interface by using a desktop client that synchronizes with the central bugzilla software. My specific experience with bugzilla was using the Deskzilla client on a Mac, which worked well.
As Bugzilla is open source then it is free to use although it needs to be installed and set up on a server in order to be ready to use.
Unfuddle is a hosted bug tracking application that also includes Git hosting and Subversion hosting allowing you to create your own secure code repositories. There are also some project management tools included such as tracking your project against specific milestones and time tracking is available in the more expensive corporate and enterprise packages.
Bugs, issues and feature requests are raised as tickets in the Unfuddle system and each ticket includes workflow management and an audit trail so you can keep track of who made changes to a ticket and when these changes were made.
If you need the code repositories and project tracking functionality along with a straightforward bug tracking system then Unfuddle could be the answer. There is a free package available that includes 1 project for 2 users. After that prices start at $9 per month.
Trac is also open source software that combines an issue tracking system with a wiki and project management features such as milestones and being able to document the roadmap for a web project or software development. Trac can also serve as a web interface for version control systems giving you a usable front-end for Subversion or Git.
Tracking progress of individual bugs, issues or feature requests is straightforward with each one having its own numbered ticket with the ability to filter tickets by severity, project component, version or owner. Keeping on top of progress is also easy with the ability to set milestones and being able to see how far you are from reaching each milestone.
Trac has a good set of features and is a lightweight and flexible system that should be very useful for most web development projects. It runs on most operating systems but does require some setting up and this is perhaps its biggest downside that I can see if you want to start tracking bugs straightaway.
FogBugz is available as a hosted application, called FogBugz on demand, or as installed software on either Windows, Mac or Unix operating systems.
There are several parts to FogBugz, the issue tracking section makes it very easy to report bugs and you can even create bugs via email or create bugs quickly without having to fill in everything so you can get the bug into the system. Whilst I think this is a good feature you do need to make sure you go back and complete all the details before you assign the bug to a developer to work on. Everything is saved in a complete case history for future reference.
But issue tracking is not all that Fogbugz does, there is a project planning section where you can outline tasks, break larger tasks into subtasks, enter estimates and set milestones. Fogbugz also has something called Evidence-Based Scheduling, which looks at what tasks are being estimated and how long they actually took to complete and then plot a probability curve to work out whether a milestone can be hit or not. The FogBugz Wiki allows you to keep all project documentation in one place and there are also a whole load of plugins available to extend the functionality plus FogBugz can integrate with another piece of software from the same company (Fog Creek Software) called Kiln, which is a version control system.
FogBugz is the most expensive of the bug tracking software applications featured here with prices starting at $25 per user per month for FogBugz on demand or $999 for 5 users for the installed version of the software.
Whichever of the above applications you use, as long as you use one of them and get your workflow organised in terms of how you report, record and assign bugs then you will find that it becomes a breeze to keep track of exactly where you are at in your website testing. You won’t miss any issues (as long as they are reported and entered into the bug tracker), you can prioritise showstoppers or critical bugs easily to fix those first and archive completed bugs for future reference.
If I had to pick one of the applications above then I would go for either Unfuddle or FogBugz (on demand) because I like web-based applications as there is nothing really to set up or install. Both of these have a trial although both cost money whereas Bugzilla and Trac are free but require time installing and setting them up on a server.
FogBugz has the better overall feature set and I am intrigued as to how the Evidence-Based Scheduling works, as I have not come across that before. But it terms of a cheap, easy to use, bug tracking system that is ready to go and will do the job, I would pick Unfuddle with the caveat of going for FogBugz if I had a bigger team (and didn’t mind paying more money).