I thought I would share this small piece of advice. It might be obvious to a lot of people but is not always apparent or sometimes we forget or even run out of time to pay enough attention to it.

The advice is simply this:

When you report a bug or issue and the developer tells you [...]]]>

I thought I would share this small piece of advice. It might be obvious to a lot of people but is not always apparent or sometimes we forget or even run out of time to pay enough attention to it.

The advice is simply this:

When you report a bug or issue and the developer tells you it’s fixed, always check that it is actually fixed.

Sounds obvious doesn’t it?

Think back to times when you have taken the word of the developer that a particular bug has been fixed and decided not to test it or verify that it had actually been completed. Ok, it sounds a bit strong that you knowingly decided not to test the bug fix, you could have been swayed by a couple of factors. Perhaps that developer has always fixed bugs satisfactorily in the past or maybe you didn’t have the time to check the bug had been fixed properly and so gambled that it had been fully dealt with.

Whatever the reasons the result is that the bug fix did not get checked out and verified. For many issues, the fix put in place would simply have worked and that would be the end of the matter. That particular bug would never be heard of again, the gamble paid off.

But what are the chances that the fix did not work, or created a new previously undetected bug or that the developer misinterpreted the bug report and his or her fix is not what you expected. In my experience, the chances of things like this happening are pretty good (website testers are such a pessimistic bunch).

On occasion, bug ‘fixes’ like this actually have a bigger impact on the website than the original bug. They can make the situation worse rather than better. A minor bug that is improperly fixed can have a much bigger effect and without testing and verifying it properly you don’t know if that has happened this time or not.

Fixing A Bug And Making It Worse

To illustrate how it is easy to fix a bug and make it worse, consider the following.

Some work I have been managing recently involved using a fairly simple script. The way the script works is that you send a series of locations to it, in the form of a CSV file, and the script fetches data about each of those locations, which is then uploaded to an email marketing system.

The data that is returned by the script is dropped into personalised emails, which are regularly sent to thousands or even tens of thousands of people at a time.

The script is not the greatest of solutions but it is a fairly quick and straightforward piece of programming, which generally works well.

I reported a bug that for one particular location, no data was being returned by the script. We had to manually add the data for that one location, which was frustrating and time consuming but not the end of the world. A developer investigated the bug and soon after pronounces the issue fixed. The information is relayed to me that the bug is fixed. Everyone rejoices (not really, everyone just gets on with the next job).

When I check the script over, send some location data to it and review the results I find that data is indeed now being returned for the offending location. However, it is the wrong data. There are 2 locations in that particular city and the script is returning data for the other location.

Just to recap, the original bug was that no data was returned by the script for that single location. An inconvenience in that the missing data had to be manually added. The bug ‘fix’ has actually introduced a 2nd bug, which is that the incorrect data is now being returned for that single location, a more significant bug than the previous one if it went unnoticed, as thousands of individuals would have had incorrect personalisation details incorporated into emails they received from the organisation creating a great deal of confusion.

It just goes to show that when someone tells you a bug has been fixed, please always check. Something may indeed have been fixed, just not always what you anticipated. On occasion, the situation may have actually been made worse and not better.

I am sure that there are lots of similar stories to this one where a bug fix did not fix the bug but perhaps made it worse. Feel free to share any in the comments.

Image credit BenSutherland

This ad hoc approach to testing means that it is possible to miss [...]]]>

As a website is being built, the step of writing or putting together a test plan is often skipped entirely and testing just ‘begins’. If a test plan is done then it is often completed immediately prior to testing and completed too quickly.

This ad hoc approach to testing means that it is possible to miss large sections of functionality and not be able to fully consider all permutations or variations of events and actions that need to be tested.

For most websites, there is a fairly long list of items that need to be tested as part of any new website development and in order to know what to test, when you tested it and whether those tests passed or failed, it is best to make a proper plan.

We are going to show you how to write a simple and straightforward test plan, which will provide a more systematic method for testing your website but the process outlined below is still essentially a manual one. We are not yet moving into using software tools to help identify and run tests, nor are we covering automated testing here, although I hope to write a post on these aspects in due course.

Writing A Test Plan

A typical test plan should identify all the areas of the website that you need to test before it launches, which you then follow through to conduct your testing, making notes of when you completed the test and adding any bug or issue reports to your bug tracking software along the way.

Start off with getting all the main sections down in a spreadsheet, divided between front end and admin or CMS if the website has one.

Then, start to break each main section down into the constituent parts that you need to test, concentrate on the important parts of the website first such as the home page, which is likely to be one of the most visited pages and the main areas of functionality, which are most likely to be the most used.

For instance, in the case of an ecommerce site, this would include individual tests for items such as adding products to the shopping basket, removing products from the shopping basket, updating the shopping basket quantities, being able to enter the checkout process and completing the checkout process (depending on your website there may be several steps to test within the checkout process).

You should be asking yourself questions for when you start testing the checkout process from the point of view of the user such as:

• Can I complete my order?
• Can I go back and amend my order?
• Do I understand where I am in the checkout process?
• Can I clearly see what I am ordering and how much it will cost, including delivery and taxes?
• Do I get a receipt or email confirmation?

As the owner of the website there will be different questions such as:

• Does the payment processing work correctly?
• What happens if the incorrect card details are entered?
• Does the order confirmation process work correctly?

And so on, when putting a testing plan together you are testing the website from the point of view of the user and also from the point of view of the owner or operator of the website. You may also have specific usability testing lined up but the primary reason for the test plan as far as this blog post is concerned is to test the functional aspects of the website to ensure they work correctly. Does the website do what it says it does? For instance, if the checkout process displays the order details including taxes and delivery then does it show the correct details for both tax and delivery?

Another important area to test is the site search or product search, as this is likely to be used a lot on the website.

Make sure that you test all the forms within the website, such as enquiry forms and newsletter signup forms. Test that the form can only be submitted after the mandatory fields have been completed and that the form is emailed to the correct email address or saved to the database correctly.

Once you go through and think about each section of the website along the lines of the above then you should have compiled a decent list in your spreadsheet of areas and items to test. Add columns for each browser that you need to test each item on, which will be the web browsers that you have agreed to support as part of the project. This would normally cover the major web browsers such as IE6, IE7, IE8, Firefox 3, Chrome and Safari but may include others too depending on the usage of your target market (or existing audience if you have those statistics). There are a range of browser checking tools that can help you with this aspect.

For the admin system or CMS that powers the website, add a row to your test plan for each function. Going back to our ecommerce example, this means that there would be tests for add new product, edit existing product, remove product, view order, edit tax rate, etc. depending on what the system was capable of.

You could break these tests down further to test individual elements of adding a new product. For instance, is there validation present to check that mandatory fields are filled in when adding a product? Does the product image have to be a certain size or dimensions? Does the product title have to be shorter than a certain number of characters? All these items could occupy separate rows in your test plan.

Of course, if you were to break your test plan down to this extent for a large website or CMS then the result will be a long spreadsheet of items to test, which will take a great deal of time to test effectively. You will need to decide whether you have the time available to test in this level of detail or if some areas can be tested more quickly.

A few other items that you may wish to make sure you have as part of every test plan are the following, in no particular order:

• Ensure your HTML and CSS code validates to W3C guidelines
• Test accessibility of the website
• Test the SEO items that you have planned for to make sure they are in place
• Test against the specification for the project that was signed off to ensure that everything has been completed that was agreed
• Test against designs and/or wireframes or prototype html mockups that were completed to make sure the finished website matches the signed off designs or wireframes
• Take into account any changes requested (should be in the form of change control documents)
• Complete any security tests that are required

You may wish to have a look at our list of 10 different forms of website testing to make sure you have covered everything. The list can go on indefinitely, there is always something that can be tested a bit more, but at some point it has to be halted at a relevant point otherwise the testing will never get done.

List or Flow Chart?

Some aspects of the website lend themselves to having a flow chart in place that describes all the permutations or possibilities that need to be tested. This is especially helpful if one action then presents another set of possibilities. When testing without a flow chart it can become extremely difficult to remember which permutations you have tested and which you have not.

Having a flow chart or mindmap worked out that shows you all the possibilities allows you to think more clearly about what you are testing, what you expect the outcome to be and understand what you have tested and what is remaining to test.

Conclusion

If there is too much testing to do for one person then divide it up amongst other people with instructions on what you want testing and what to do with bugs when they are found. Make sure they follow their bit of the test plan and feed back on what they find, handing bugs to you or inputting them into your bug tracking software themselves.

As you test then fill in the test plan to show what you tested and when. If you keep the document client friendly then it can be shown as proof of what testing was completed prior to launch.

Testing is a vital part of any website development and having a decent plan before you start testing saves a great deal of time and effort whilst making sure that all the required testing gets done.

Forms can be one of the most common areas for problems on a newly launched web site. It can often be the case where a web site visitor takes the time to fill in a form, perhaps to enter a competition or make an enquiry to your company, they submit it and [...]]]>

Testing Web Forms

Forms can be one of the most common areas for problems on a newly launched web site. It can often be the case where a web site visitor takes the time to fill in a form, perhaps to enter a competition or make an enquiry to your company, they submit it and what happens to the details they just filled in? Nothing, the details do not get captured correctly and so are lost.

I have seen it happen many times where a contact form that was believed to be working correctly on a live web site was not sending the details correctly and so countless enquiries were being lost. This can obviously be extremely damaging to your company and very frustrating.

Even more worrying, sometimes this problem goes unnoticed for some time. Think about the forms on your web site, when did you last check that each one was working 100% correctly?

So what should you look out for when testing forms? Our quick guide aims to help you with the important points in 7 easy steps.

1. Fill in the form and check what happens to the data. If it is supposed to go into a database then either check the database yourself or have a developer check it for you and show you the results. Make sure that the data you input is placed in the correct fields in the database. If the form is going to be sent by email then make sure that the form does go to that email address and review the results.

If you are testing on a development or staging web site address, i.e. not the final live web site, then also complete this test once the web site is live. Put it on your post launch checklist now.

2. Some fields will most likely be required – fields such as Email Address are often mandatory. Make sure that they are labeled as being required (usually with an asterisk placed next to the field name) and test completing the form without filling in that field to check that the correct fields are mandatory. Also check the error message is display and make sure that it makes sense.

3. Some fields will most likely require validation – it is generally good practice to validate some fields to check that site visitors input the right type of data. For instance, you may wish to validate the email address field to make sure people type in correctly formatted email addresses. This will not check that the email address itself is correctly but at least the data input resembles a correctly formatted email address.

If you input an incorrectly formatted email address (or whatever validation you are performing) then check that the form does not submit and that an error message is displayed that makes sense to the user.

4. What happens if you don’t enter anything? – make sure that the form does not submit and highlights the fields that are required to be completed displaying helpful and easy to understand error messages.

5. Test that you cannot input html or sql code that you should not be able to do. We will cover code injection in more detail in a later post but for now please understand that code injection can be a serious security risk and it is important that any vulnerabilities are closed. For more information on sql injection please read the following article – http://www.owasp.org/index.php/Testing_for_SQL_Injection_(OWASP-DV-005).

6. Make sure that the form displays correctly and works as intended (and as already tested) for each of the web browsers that you have determined to support. This means going through the same points from 1 to 5 for each web browser.

7. If by submitting the form the user should receive an autoresponse response email, such as a thank you for submitting the form then make sure you receive that autoresponder email. Also check that the sender name, sender email and subject are correct, that the email does not go into your spam folder and that the copy within the email is correct. Finally, check that all links within the autoresponder email direct the user to the correct location. If the autoresponder email is graphical then make sure it displays correctly in all the main email clients.

By following those 7 steps you should have a pretty robust form that has been tested thoroughly, which not only works correctly but is also user friendly.

Right, I’m now going to make sure the contact form on this web site stands up to these 7 steps.

This short guide will hopefully give you an idea of where to start so you [...]]]>

It can be pretty daunting before you set out to test a web site because it is important to get it right and to have a smooth launch with no bitter aftertaste of experiencing bugs once the web site is live.

This short guide will hopefully give you an idea of where to start so you can successfully prepare for and carry out the testing of your web site.

Get Organised

The first thing to do when testing any form of web site or web application is to get organised, especially if testing is relatively new to you. There are a number of things to have in place before you start testing that will make your life a lot easier and help you to stay organised throughout the testing process.

Ensure You Have a Testing Process

If you have a small development team then the process to follow of finding bugs, fixing bugs, checking they are fixed is fairly straightforward. Larger teams may get a bit more complicated but let’s start at the basic process. This assumes that you are going to use or already use some form of bug tracking software, such as Bugzilla or Unfuddle.

• Project Manager carries out tests as per testing plan (more on that in a moment) and enters any bugs he or she finds into bugzilla.
• Project Manager assigns bugs to a Developer to fix.
• Developer reviews bug, fixes the item and marks it as fixed and resolved (if you are using bugzilla) assigning it back to the Project Manager for checking.
• Project Manager checks the bug and if it is fixed to his or her satisfaction then marks it as closed. If the bug still has issues then the Project Manager assigns it back to the Developer with further instructions and we go back to point (iii) above.

The above process is about as simple as you get, just 4 steps to follow and the process assumes that you have just the one Developer and one Project Manager who is testing, assigning bugs and checking the Developer’s work.

Add in another Developer or several Developers to this process. All the Project Manager needs to do is when he or she finds bugs to assign them to the most relevant Developer. So you might have organised one Developer to work on a particular section of the site or quite often one Developer will work on the front end whilst another works on the back end admin system. This then makes life easier when assigning bugs to Developers so they do not step on each others toes when fixing bugs. Even with source control it can happen where code gets overwritten or mistakes creep in when Developers are working in the same section.

So now we have a Project Manager and several Developers in the testing process. What about if we add in more people testing? You might have several project stakeholders wishing to test and offer their feedback or you may let the designer review the site and offer feedback regarding how the finished site corresponds to the original designs that were completed.

The ideal situation is that all people testing use the same bug tracking software, enter any bugs they find but ensure that they assign the bugs to you, the Project Manager. All bugs from other sources need to go to one person to review them, weed out any duplications that are already being worked on, organise who the bugs are going to be assigned to and set priorities (see below about setting priorities).

If you cannot get people to use the bug tracking software, and let’s face it bugzilla’s interface is horrible so some people just refuse to learn it, then make sure they prepare a list and give that list to you, in an email if necessary. You then need to add those bugs into the bug tracking software yourself but it does give you a chance to review them along the way.

You will easily be able to see how many bugs are at each stage of the process, i.e. assigned to Developers (effectively in progress), resolved and assigned to you for checking and closed.

The final point to add with regards to the process is if the web site you are testing is for a client. It is up to you whether you update the client on all bugs found and being worked on and most bug tracking tools allow the export of all bugs so you can send a full update to the client. Some web based bug tracking tools even allow the client to login so they can see exactly what is happening.

Prepare a Testing Plan

Getting a testing plan down on paper helps to set your mind on what actually requires testing and allows you to think things through properly so you do not run the risk of missing anything out.

It can often happen when you are short of time with the deadline fast approaching that you just ’start testing’. This is often a false economy because you launch yourself into the testing and miss important items because you did not take a step back to think it through properly.

Take a few minutes and start putting down the items that require testing. This should really consist of each expected outcome, for example, you click on a link and what do you expect to happen? If it is an add to basket link on a product page then the expected outcome would be that the product is added to the basket.

For large web sites this approach is time consuming and ends up being a very long list. If you are short of time then cut this down to list the main elements that you need to test or at least go through every page on the site.

For content managed web sites or sites that have an admin system it is often helpful to split the testing into the front end web site and then the functions within the content management system as a separate set of testing. By doing this you can concentrate first of the web site that the general public will see and make sure that is right and then focus on the CMS to make sure that works correctly for whoever will be maintaining the web site.

Your testing plan should also take the original specification for the web site project into account. I am assuming that this specification was updated during the project to incorporate any changes requested and so is up to date. If the specification is not up to date then it gets very confusing very quickly when you reach the testing stage.

You can run through the specification and list out the functionality stipulated into your testing plan with the expected outcomes being as the specification described. When you test each item you can then make sure that the web site works in the way that the specification envisaged. If the web site is different from the specification then you can go back to the developer to ask them why and either get it changed to how it was originally specified or retain the existing functionality whilst preparing your response to the stakeholders as to why it is different.

Another set of items that your testing plan should accommodate are any signed off designs that were completed. The site should be tested against these designs to ensure that the layout, style and specific design elements that your designers sweated over have been carried over correctly to the site build. It can often happen where developers have not been quite so careful on replicating the exact layout that the designer worked hard on and, even if there are only minor discrepancies throughout the site, these all add up to a web site that is not as good as it could be.

Finally, make sure your testing plan includes all the items outside of actually clicking around the web site. For instance, what emails does the web site send out? Make sure testing each of these emails is in your testing plan. Do you need to test that the HTML is standards compliant? Make sure this is in your testing plan. Have you listed the browsers you are going to support in your testing plan to make sure you test in each browser?

Get Your Testing Tools Together

In order to keep your momentum up once you start testing, prepare the tools you are going to use in advance. Browser testing can be completed in a number of ways, from having a set of machines available with each browser version installed or using a browser testing tool such as Xenocode or Browsercam.

If you have these tools ready before you start testing then you can easily hop onto another browser and carry on testing without having to stop, work out how you are going to continue, get it organised and then start again. Too many of these interruptions really adds time to the overall testing phase of the project.

Conclusion

That is a fairly quick run through of things you can get in place before you start testing that will help you save time and confusion whilst you are in the thick of things.

Let me know if there are any other items that help you to test that you can prepare for in advance.

With respect to the above statement, I have put a general testing process together that I normally look to follow, which looks something like this:

Plan testing at the start of the project – [...]]]>

It is often said that no web project is the same and when applied to the testing of web sites the truth remains.

With respect to the above statement, I have put a general testing process together that I normally look to follow, which looks something like this:

Plan testing at the start of the project – allocate the right amount of time for testing, if you are not sure how much time you need then make your best guess and double it. Always allocate more time than you think you need. If you have the time now then write a basic testing plan at the start of the project because this will give you an idea of how much time you will need to carry out all of your testing.

Testing plan – when we prepare to start projects we don’t always have time to complete all the supporting documentation and the testing plan is one item often put off to later. Try to write your testing plan early on but definitely before you start testing because then you know all the items that you are going to test and do not head into the actual testing without really knowing what you are testing.

Development – as the development of the web site comes to an end, I start to casually check it over, not really testing this point but just seeing if there are any big gaps in the development and asking the developer questions to make sure he/she is aware of items they have left to complete. This is really used to find out how far off finished I think the site is versus how far off the developer believes it to be. If this exercise sets off any alarm bells because of any huge holes or missing functionality then hopefully there is enough time to put things right.

Once the developer has completed the web site I make sure that they have tested their work first, including browser compatibility so that they are comfortable before they hand it over to me to test. This is important, as I believe that developers should make sure they test their work in the first instance and do not waste my time with anything that they have not tested. If I hit too many problems in my first round of testing I normally stop and hand it straight back to the developer with what I have noticed so far. I ask them to finish the web site and give it to me when they have tested properly and not waste my time any further.

It really depends on your relationship with the developer or development company on how you go about this. With some developers I will be blunt and others I will be more diplomatic and tactful. The way I approach this also depends on the timescales that we are running to and how the project has been going to date. If we are really up against the deadline and/or it has been a difficult project then I will be a lot more accommodating with the developer and go through problems with him/her so that they feel that we are all together trying to get this web site finished on time.

If the work is below par and time is not so much of an issue then I will be blunt and ask them to only give it back once they have properly tested and not waste my time testing sub-standard work.

Carry out testing – I often complete my testing in rounds so that I go through my testing once, find a load of bugs, get those fixed and then carry out a second round of testing. This second round is used to not only check that the bugs from the first round have been fixed satisfactorily but also to delve a bit deeper into areas that perhaps I was not able to test fully the first time around (usually because the presence of bugs preventing me from doing do).

This can be time consuming to carry out testing in rounds, especially if you have to wait for bugs to be fixed before you can complete the next round of testing. However, I believe that it ensures that the majority of the bugs are picked up. If the web site is more complex then generally you will need more rounds of testing before you can be confident that you have found everything.

Once you are happy that the web site has been fully tested and all bugs have been captured (or as many as possible) then it is case of getting as many fixed before the deadline and you will probably have to prioritise.

Prioritise – If you are running out of time then you will need to decide which bugs absolutely have to be fixed ahead of launch and which can wait until after launch or be fixed at a later date. This is crunch time and the worse case scenario here is to move the launch date back so that any showstopping items can be fixed.

This is where using a bug tracking system can be hugely beneficial, as you can set priorities for each bug and then understand how many bugs you have at each priority level. It gives you a much better understanding of how many bugs you have to fix before you can launch and calculate how long you expect that process to take.

Launch Checklist – So you have fixed all the bugs that you are going to fix before launch and now the launch of the web site is upon you. I find it hugely beneficial to prepare a checklist of all the steps that we need to go through in order to launch the web site or final checks that need to be completed so that I can be confident that all is well when launching.

The length of this checklist will be greatly dependent on the complexity of the project, for some projects it may only be half a dozen to a dozen points and for others it could run into a lot more. A series of projects I managed had launch checklists of over 50 items with pre-launch, launch and post-launch sections.

You can have a standard checklist that you adapt for each project you make live such as whether the site you are launching is new or replacing a current site and if it is replacing a current site whether it is launching on a new domain name or not.

What are you going to do with current subscribers, have you checked that your Google Analytics code is in place, do you need a holding page to go up and, if so, has this been readied and signed off by the client for use? There are a lot of final items such as these they need to be checked over and should not be missed because, during the pressure of making your web site live, missing one of these items out could make all the difference.

Post Launch – The checklist that I maintain has a number of items on it with regards to post launch. There are some checks that I carry out a day or so after launch to ensure that the site is running smoothly and all is well. There may also be a few bugs or improvements to finish off that you could not squeeze in before you made the site live.

Ongoing – There may also be some ongoing tests that you wish to carry out from time to time to ensure that the continued operation of the web site is successful. Just because you launched successfully does not mean that the web site will sit there happily letting visitors browse and register and login and do whatever without throwing a wobbly from time to time.

The busier your web site gets (and I hope it does get busier) the more issues could be made apparent by the site slowing down or not being able to cope with the visitor levels hitting the site. Reviewing your web site analytics and server logs can give you a good picture for what is happening and also some regular site monitoring can give you an indication of when your site hits problems.

If you carry out any updates to your web site then these will need to be retested each time before going live and could potentially introduce further problems so ongoing and continued testing can be important.

Client Acceptance Testing – of course, throughout the above process you have to consider the client. I have specifically left the client out of my description of typical events above because it depends on who the client is in your situation, whether the web site is for you, for the organisation that you work for or a traditional agency/client relationship. Whoever the client is they will need to review and approve the web site before it is launched.

Please let me know if you have any feedback on this typical web testing process, if you would add anything or if you disagree with any items above, all comments are welcome.

You might be searching for something in particular and you a click on a link and find a ‘Page not found’ message or you want to buy a product and you hit ‘Buy now’ [...]]]>

How frustrating is it when you are browsing the web and you hit an error or a problem on a web site?

You might be searching for something in particular and you a click on a link and find a ‘Page not found’ message or you want to buy a product and you hit ‘Buy now’ but nothing happens.

Very frustrating. But your need has not been fulfilled, you still want to find that thing you were looking for or you still want to purchase that product. So you click back to your search engine results and click on the next web site that looks like it might have what you are looking for.

The owner or operator of those web sites have lost out because their web site had a problem, either an intermittent or permanent issue that prevented the user, or customer, from getting what they wanted.

How do you know if your web site suffers from problems such as this? How do you prevent similar problems if you are launching a new web site?

Through testing your web site, again and again.

Testing Web Sites is a new blog that aims to offer hints and tips for free to make it easier for you to test your own web site. We hope you find the information provided useful and that you will contact us if you do not have the inclination, motivation or simply the time to complete the testing yourself.

As we post articles on this blog, please let us know what you think by providing feedback or if you have any particular testing related problems that you would like answers on then please also let us know and we will do our best to respond, either directly or by posting an article on this blog.

Perhaps you need more convincing on the need to test web sites – Why Test Web Sites?