Continuing our series reviewing browser testing tools we are having a look at CrossBrowserTesting, which allows you to check your website live across the major web browsers and operating systems and now also includes several mobile devices.

Previous browser compatibility testing reviews that you might want to have a look at include Mogotest and Adobe Browserlab.

The main difference between CrossBrowserTesting and other browser checking tools like Mogotest or Browserlab is that with CBT you can actually test your website ‘live’ by using the browser instead of receiving screenshots or a report showing you what is wrong.

This means that you are able to pick up issues regarding JavaScript menus or AJAX functionality that you wouldn’t see with a simple screenshot.

Of course, this additional offering doesn’t come cheap. Pricing starts at $19.95 per month for just 150 minutes of use. However, if you consider that the alternative could be to put together your own testing computers with all the operating systems and browsers you require then $20 a month might not be so bad.

Let’s see if CrossBrowserTesting is worth it.

Signing Up

CrossBrowserTesting is a paid service but there is a 7 day free trial with each package, which allows up to 1 hour usage for 1 person.

Unfortunately, in order to sign up, you do have to input your credit card details but if you cancel before the 7 days are up then you won’t get charged.

Once signed up you are dropped into your Test Center and have the option of running a live test, taking screenshots of a URL or installing the bookmarklet (more about that later).

As the main purpose of CrossBrowserTesting is the ability to run live tests, we’ll look at that first of all.

Live Testing

Clicking on ‘Run Live Test’ opens a new window with all the available operating systems and browser combinations plus the screen resolution can be picked from a list.

You can also view what additional software is installed on each operating system if you need applications such as Adobe Acrobat Reader, Flash, Silverlight, Quicktime, etc. This is extremely helpful so you can test the full website including animations, PDF downloads and video clips.

The availability of browsers and operating systems is pretty good with all versions of the main browsers available including Internet Explorer, Google Chrome, Firefox, Opera, Netscape and even older browsers going back to Firefox 1.0, Internet Explorer 5.0 and Netscape 4. You don’t have to test your website in these seldom used browsers but they are there if you need them.

Newer browsers are also present including Internet Explorer 9 RC and Firefox 4 Beta, not yet fully released but very useful to start testing your sites on.

I chose a Windows 7 OS that had Internet Explorer 9 RC installed at a resolution of 1024 x 768 in order to test the WebDepend website on.

The desktop loaded in pretty quickly and I double clicked on the Internet Explorer icon to get started straightaway. Browsing to www.webdepend.co.uk was also fairly quick, there is a slight delay as you type the web address into the address bar and it took a couple of seconds for the WebDepend website to appear but first impressions were that the system was fairly responsive.

There is an additional bar at the top of the screen, which shows the operating system you are using and also allows you to end the testing session, take a screenshot of the page you are viewing and record your sessions to playback later on.

I wanted to test the dropdown navigation on the WebDepend site, as this can’t be tested using screenshot applications, and so I hovered the mouse over each menu item to see how the navigation displayed once it had opened out. I did have to wait a second or two each time for the browser to catch up with where my mouse was pointing.

Using the scrollbar also took a couple of seconds to scroll to the part of the page I wanted to view and so I can see a lot of browser testing starting to become quite tedious in this respect (but browser testing can be fairly tedious anyway I suppose).

Another area that you can’t properly test with screenshot tools are enquiry or contact forms. I wanted to fill in an enquiry form to make sure the validation and submission worked correctly in IE9. The form was completed pretty swiftly and submitted fine, I received the test email within seconds so there was no delay in being able to test filling forms.

Ecommerce websites could make great use of CrossBrowserTesting to fully test the shopping process in each web browser and operating system configuration.

Taking A Snapshot

Whilst testing a website live, simply click on the snapshot icon and it will grab the screen that you are on and save it in your test center. You can then add notes to the snapshot plus what browser you were using at the time for future reference. A publicly accessible link to the screenshot can be shared on social networks.

One useful enhancement to this would be the ability to send snapshots straight into a bug tracking tools such as Unfuddle, Pivotal Tracker or Fogbugz and create a new bug. I would certainly appreciate being able to create new bugs in Pivotal Tracker, which we use at WebDepend, in this manner.

Recording A Video

Once you have recorded a video you can watch it back and add notes plus the browser you were using for future reference, as with snapshots. A publicly accessible link to the video clip can be shared on social networks.

Mobile Testing

Fairly new to CrossBrowserTesting is the fact that you can test how websites appear on an iPad, iPhone or an Android 2.2 device.

Click on ‘Run Live Test’ as with the Live Testing carried out above but choose one of the mobile operating systems to launch.

For this test I chose Android 2.2, which has the Android 2.2 web browser at a resolution of 320 x 480. I waited for around 60 seconds whilst my configuration was being prepared but then an on screen error message appeared to say that it had not successfully launched, the technical team had been informed and to try again.

Once I did try again, after waiting another 60 seconds, a virtualized Android operating system appeared and I could launch the Android web browser.

As described above, there is a slight lag whilst the browser interprets your mouse click or keyboard action but it works pretty well and you can successfully browse the website you want to test, using the Android device including navigation buttons and keyboard.

You will probably want a greater range of mobile browsers and operating systems if you intend on doing a lot of mobile device testing but having iPhone, iPad and Android represented is a good start.

Automated Screenshots

CrossBrowserTesting does also include automated screenshots with every pricing plan. In your test center click on ‘Take Screenshots’ and then input your URL including any advanced options such as whether a login is required or if you want to delay the screenshot by a couple of seconds.

Then select the browsers you need screenshots take for, you can customise the list or selecting all browsers will give you a pretty healthy list.

It is also possible to specify the resolution for each browser, set to 1024 x 768 by default or 640 x 960 for the iPhone 4. At present it doesn’t appear possible to have screenshots taken for an Android 2.2 device.

The final step is to click ‘Take Screenshots’ and CBT will start churning through all the screenshots and presenting them to you as they are taken.

Screenshots are taken pretty quickly with all 41 browsers returning a screenshot after about 3 or 4 minutes. I was running these tests at 2pm UK time, which is 8am where CrossBrowserTesting is based. Hence, all times are recorded as being 8am, which might confuse me when I come to refer to results and figure out when I was carrying out the testing.

Checking through the screenshots I spotted a small layout issue on the WebDepend home page when viewed on an iPad (I don’t own an iPad unfortunately and so hadn’t tested on this device previously) so the test highlighted that bug to me, which was useful.

But be careful as using the screenshots facility adds to the overall minutes used. I managed to rack up 31 minutes of usage in checking through the screenshots produced without realising.

Against each screenshot is a handy link to launch a live test session for that specific browser and OS combination so if you spot a problem you can quickly investigate in more detail plus test the rest of your website to find any further problems.

Bookmarklet

There is also a bookmarklet that allows you to take screenshots or launch a live testing session on any web page that you are visiting. This allows quick and easy access to starting your browser compatibility testing.

API

One of the areas not utilised as part of this review is the API, which allows you to automatically trigger screenshots of a URL and then email them to an email address to be checked. I suppose it could also enable you to send screenshots into a bug tracking system as described above.

At present, I believe only the screenshots functionality is included in the API but there are plans to extend that to the live testing side of the system too.

Conclusion

Overall, CrossBrowserTesting is a very useful service for any browsers or operating systems you don’t normally have access to. For me this would be several browsers including IE6, older versions of Firefox, Safari, Netscape, Opera and mobile browsers including the iPad, iPhone and Android 2.2. That’s quite a number and whilst I might not test everything on all the older browsers, I do want to make sure everything displays and works on mobile browsers plus some clients still require IE6 compatibility.

Another advantage I can see is for website developers and apps companies to react to users problems and try to reproduce the issue on the specific OS and browser combination that user has installed. For the smaller organisations they may not have all these combinations to hand and so a service like CrossBrowserTesting becomes essential.

I can also see scenarios where I am out of the office and away from my normal testing environment. Being able to test on up to 41 web browsers from anywhere I can get online is a huge bonus.

CrossBrowserTesting is very easy to use and get to grips with straightaway. There are little details that make browser testing easier, such as the ability to go into a live test from a particular screenshot you had taken. Having the bookmarklet and API add to the overall sense that the application is easy to use however you want to use it.

On the downside there are small delays in using the live testing so over a full session that means browser testing takes longer than it would if you were using the machine locally.

The main drawback is that minutes soon rack up, whether you are using Live Testing or Screenshots so the base package of just 150 minutes per month would get used up very quickly. I wonder whether the ability to purchase extra minutes or a pay as you go service would be something for the folks at CrossBrowserTesting to consider.

However, even with the slight negative points, using CrossBrowserTesting has to be less hassle, cheaper and far more convenient than getting together all the operating systems and browser combinations available plus the maintenance and space required to house all the equipment.

I certainly know what I prefer.

Adobe Browserlab takes screenshots of your web page in several different web browsers

Adobe Browserlab is a browser testing service that allows you to check how each page appears in a variety of mainstream web browsers.

The web-based application allows you to compare your web page in 2 web browsers side by side and in that respect, shares a similar principle with Mogotest, which I reviewed recently. Read the Mogotest review.

One benefit to Adobe Browserlab is that it is free for 1 year if you sign up before 30th April 2011.

Sign up

First of all, you have to sign up to Adobe CS Live in order to gain access to Browserlab. By signing up to CS Live before 30th April 2011 you get 1 year complimentary access but we are not sure what the costs will be once the first year is finished.

CS Live includes other products including CS Review, Acrobat.com, Adobe Story and SiteCatalyst NetAverages.

Features

Once you have signed up to Browserlab, you can check your web page by inputing the address and selecting the browser you would like to view it in.

Adobe Browserlab then fetches the screenshot in that browser and presents it to you.

There are several different views available to you, as follows:

1-Up View – the default view, this allows you to see the screenshot of the web page in a single web browser.

2-Up View – the view I use the most, you can compare screenshots from two different browsers side by side to see where your web page may differ in one browser over another.

Onion Skin View – an interesting feature and places the screenshot of one browser over the other in a semi-transparent view, allowing you to see more easily the differences between each browser.

Additionally, there are the following features:

Smart Align – allows you to select a specific point on your screenshot and then determine which of the other web browsers display the screenshot at exactly the same alignment. This helps to pick up graphics or other elements that may be slightly out of line from browser to browser that you cannot easily see.

Delay – sets a delay for any animations that come in or start after a certain number of seconds.

Integration – Browserlab integrates with CS5 but I am not sure how, as I don’t own a copy and so have not covered that particular aspect in this review.

Screenshots rather than full testing

Adobe Browserlab only takes screenshots of your web page to display in each browser. This means that you can only realistically check for layout issues rather than being able to actually browse the website with that web browser. It also means that you will not be able to pick up JavaScript issues or items that may occur as you navigate around the site.

For full testing you need to use something such as CrossBrowserTesting.com or Spoon.net/browsers.

Browser Sets

This allows you to set up groups of browsers that you would like to test your web page in. You could create browser sets to test in all the IE browsers for instance or a list of browsers that you support as standard or any browsers that you don’t have readily available.

Supported Browsers

Adobe Browserlab supports all of the main web browsers including the following:

Windows:

• Internet Explorer 6
• Internet Explorer 7
• Internet Explorer 8
• Internet Explorer 9 Beta
• Firefox 2
• Firefox 3
• Firefox 3.6
• Firefox 4 Beta
• Chrome 7
• Chrome 8

Mac:

• Firefox 3
• Firefox 3.6
• Safari 3
• Safari 4
• Safari 5

The range of browsers supported is better than Mogotest but you do have to check each one manually for issues rather than being given a report in the case of using Mogotest.

Conclusion

Adobe Browserlab is a very useful browser testing tool. You gain access to browsers and browser versions you may not have easily available to you, in my case I don’t have Internet Explorer 6 or older versions of Firefox or Safari installed on any of my machines.

Adobe have created a tool that is pretty easy to use and if you have CS5 then Browserlab is probably even easier to use. The 2-up and onion skin views allow you to easily compare one browser with another but I can’t see myself using the smart align feature that much to be honest.

Browserlab is a good free (for now) service but we’ll see what happens if and when Adobe starts charging for it.

Sometimes however the issues can be much more important – whole pages can be rendered incorrectly by certain browsers and, if not checked, it may mean that users visiting your website with that browser cannot use the site.

Most designers, developers and project managers manually check the website in each of the main web browsers, which is time consuming and requires detailed and concentrated testing of each page or page type within the website.

Introducing Mogotest

Mogotest aims to make browser compatibility testing easier

However, all is not lost, a new web app aims to make the process much easier. Its called Mogotest and takes the strain out of browser testing by comparing how your website displays in each browser and giving you the results.

Free Test

To see Mogotest in action you can run a free page test from their home page, which tests a single page and shows you the results once completed so you get feedback straightaway. This is a great introduction to the Mogotest service and gives you instant access to many of the features of a paid plan. Mogotest includes up to 3 page tests for free.

Pricing

Starts at $15 per month for a personal plan, which gives you a single user account, up to 50 tests each month across a maximum of 3 sites. You don’t get certain features such as being able to test authenticated pages or being able to schedule tests. These features come along in the freelancer plan, which is $45 per month. Pricing goes up much more steeply after that with a Team plan costing $125 per month for 10 sites and the Agency plan is $595 for 50 sites.

Running A Test

To run a test you follow a 4 step process:

Configure Site – just input your website address and tick if you want to see the advanced options.

Mogotest is easy to get started

Create Test Group – here is where you add paths for each URL that you want to test specifically. If you have lots of URLs to test then this could be a bit more time consuming but nothing like how long it would take to test each URL manually.

It is also possible to schedule tests to run every day, every 2 days, etc., which is a great feature for sites that are undergoing regular updates. Within this step you can also set whether the test should wait for any Ajax to load in before starting.

For my test I added 3 URLs, all for the WebDepend website including the home page, one of the service pages within the Website Testing section and the Blog main page.

Once you’re happy you save your test group and proceed to the next step.

Configure Authentication – you can set whether your site requires authentication, useful for being able to test password protected sites that are still in development or for testing pages within a members section.

Prepare Test Run – final step shows all the items you have configured and allows you to go back to edit settings before clicking on Run Test!

Test Results

Whilst the test is running you can view a Test In Progress page that shows how your test is coming along and the estimated time to complete (900 seconds) although my test took far less than that.

Once the test has run your screen shows 3 tabs:

Easily see which browsers your test failed in

Browser Compatibility Report – from the screenshot you can see that the WebDepend site’s blog page and home page did not pass for Firefox 3.5 on Linux (not a combination I regularly test for it must be said) but that all other tests passed successfully.

Clicking on the items that didn’t pass will display the browser rendering results where it will show the browser you have clicked on side by side with your reference browsers (mine is Firefox 3.5 on Windows) so you can easily pick out the differences.

Any issues will be displayed underneath the side by side view, in this case 3 elements for my Blog main page were incorrectly positioned in Firefox 3.5 on Linux when compared with Firefox 3.5 on Windows.

As it turned out, the issues that Mogotest had found were very minor problems but you can see how useful it is being able to find these problems by running a test rather than manually trying to pick out browser related problems.

Site Problems – shows any HTML or CSS validation errors, broken links, server errors, empty content warnings or redirects. Mogotest found 5 validation errors and warnings that were not picked up when I manually ran each page through W3C’s validator a couple of weeks ago.

Validation Reports – displays a list of the URLs you tested and shows which of those have problems. Clicking on a problem pages takes you to a view of the HTML for that page with row numbers against each line of code (very helpful) plus a list of the issues found underneath. Rolling your mouse over an offending line with also display the issues, which is very helpful when investigating the problems that need attention.

Once you have fixed any issues found, you can go back into Mogotest and click Retest in order to carry out the same tests again, which will hopefully now pass for each browser.

Conclusion

Overall, Mogotest is a really useful service. By automating a lot of the browser checking process it really does make life easier and will pick up lots of layout problems between browsers that could go unnoticed or take time to find.

However, some manual testing is still needed in my view, it is important to browse and use the website in each browser to notice issues that Mogotest is not yet capable of, such as Ajax related content or JavaScript menus.

The browser support could be extended, as Mogotest is running tests in Chrome 5 whereas we’re up to Chrome 9 now and there is currently no support for Mac browsers at all.

If you run tests regularly, across several sites, you could find yourself running out of available tests pretty quickly and needing to upgrade. Also, there doesn’t appear to be anywhere that I can see how many tests I’ve got left or if I could purchase a block of tests without having to upgrade to the next plan up.

When running tests, I was lucky in that there were a small number of issues found. I wonder for projects in early stages of development what happens when there are large amounts of problems to wade through and how easy it is to see the most important ones that need fixing first.

A good addition to any website tester’s, web developer’s or project manager’s list of testing tools to make life easier, Mogotest does do what it sets out to and can only get better from here.

Over on WebDepend, we’ve posted an article asking is your website ready for Christmas? Now is the time to review your site in detail and tackle any issues so that you can be confident your users will not hit any problems when trying to make a purchase.

We’re not just talking about site performance issues here although that is part of what a thorough website audit would normally include. Functional problems and browser related bugs can creep into your website left over from updates or other development work carried out on your site. Even SEO work can sometimes cause issues that need to be identified and resolved.

A website audit is essentially a manual undertaking although there are some testing tools that can help and point the way to further investigation required.

Tools For Functional Testing

For an automated scanning tool, I’ve found SortSite to offer good all round results and pick up issues across a range of different tests (or checkpoints as SortSite calls them) including broken links, standards compliance, browser compatibility, accessibility, usability and search engine issues.

SortSite can be a good starting point to get a feel for the overall quality of your website but I would always go through at least the main pages of the website manually to determine what issues were apparent and their relative importance.

SortSite also provide a free online version of their tool that allows you to scan up to 10 pages of your website.

Browser Compatibility Checking

There are a number of browser testing tools that allow you to check your website in a number of different web browsers.

A tool that aims to take a lot of the hassle out of browser testing is Mogotest, which checks that pages are consistent across a range of web browsers. You can also schedule tests to run automatically and get notifications of problems by email. Look out for our review of this tool coming soon.

Site Performance Tools

Heading into the main Christmas selling period you need to ensure that your website is performing well and can cope with the extra traffic.

A free web tool that I’ve found very useful in quickly reviewing the performance of a website or web page is WebPagetest, which checks to see how long each element of the web page takes to download and makes a series of suggestions as to how performance can be improved.

For load testing, to see how much traffic your website can cope with, there are a couple of affordable options including Load Impact, Loadstorm and Browsermob.

Web Standards

To validate your website against a number of accepted web standards, try one of the Validators and Accessibility Testing Tools we have listed in our Testing Tools Directory.

The most popular are W3C’s free HTML/XHTML validation service, their CSS validation service and for accessibility checking I often use Truwex Online.

SEO Tools

There are many SEO tools to audit your website’s SEO friendliness and performance, generally the best web tools are produced by SEOmoz. Simple crawl tests and HTTP tests can highlight problems with search engines indexing your content and confirm whether your HTTP status codes are correct.

Google Webmaster Tools is also a source of good information allowing you to find out how Google sees your site plus what search queries you are being found for and how many people are clicking on your site.

Prioritise Issues Found

When you find an issue or bug that needs to be dealt with make sure you use a bug tracking or issue tracking tool so you can not only keep track of everything you find but also prioritise those issues to be fixed first.

There is only a short amount of time to tackle issues before the peak Christmas sales period and so critical and important items should be resolved as a priority.

One such bug tracking tool I like is PivotalTracker, which is free and I find great for keeping up to date with bugs and issues across a number of projects. It is easy for a team to use and shows a prioritised backlog of issues for developers to work through.

Of course, even once you have used those testing tools you need to ensure you spend time reviewing the website manually and you will be surprised at what you can pick up.

If you would like someone to carry out a website audit on your behalf then look no further than WebDepend.

Is your website ready for Christmas? Let us know what challenges you faced in the comments.

I thought I would provide a short update from my website testing consultancy called WebDepend, as there are a couple of items to mention that may be of interest to readers of this blog.

Meanwhile, on Testing Web Sites, I’ll be looking at more testing tools and software reviews coming up over the next few weeks.

From The WebDepend Blog

The WebDepend blog is where I focus on issues and developments that affect agencies and freelancers regarding website testing, project management and SEO.

Recently I’ve been focusing more on website testing and putting forward that maintaining a level of website quality is hard to do whilst also recognising that pressure to launch websites is causing more bugs to be found on live websites, as insufficient testing is carried out.

As well as pointing out some of the hardships of website testing I provided some answers and couldn’t resist plugging my services such as a website audit.

Announcing Free Website Testing

Update – we have temporarily discontinued this free testing, whilst we concentrate on a backlog of work. A revamped free testing service will resume shortly.

Another opportunity to help agencies and freelancers is my free website testing trial, where you can submit a website to me and I will spend 30 minutes reviewing it, for free.

In return, you’ll get a list of issues that will encompass website quality issues, browser compatibility bugs and the odd usability problem.

So far, these free reviews have found issues such as a website’s main enquiry form submitting details to an invalid email address, the main navigation of a website not working at all in Safari, a login form not displaying username and password fields and many layout problems, broken links and image and of course typos.

It just goes to show that some meaningful testing can be completed in as little as 30 minutes or less.

WebDepend

WebDepend is a company that I run, which focuses on website testing and web project management. We help agencies and freelancers test the websites they produce and manage their web projects. We also work with direct clients too. For more information please visit www.webdepend.co.uk.

Are you testing your websites with Google Chrome?

According to statistics from W3Schools.com, usage of Google Chrome has now risen to 16.7%.

That figure is in many ways not surprising because Google Chrome is an excellent web browser, is fast and does not crash as frequently as other browsers. What is surprising is that, whilst Chrome is still a long way behind Firefox at 46.4%, it is now ahead of IE8 (15.6%) and a long way ahead of IE7 (7.6%) and thankfully IE6 (7.2%), which is finally fading away.

Google Chrome’s share of the market continues to increase whilst all the major web browsers, except Opera, which picked up from 2.1% to 2.3% over the last month, are on a steady decline, including Firefox and Safari.

Now I’m not suggesting that this trend will continue, browser usage does fluctuate as new versions come out and entirely new browsers are launched.

What I am getting at in terms of Google Chrome’s greater market share, up from 10.8.% in January to 16.7% in July, is to make sure you are consistently testing your sites in Chrome.

Chrome Ahead of IE8

As I pointed out in a recent Tweet (please follow TestingWebSites by the way), the number of people using Google Chrome is now above IE8 and so you should be regularly testing in Chrome to pick up on any layout inconsistencies.

And there are quite a few that I have spotted when testing newly launched websites recently.

My belief is that the surge in Chrome usage over the last few months had caught a few web designers and developers out who traditionally test in IE6, IE7, IE8 and Firefox. I know from recent work with several digital agencies plus graphic design agencies and freelancers that there are still many who do not always test in Google Chrome.

Test With Google Chrome

If you don’t have Google Chrome installed already then you can get it from Google.com/chrome, available for PC, Mac or Linux. You can install it alongside Firefox and IE8 on a PC so can have all 3 accessible at once.

Google Chrome uses the WebKit rendering engine, which is also used by Safari and a number of lesser used browsers. It is possible that by testing on Google Chrome that you may find issues also associated with the Safari browser.

If for some reason you don’t want to install Google Chrome then using a browser checking tool such as Adobe Browserlab or CrossBrowserTesting.com will help you pick up on any issues.

I recently covered 3 browser checking tools or you can view all 9 browser testing tools in the Testing Tools Directory, around half of which support Google Chrome although none yet support Chrome on a Mac as far as I can tell.

Do you currently include Google Chrome in your browser testing?

Browser checking tools allow you to test your website in a variety of different web browsers easily

Browser checking tools are important and can be a life saver, as they allow you to test or check your website in a variety of different web browsers.

It is not possible for many web designers and developers to have physical installations of each browser type and version on each operating system out there. I do know of digital agencies that have a number of testing PCs dotted around the development studio for quick testing of work or perhaps a testing server with 8 or 10 different browser versions and operating systems installed.

But for many of us, especially freelancers, having this number of browsers and operating systems at our finger tips is not possible unless we fork out money for the hardware ourselves and install the required browsers.

For instance, I have 3 computers, running Windows 7, Windows Vista and Mac OSX 10.5. On those computers I can test using the following browsers – IE7, IE8, Chrome, Firefox 3.6 (PC), Opera 10, Safari 3 (Mac) and Firefox 3 (Mac).

So that is a decent number of browsers, but there are times when I need to test on IE6 or a different version of Firefox, Safari or Chrome that I don’t have running on any of my machines. Plus, when new browsers are released, I have to decide whether I can accommodate it within my existing setup or whether I need to do things differently.

Thankfully, the last couple of years have seen a number of good browser checking software solutions launched, which allow me to check the website over in pretty much whichever browser I want installed on any of the main operating systems.

Here are 3 of the most popular browser checking tools taken from the relevant category of our Testing Tools Directory, which contains 9 browser testing tools in total.

CrossBrowserTesting.com

CrossBrowserTesting.com allows live testing and automated screenshots

Allows live testing on an OS and Browser of your choice plus the ability to take screenshots across a range of browsers.

Features:

• Offers live testing – for in depth testing, using the browser you need remotely
• Automated screenshots – to identify display issues
• From 150 minutes per month
• 1 week free trial
• Pricing starts at $20 per month

When you sign up you get access to the live testing and automated screenshot features, whilst all packages come with a 1 week free trial. Packages are based on a usage allowance with $20 per month getting you 150 minutes.

Browsrcamp

Browsrcamp focuses on Mac browsers

Focusing on web browsers running on a Mac operating system, such as Safari, Firefox and Flock, which allows those who don’t use a Mac to test the compatibility of their websites on Mac browsers.

Features:

• Gain access to 12 browsers on a remote machine including Safari, Firefox, Opera and Internet Explorer plus other lesser known browsers.

With prices starting at $3 for a 2 day subscription, Browsrcamp represents good value for money for website testing on a wide range of Mac browsers.

Adobe Browserlab

Adobe Browserlab is free for a year if you sign up to CS Live before the end of April 2011

Now a part of Adobe’s CS Live, Browserlab integrates with CS5 allowing you to access it from Dreamweaver and starting testing straightaway.

Features:

• Zoom – allows you to zoom in to see more detail and spot bugs more easily
• Rulers and Guides – display rulers along the x and y axes and display a guideline at the point of the screenshot
• Screenshot Delay – you can take a screenshot several seconds after the page loads allowing time for flash animations to start to play and show content
• Customisable settings – configure up to 10 browsers sets for different projects
• Multiple viewing modes – compare 2 browsers side by side or overlay one over the top of another with the Onion Skin view
• Covers all leading browsers

If you sign up by end of April 2011 you get Adobe Browserlab free for a year as part of CS Live, which also includes CS Review, Acrobat.com and a couple of other online services.

Conclusion

I think each of the 3 browser testing services has its merits. Browserlab from Adobe has some good features, getting free use for a year is also a big bonus and, even if you don’t use their Creative Suite then Browserlab is still a good standalone testing suite.

CrossBrowserTesting.com was pretty much the first service to move beyond just taking screenshots and allow you to access the browser itself to conduct real testing. They have the widest range of operating system and browser combinations and so are worth a look from that point of view.

I would use Browsrcamp for any specific Mac browser that you can’t get on either Browserlab or CrossBrowserTesting.com or if you just need access for a short period of time, as $3 is peanuts for 2 days access.

This ad hoc approach to testing means that it is possible to miss large sections of functionality and not be able to fully consider all permutations or variations of events and actions that need to be tested.

For most websites, there is a fairly long list of items that need to be tested as part of any new website development and in order to know what to test, when you tested it and whether those tests passed or failed, it is best to make a proper plan.

We are going to show you how to write a simple and straightforward test plan, which will provide a more systematic method for testing your website but the process outlined below is still essentially a manual one. We are not yet moving into using software tools to help identify and run tests, nor are we covering automated testing here, although I hope to write a post on these aspects in due course.

Writing A Test Plan

A typical test plan should identify all the areas of the website that you need to test before it launches, which you then follow through to conduct your testing, making notes of when you completed the test and adding any bug or issue reports to your bug tracking software along the way.

Start off with getting all the main sections down in a spreadsheet, divided between front end and admin or CMS if the website has one.

Then, start to break each main section down into the constituent parts that you need to test, concentrate on the important parts of the website first such as the home page, which is likely to be one of the most visited pages and the main areas of functionality, which are most likely to be the most used.

For instance, in the case of an ecommerce site, this would include individual tests for items such as adding products to the shopping basket, removing products from the shopping basket, updating the shopping basket quantities, being able to enter the checkout process and completing the checkout process (depending on your website there may be several steps to test within the checkout process).

You should be asking yourself questions for when you start testing the checkout process from the point of view of the user such as:

• Can I complete my order?
• Can I go back and amend my order?
• Do I understand where I am in the checkout process?
• Can I clearly see what I am ordering and how much it will cost, including delivery and taxes?
• Do I get a receipt or email confirmation?

As the owner of the website there will be different questions such as:

• Does the payment processing work correctly?
• What happens if the incorrect card details are entered?
• Does the order confirmation process work correctly?

And so on, when putting a testing plan together you are testing the website from the point of view of the user and also from the point of view of the owner or operator of the website. You may also have specific usability testing lined up but the primary reason for the test plan as far as this blog post is concerned is to test the functional aspects of the website to ensure they work correctly. Does the website do what it says it does? For instance, if the checkout process displays the order details including taxes and delivery then does it show the correct details for both tax and delivery?

Another important area to test is the site search or product search, as this is likely to be used a lot on the website.

Make sure that you test all the forms within the website, such as enquiry forms and newsletter signup forms. Test that the form can only be submitted after the mandatory fields have been completed and that the form is emailed to the correct email address or saved to the database correctly.

Once you go through and think about each section of the website along the lines of the above then you should have compiled a decent list in your spreadsheet of areas and items to test. Add columns for each browser that you need to test each item on, which will be the web browsers that you have agreed to support as part of the project. This would normally cover the major web browsers such as IE6, IE7, IE8, Firefox 3, Chrome and Safari but may include others too depending on the usage of your target market (or existing audience if you have those statistics). There are a range of browser checking tools that can help you with this aspect.

For the admin system or CMS that powers the website, add a row to your test plan for each function. Going back to our ecommerce example, this means that there would be tests for add new product, edit existing product, remove product, view order, edit tax rate, etc. depending on what the system was capable of.

You could break these tests down further to test individual elements of adding a new product. For instance, is there validation present to check that mandatory fields are filled in when adding a product? Does the product image have to be a certain size or dimensions? Does the product title have to be shorter than a certain number of characters? All these items could occupy separate rows in your test plan.

Of course, if you were to break your test plan down to this extent for a large website or CMS then the result will be a long spreadsheet of items to test, which will take a great deal of time to test effectively. You will need to decide whether you have the time available to test in this level of detail or if some areas can be tested more quickly.

A few other items that you may wish to make sure you have as part of every test plan are the following, in no particular order:

• Ensure your HTML and CSS code validates to W3C guidelines
• Test accessibility of the website
• Test the SEO items that you have planned for to make sure they are in place
• Test against the specification for the project that was signed off to ensure that everything has been completed that was agreed
• Test against designs and/or wireframes or prototype html mockups that were completed to make sure the finished website matches the signed off designs or wireframes
• Take into account any changes requested (should be in the form of change control documents)
• Complete any security tests that are required

You may wish to have a look at our list of 10 different forms of website testing to make sure you have covered everything. The list can go on indefinitely, there is always something that can be tested a bit more, but at some point it has to be halted at a relevant point otherwise the testing will never get done.

List or Flow Chart?

Some aspects of the website lend themselves to having a flow chart in place that describes all the permutations or possibilities that need to be tested. This is especially helpful if one action then presents another set of possibilities. When testing without a flow chart it can become extremely difficult to remember which permutations you have tested and which you have not.

Having a flow chart or mindmap worked out that shows you all the possibilities allows you to think more clearly about what you are testing, what you expect the outcome to be and understand what you have tested and what is remaining to test.

Conclusion

If there is too much testing to do for one person then divide it up amongst other people with instructions on what you want testing and what to do with bugs when they are found. Make sure they follow their bit of the test plan and feed back on what they find, handing bugs to you or inputting them into your bug tracking software themselves.

As you test then fill in the test plan to show what you tested and when. If you keep the document client friendly then it can be shown as proof of what testing was completed prior to launch.

Testing is a vital part of any website development and having a decent plan before you start testing saves a great deal of time and effort whilst making sure that all the required testing gets done.

One of the first posts I wrote, which described a typical Web Testing process, mentioned that a bug tracking system ‘can be hugely beneficial, as you can set priorities for each bug and then understand how many bugs you have at each priority level. It gives you a much better understanding of how many bugs you have to fix before you can launch and calculate how long you expect that process to take.’

There are a good range of bug tracking applications out there, a lot of them web-based, which are vital when tracking bugs and associated information about each issue. Here we take a look at 4 of the main players in bug tracking software, which are also available in our Testing Tools Directory.

Why Use Bug Tracking Software?

Once you start testing a website you can come up with all sorts of bugs and issues that need addressing. I’m talking a bit more than the odd typo or broken image. Perhaps you find problems with the site search returning strange results or the checkout process not working correctly or there are specific issues in different browsers that work fine in other browsers. There are several different reasons to use an application that tracks bugs and feature requests.

Complicated Bugs

As bugs start to get a little more complex it is important to hand the developer all of the required details about each bug so that he or she can fix it in a reasonable amount of time, without chasing around trying to reproduce it or get a screenshot or URL showing where that bug was found. Bug tracking software gives you a structured format to report bugs so that you include the URL where you found the bug, provide a full description and a screenshot and what browser or operating system you were using. These details will shorten the amount of time that a developer needs in order to squash the bug and improve their productivity. Get more bugs squashed in less time.

Lots of Bugs

Alongside the complicated bugs will still be the relatively quick and simple items to fix such as the odd layout problem, typo, broken image or link, etc. These bugs, whilst they do need to be fixed, are sometimes not as important as other bugs and, when presented in a list or on a spreadsheet, can often be completed ahead of more important items because the developer naturally follows the list through from start to finish. Bug tracking software allows you to prioritise bugs so that developers work on the most critical issues first and leave the less important bugs until later. Rearranging priorities in a list or on a spreadsheet gets too cumbersome when dealing with a large number of bugs but issue tracking software handles this aspect with ease.

Improved Workflow

With bug tracking software you can assign bugs to different developers so everyone knows what they are working on. Once those bugs are completed then the developer assigns them back to you for checking or verifying. If you are happy you can close the bug, if not then reopen it and give it back to the developer. Do this for each bug and you can understand exactly where each bug is in the process, how many are assigned, how many are to be verified, etc. It allows you to have a much better handle on exactly where your web project is at instead of continually poring over your list or spreadsheet and trying to make an educated guess.

This improved workflow eases pressure on you and your developers and improves productivity because developers can just get on with fixing bugs or working on feature requests. It also means that you are not constantly updating lists and spreadsheets with the latest status of all the bugs reported.

Bug Archives

Once bugs are fixed, tested and closed they can be referred to at any time so that you can see when the bug was reported, when it was fixed, who fixed it and what the developer did to fix it (assuming they fill their comments in correctly). This provides an extremely useful archive if you ever need to refer to a particular bug again instead of trying to go back through emails or out of date spreadsheets to find details on that bug or feature request.

Some bug tracking systems allow time tracking so you can see how much time was spent on fixing a bug. Using bug tracking software ensures that developers become more accountable because their comments and times are recorded and can easily be referred to.

Bug Tracking Software

Now that we’ve established what bug tracking software offers and why web project managers and testers should incorporate it as part of most web projects (definitely on all large web projects and probably on most small to medium projects too) let’s have a look at some of the main bug tracking applications currently being used.

My specific experience is with using Bugzilla so let’s start with that one.

Bugzilla

Bugzilla - classic open source bug tracking application

Bugzilla was originally released by Netscape in 1998 and so has been around for a long while now. This open source software is one of the first bug tracking applications that spring to mind and, although it takes some setting up and the interface is a bit tricky to use, it has everything you need to record and track bugs effectively.

It is possible to sidestep the web interface by using a desktop client that synchronizes with the central bugzilla software. My specific experience with bugzilla was using the Deskzilla client on a Mac, which worked well.

As Bugzilla is open source then it is free to use although it needs to be installed and set up on a server in order to be ready to use.

Unfuddle

Unfuddle - web-based bug tracking software

Unfuddle is a hosted bug tracking application that also includes Git hosting and Subversion hosting allowing you to create your own secure code repositories. There are also some project management tools included such as tracking your project against specific milestones and time tracking is available in the more expensive corporate and enterprise packages.

Bugs, issues and feature requests are raised as tickets in the Unfuddle system and each ticket includes workflow management and an audit trail so you can keep track of who made changes to a ticket and when these changes were made.

If you need the code repositories and project tracking functionality along with a straightforward bug tracking system then Unfuddle could be the answer. There is a free package available that includes 1 project for 2 users. After that prices start at $9 per month.

Trac

Trac - open source issue tracking software with project management tools and a wiki

Trac is also open source software that combines an issue tracking system with a wiki and project management features such as milestones and being able to document the roadmap for a web project or software development. Trac can also serve as a web interface for version control systems giving you a usable front-end for Subversion or Git.

Tracking progress of individual bugs, issues or feature requests is straightforward with each one having its own numbered ticket with the ability to filter tickets by severity, project component, version or owner. Keeping on top of progress is also easy with the ability to set milestones and being able to see how far you are from reaching each milestone.

Trac has a good set of features and is a lightweight and flexible system that should be very useful for most web development projects. It runs on most operating systems but does require some setting up and this is perhaps its biggest downside that I can see if you want to start tracking bugs straightaway.

FogBugz

FogBugz - web-based or installed bug tracking software with large feature set

FogBugz is available as a hosted application, called FogBugz on demand, or as installed software on either Windows, Mac or Unix operating systems.

There are several parts to FogBugz, the issue tracking section makes it very easy to report bugs and you can even create bugs via email or create bugs quickly without having to fill in everything so you can get the bug into the system. Whilst I think this is a good feature you do need to make sure you go back and complete all the details before you assign the bug to a developer to work on. Everything is saved in a complete case history for future reference.

But issue tracking is not all that Fogbugz does, there is a project planning section where you can outline tasks, break larger tasks into subtasks, enter estimates and set milestones. Fogbugz also has something called Evidence-Based Scheduling, which looks at what tasks are being estimated and how long they actually took to complete and then plot a probability curve to work out whether a milestone can be hit or not. The FogBugz Wiki allows you to keep all project documentation in one place and there are also a whole load of plugins available to extend the functionality plus FogBugz can integrate with another piece of software from the same company (Fog Creek Software) called Kiln, which is a version control system.

FogBugz is the most expensive of the bug tracking software applications featured here with prices starting at $25 per user per month for FogBugz on demand or $999 for 5 users for the installed version of the software.

Conclusion

Whichever of the above applications you use, as long as you use one of them and get your workflow organised in terms of how you report, record and assign bugs then you will find that it becomes a breeze to keep track of exactly where you are at in your website testing. You won’t miss any issues (as long as they are reported and entered into the bug tracker), you can prioritise showstoppers or critical bugs easily to fix those first and archive completed bugs for future reference.

If I had to pick one of the applications above then I would go for either Unfuddle or FogBugz (on demand) because I like web-based applications as there is nothing really to set up or install. Both of these have a trial although both cost money whereas Bugzilla and Trac are free but require time installing and setting them up on a server.

FogBugz has the better overall feature set and I am intrigued as to how the Evidence-Based Scheduling works, as I have not come across that before. But it terms of a cheap, easy to use, bug tracking system that is ready to go and will do the job, I would pick Unfuddle with the caveat of going for FogBugz if I had a bigger team (and didn’t mind paying more money).

At the moment that testing subject of choice is known as Conversion Rate Testing, or sometimes called CRO (Conversion Rate Optimisation). This would normally mean that the tests being carried out are Split Tests or Multivariate Tests, using software such as Google Weboptimizer.

Don’t get me wrong, conversion rate testing is really important and can greatly increase the number of conversions on a particular page, which can raise sales, enquiries, profits or whatever aspect of conversion you are trying to improve.

But there are other forms of testing websites, quite a few in fact, and some that are quite basic that get overlooked. I wanted to give a brief description for each type of testing to help you understand all the areas that can (and should) be tested when launching a new website, rolling out a set of updates or at least checking on a regular basis.

Different Forms of Testing

Functional testing

This form of testing aims to ensure that the website works correctly or as it was intended to. You click on a link, does the website display the correct page? You complete a search, does the website return the correct search results? You add a product to the basket, does the website add the correct product to the basket? Functional testing is going through the entire website, including content management system or admin area, to make sure that each function within the website is performing as it should be.

Browser compatibility testing

Different web browsers display web pages differently from each other. This is one of the least helpful aspects of the Internet and browser compatibility testing or browser checking involves testing the website on each of the major browsers (or each browser that you have decided to support), which generally includes IE6, IE7, IE8, Firefox 3, Chrome, Safari 3 and Safari 4 on both PCs and Macs. You could possibly include Opera 9 and Opera 10 in that list in addition to testing your website on Linux as well as Windows and OSX. You could also check your analytics to see which browsers are most commonly visiting your site. There are several decent browser checking tools listed in our testing tools directory.

Usability testing

Usability testing assumes that your website works correctly and displays in each major browser correctly. However, a website that functions correctly is not necessarily a usable website for your target audience. Usability testing enables you to show your website to a group of users, ask them to perform certain actions and see how they accomplish those tasks. There are many reasonably priced usability testing tools on the market that have made this type of testing much more straightforward, many of which are listed in our testing tools directory. It is possible to gain great insight into how people actually use your website and a lot of the experts agree that you can pick up most usability issues when testing with only 5 users.

Accessibility testing

Accessibility testing is extremely important to make sure that your website complies with best practice and accessibility guidelines plus legislation surrounding accessible websites, which is in force in both the US and UK. In the US there is Section 508, which is part of the Rehabilitation Act and in the UK we have the Disability Discrimination Act or DDA. A fully accessible website not only benefits those users that would otherwise have difficulty using your website but search engines will also be able to navigate and understand your content more easily, thereby helping your search engine rankings. To help you test your website’s accessibility there are some accessibility testing tools available in our directory although manual testing is still recommended.

Performance testing

Performance testing is an area that has always been important but possibly even more so now that Google is starting to incorporate the performance of a website into its ranking algorithms. There used to be an 8 second rule, which believed that a user will wait no longer than 8 seconds for a page to download before they click back and try another website. That rule is a long way out of date, it is now no more than 1 or 2 seconds. Performance testing can show you when your website is slowing down, which pages are taking time to load and what element of the page is causing the problem. Performance testing tools are covered in our testing tools directory.

Stress or load testing

There is a difference between stress and load testing in that stress testing concerns itself with trying to break the system you are testing whilst load testing aims to find out how much volume you can put through the system. Both of these forms of testing are generally for larger websites and web applications. A range of stress and load testing tools can be found in our testing tools directory.

Site monitoring

A really simple test that you can put in place is to automatically monitor your website to check for downtime or outages. Nobody likes a website to be down including users and search engines. Regular downtime will lower confidence in your website and they could be affecting your traffic levels and therefore sales or enquiries without you knowing. A selection of site monitoring tools can be found in our testing tools directory.

Conversion rate testing

The current favourite form of testing, and an important one too. Conversion rate testing or optimisation can greatly improve the amount of visitors your website converts into enquiries and/or sales. If your SEO brings in 100 visitors a day and your site currently converts 5 of those into becoming customers then, with successful conversion rate testing, you could convert more visitors into customers and make more sales. Conversion rate testing normally takes the form of running split or multivariate tests on your website, many of which are listed in our testing tools directory.

Security testing

Websites can have security holes that could be extremely damaging for your business if they were to be found and exploited. Security testing aims to find where those security holes exist so that you can do something about them.

Spelling and Grammar checking

It is always useful to fully read through all text on the website to ensure there are no typos or grammar mistakes. You will also spot how you can improve the readability of text and other areas that perhaps need some attention. Also make sure you read through all messages that appear when signing up for a newsletter, adding products to a basket, or registering including any error messages. Simple changes to the wording of an error message into a nicely worded error message can make a big difference.

So there you have it, website testing can take on many forms and all of them should be completed to a greater or lesser degree. At least by completing each type of testing you can be aware of any issues or areas for improvement so then properly plan out what action you are going to take and when. Good luck with whatever testing you complete.