Web-based usability testing tools are starting to become more prevalent these days with the likes of Clixpy, Loop11 and UserTesting all appearing recently.
We shall have a look at Loop11 first of all, which has been in private beta for around 4 months now but is soon to be released to the general public.
According to their web site – Loop11 is a web-based user-experience testing tool, allowing companies to conduct online, unmoderated user testing on any kind of digital interface. Loop11 is not a survey or web analytics tool, but a user experience tool… helping you to understand user behaviour.
Sounds good so far, so how do you get started?
First of all, once you log in, click on the big ‘Create New User Test’ button in bright orange so you can’t miss it. You will be presented with a form with the following fields:
Public Title – this will be seen by your test participants
Working Title – used to identify your test for you only
Copy User Test From – so you can duplicate an existing test or just start with a blank one.
Once you have filled those fields in you enter into a 5 step process to complete the creation of your user test.
Step 1 – User Test Details
This includes the ability to set your language and Loop11 supports over 40 languages.
Step 2 – Tasks & Questions
You have the ability to add tasks or questions to your user test and Loop11 provides some help if you are unsure what to do in this respect. A task would be something that you would like the user to perform such as find a particular piece of information on your web site and to add one you give it a name, a scenario that tells the user what it is you want them to do, the start URL where they commence the task and the success URL, which the user has to navigate to in order to complete the task.
I created a task to ask the user to find step 6 from the article ‘How To Test Web Forms in 7 Steps’. When the user is carrying out this task they can either abandon the task if they could not accomplish it or mark it as complete.
To add a question, you choose the question type from a list including multiple choice, rating scale and open ended. So you could ask ‘How Usable In This Web Site?’ and ask the user to rate on a scale from 1 to 10 or using words such as Poor, Average, Good, Excellent.
I entered the question, ‘How easy is it to use the search facility on this web site?’ and gave the possible answers as being Poor, Fair, Good and Excellent (I copied them from Loop11’s help text) plus I made the question mandatory.
If you want the user to type in an answer such as correctly answering how much one of your services costs then choose one of the open ended questions.
Each time you had a task or a question you are presented with the Step 2 – Task & Questions screen helpfully showing all the items you have added with the ability to either edit, delete, duplicate, preview or move them up or down in the list. Once you are happy with what you have set you click ‘save and continue’ to move onto step 3.
Step 3 – User Test Options
This is where you set the number of participants that you would like to complete the user test, from 1 to 999 and specify the thank you text the user sees on completion of the site. There are some other useful settings here such as being able to block or allow specific IP addresses.
Step 4 – Invite Participants
So now you get to invite the lucky participants to take part in your user test. Here you have 3 options:
i) Create a link to your user test – Loop11 generates a link for you that you can distribute any way you like.
ii) Create a popup invitation that can reside on your own web site – Loop11 gives you the code for a popup window to be placed on your web site.
iii) Purchase participants – Loop11 gives you the names of 6 companies that you can potentially purchase participants from with a link to direct users to your user test. This seems to essentially be the same as choosing item 1 although you get the details of 6 companies that you can buy users from.
Step 5 – Launch
Once you have sorted out how you are going to invite your participants you end up on the final step, to launch your user test. This gives you a summary of what you have selected and allows you to preview your test before you launch it. You should be careful because once you have launched your test you are not able to go back and edit it unfortunately.
If, like me, you realise that the questions you have set make little sense to someone seeing your web site for the first time then you should go back and edit them now before you finalise your user test.
Preview – when I clicked on the Preview button when still on step 1 I got a message telling me there was an internal server error and that someone had been notified of it. Once I moved onto step 2 the Preview button did seem to work fine and showed me the first couple of pages that a user will see when participating in the web site evaluation.
One of the tasks I had prepared was for a visitor to submit an enquiry form because I wanted to test that visitors could use the enquiry form including field validation. When previewing the test, I submitted an enquiry but my enquiry form kept telling me that I had not input the validation code that I have to prevent automated software from filling it in (even though I had input the code). This is possibly due to Loop11 conflicting with the validation method of my enquiry form so I removed that task from my test. I will raise this separately with Loop11 because many tests could require this type of input and so the preview does need to be able to handle that aspect.
Once you have launched your test you can view the results from within the My Projects tab. For each project you can see the date it was launched, how many responses received, its current status and then you can view the design of the project (what tasks, questions and other settings you selected when creating the project) and its report.
Once your test starts gaining participants you can view your report, which gives you access to the following screens:
Dashboard – first of all this shows the average task completion rate for your test presented as a pie chart. As I only had one task I can see that the average completion rate was about two thirds whilst one third abandoned the task. Underneath is presented the task results overview which shows the average page views and average time taken to reach a resolution for each task. You also get a bar chart showing the task completion rate for each task.
You are able to export the results of the report at any time in either CSV, Excel XML or PDF formats, which I think is pretty useful for any presentations that you are putting together and/or any followup actions required as a result of this testing.
Tasks – this page shows more detail for each task including the completion rate taken from the dashboard but also further aspects such as page views to complete task, time to complete task, most common success page, most common abandon page, most common first click and most common navigation path.
There is also a function called Participant path analysis, which allows you to go through each participant and what they clicked on when completing the task.
These reports give you invaluable information for how users approach each task that you have set them, what they click on first (possibly giving you the most eye opening results), how long it takes them to complete the task and more.
Questions – this report goes through each of the questions that you posed for your users and the results or answers that they gave. For the question in my test, ‘How easy is it to use the search facility on this web site?’, two thirds of respondents felt it was good and one third said it was excellent. However, for my open ended second question, ‘How much does the monthly link check service cost?’ I got a variety of answers where I think respondents did not really understand the question I was asking.
Not only are you testing how users experience your web site, you are also testing your ability to set good tasks and questions that your users will understand.
Participants – the participants section gives you detailed information on each person completing the test including the time they spent on the test, average time taken per task, average page views per task, their IP address, their user agent or browser and the date they completed the test.
Overall I think the results that you get are pretty comprehensive and will allow some very worthwhile followup actions to be completed to improve the user experience of your web site.
The pricing of Loop11 has still to be announced and, if the pricing is not too steep and indications that I have received from Loop11 are that it will be a reasonable price per test, my belief is that you will get a lot of great feedback that will easily pay for the cost of the test as long as you think carefully about the tasks and questions that you set for any participants.
Update (26th August) – I have been informed that pricing for Loop11 will be set at $350 USD per project, which includes unlimited tasks and questions and up to 1,000 participants. This pricing will be in place once the private beta concludes on 1st September.