Earlier this year I conducted a remote usability study on our own website, nngroup.com, to evaluate its content.

I could have conducted a traditional in-person usability study; however, I wanted to take advantage of the benefits of moderated remote usability testing: saving money and capturing users in their own environment in real time, as they were doing their tasks. I was even able to conduct several international usability sessions, without leaving my office in California.

I had 4 requirements for the study:

• Participants had to be interested in the content on our site because they were considering attending a UX Conference seminar or purchasing a research report.
• I had to be able to directly communicate with the participants and see their screen.
• The sessions needed to be recorded.
• Observers had to be able to watch from anywhere.

Traditionally, for a usability study, representative users are recruited and scheduled in advance. Recruiting representative users is difficult and expensive for a B2B site with a narrowly targeted audience. Also, since I was specifically interested in the content on the website, I wanted to ensure engagement. To do this, I opted for recruiting people who were using our website and running a test with them right then and there, during their visit to the site–that is, run a live-intercept study. This ensured that they were: 1) interested in the information on our website, and 2) actual users of our website.

Here's the quick 9-step summary of my test setup, described in more detail below:

1. Participants were recruited on nngroup.com through ethnio. 
2. If they qualified, I called the participant on the number they provided in the screener.
3. I started a GoToMeeting (GTM) session on my computer.
4. Over the phone I gave him the instructions to join my GTM session.
5. At this point I sent an email out to the observers (who were also remote) with the same GTM session details.
6. To get the audio connected without having to end the call in step 3, I started a 3-way call with the GTM phone line.
7. I made the participant the presenter, so I could see her screen.
8. Then I started the recording in GTM.
9. Immediately after the session, I emailed the participant an Amazon gift card and a thank-you note.

It's obviously possible to use other tools than the ones listed here, and we certainly don't endorse these tools as necessarily being the best. However, they were suitable for our project.

Recruiting Participants with ethnio

Ethnio is a customizable screener that can be placed as a pop-up on your site or accessed through a direct link to an ethnio URL.

Since I wanted to recruit users from nngroup.com, I opted for the intercept from our site. Ethnio provides a 2-line JavaScript code, which I placed in our CMS. The resulting pop-up looked like this:

The initial screener pop-up placed on course- and report-description pages on nngroup.com. This pop-up appeared 15 seconds after users first arrived on a page.

I placed the screener pop-up on pages for both seminar and research reports on our website. The screener appeared with a 15-second delay — that is 15 seconds after the page was loaded on the participant's computer. From our analytics, I knew users spent an average of 80 seconds on these pages. Ethnio offers the option to present the pop-up immediately on page load; however I wanted to give participants a chance to determine if the page was right for them and to start looking for the information they were interested in.

Thus, by introducing the pop-up with a delay I weeded out those visitors not interested in the content on that page. I also found that a 15-second delay worked better than a 30-second delay. This could be because in 30 seconds users may have determined that the seminar or report was not for them and may have left the page, or they may have been already deeply engaged in the content and may have dismissed the pop-up because it disrupted their reading.

It also took some trial and error to determine the right balance between session length and incentive. More site visitors responded to the screener when I asked for 20 minutes of their time instead of 30 minutes; however 20 minutes was too short to conduct a thorough test. The extra 10 minutes were necessary for setup and debriefing. In terms of incentive, as you can imagine, the more money I offered, the more users responded to the screener.

Participants were offered an Amazon gift card for their time. I customized the gift card with our logo and emailed them their gift card after the session. Amazon gift cards worked well for this study, because they can be purchased ahead of time, are familiar and trustworthy, and available in several countries.

In the ethnio pop-up, if the user clicked Continue, he was taken to the screener questions. Ethnio screeners allow for 10 customizable questions. Ethnio also provides the option to automate screening and will send participants directly to, for example, a usertesting.com session based on their answers.

However, I needed more than 10 questions and wanted to review their answers. So I used 7 questions from ethnio to ask for basic information and then completed the screening process over the phone.

Basic screening questions asked through the ethnio interface.

The final screener page, in which participants were told they’d be contacted by phone if they qualified.

I set up ethnio to email me whenever someone completed the screener. I made sure to have the ethnio screener live on our website only when I was available, so users wouldn’t complete it and not get a phone call.

An example of an email from ethnio notifying me of a new recruit.

Starting the GoToMeeting Session

If the participant qualified and agreed to the study, I walked him through the instructions provided by GoToMeeting when you click on Invite Others:

I provided this information to the participant over the phone so he could join the GTM session.

I also emailed these same instructions to observers (using Copy to Clipboard in the Invite Others window).

If you’re going to have observers dropping in and out of sessions, you might want to disable the beep that plays by default when people join or exit a meeting. You can do this by going to the Audio section in the Edit menu and un-checking Play Entry/Exit Chimes.

When the participant joined, he saw a white screen with some details about the meeting.

The initial screen seen by participants who join a GoToMeeting session.

Only 2 of the 15 participants had trouble installing GoToMeeting, due to firewall settings at their work.

Screen Sharing and Audio

To get the audio connected without having to end the existing call with the participant, I added the phone line that GoToMeeting provided to the call with the participant. I told the participant I'd be putting them on a brief hold; then I started a 3-way conference call. To GTM it looks like there’s only one audio line, but it actually captures both the facilitator and the participant. (When you conduct scheduled moderated remote usability sessions, you can provide the participant with the phone number to dial into).

The phone-number information provided by GTM, needed to capture audio in the recording.

Then I made the participant the presenter:

To see the participant’s screen you have to make him the presenter, by selecting his name in the Change Presenter dropdown.

I also made sure to disable most of the attendee features. Participants didn’t need to be able to chat, see attendees, or share webcam.

Unselect all three items in the Options menu to disable these features for attendees.

Recording Sessions

The final step was to start the recording, by clicking the Start Recording button. Make sure you have the same options selected in the Settings dialog as in the screenshot below:

Select the Settings link next to the recording button. Then make the indicated selections in the next window.

To end the session, you can either end the meeting in the File menu or stop the recording first. If you end the meeting, the recording automatically stops and is saved where you indicated.

Conclusion

Recruiting participants live from our website helped me locate engaged target users and use the tasks that they had in mind. By intercepting them when I was available to conduct a study, I could start the session immediately.

Keep in mind that intercept recruiting has a low response rate (that is, few users will complete the screener). In this study 2% of the users who viewed the screener completed it. (This matches ethnio’s reported average 2% response rate.) Of the users who completed the screener, 30% qualified for the study. Of those users, 60% participated (this number is 18% of the people who completed the screener and 0.3% of those who viewed the screener — in other words, we had one successful test for every 500 visitors to a page of interest).

The number of users who completed the screener (Response rate), qualified for the study (Selection rate), and actually participated (Success rate).

I found that GoToMeeting offered all the functionality needed to see the participants’ screens and record the sessions. The only work around was for the audio connection, for which I placed a 3-way call between me, the participant, and the GTM line. This was only necessary because I wanted to call the participant before the session to qualify them for the study, rather than emailing them the GoToMeeting details.

For more detailed advice on this type of user research (including hands-on practice), please come to our full-day training course Usability Testing.