User testing in Second Life

Oz-IA ran over the weekend in Sydney and it was a fantastic event.

There were some really interesting presentations, and while I didn’t get to all of them, without doubt the most interesting for me was User Research in Virtual Worlds, a presentation I highlighted earlier as being one I wanted to see.

Gary Bunker and Gabriele Hermansson, both from Hyro, spoke for close to an hour about their experiences in setting up and running an experimental focus group within Second Life.

Now that virtual worlds (such as SecondLife and World of Warcraft) are becoming vastly more popular, Hyro set out to build a research platform that would allow us to research users within those worlds, not only for their experiences there but also for their needs outside of it. We wanted to know if we could use virtual research – focus groups, interviews and user testing – in a practical way in design projects requiring complex user input. We also wanted to understand how user behavior would change between real-world and virtual forums.

Going into the presentation I was somewhat skeptical about how they might have conducted the evaluation, but by the end of the session I was really very engaged and intrigued.

The Focus group in Second Life was conducted as a trial to test its feasibility for future use, and as such a second focus group was conducted in parallel, in the physical world. This allowed them to benchmark the findings gathered in Second Life against those that were recorded during the traditional focus group session.

The Session in Second Life was run similarly to how we would conduct a focus group ‘in real life’. Participants were recruited (as they teleported in), screened and scheduled for the session just as we normally would.

During the session, participants signed a code of conduct and consent forms, and received instructions on how the session would be run. ‘Virtual Participants’ were recruited from both Australia and the UK, with the focus group being recorded with a media camera and chat logger.

Things that worked

  • There was a high level of feedback
  • Participants were comfortable
  • There was a good level of interaction with the participants
  • The focus group had an international reach, which was a requirement of the testing
  • Findings of the online session matched those recorded in the offline session

Things that didn’t work so well

  • Online sessions took about one and a half times as long (i.e. 1.5 hrs online and 1hr offline)
  • There were multiple conversation threads running at the same time, which were difficult to track
  • The response time of some participants was slow
  • It was confusing if participants weren’t identified directly by name during discussion, as it was at times difficult to indicate who the focus group facilitator was addressing

Lessons Learned

  • Fewer participants (maximum 4) would be easier to manage
  • There was a need for more facilitators than usual (minimum two)
  • More time was required than usual
  • Using coloured text instead of everyone having the same colour would be useful, particularly between facilitators and participants
  • Payment to participants must be immediate in the virtual world or people get nervous
  • It’s essential (as in the real world) that the screener isn’t leading, as it’s impossible to know what the participants are really like
  • Get the right people – open, talkative, respectful and interested
  • Facilitators must be able to type quickly, multitask and know the platform
  • Give a focused response – use the participants name when addressing them
  • Summarise the participants feedback so that they know you have heard and understood them (eye contact isn’t possible online)
  • Be clear that you’ve finished talking about a particular topic and that you’re moving on to the next one.

From my perspective, the biggest barrier to entry would be setting up the virtual space and building the focus group facilities online. As an independent consultant, that’s a massive effort that I’m unlikely to invest in. Even for smaller companies it’s a huge amount of work.

But it is interesting… *very interesting*.

It’s an example of how technology is allowing us to change the way we work. No doubt there are a lot of questions about the validity of the participants recruited for the Second Life Focus Group. But I’m a firm believer that most of what happens online can happen offline and vice versa. I’ve interviewed a few doubtful characters myself over the last few years, despite rigorous screeners. So Hyro’s report that results from the virtual focus group were very similar to that of the real life focus group is a good sign and perhaps not too much of a surprise after all.

Replay

Category: Patterns & Practices Time: 2007-09-25 Views: 0
Tags:

Related post

  • User Testing: 5-second test vs. 8-second test 2011-01-27

    I'm planning a "User Testing" and one of the tasks is the 5 (or 8) second test, so I want to know which fits better in my interface. I can't remember what article talks about the difference between 5-second test and 8-second test, can someone re

  • Second Life for a weekend 2006-07-17

    Had a free weekend to burn so rather than doing something sensible, like enjoying the great weather, decided to check out Second Life from Linden Labs, which has been popping up on more and more radars in the last few months, and just starting to rea

  • Can't run Second life firestorm viewer (32bit) in ubuntu 64 bit 2013-06-02

    I'm trying to install a Second Life client called Phoenix Viewer. On the first attempt, I got a message about X errors. I searched the net all I could see to fix 32 bit apps and run them under 64 bit Ubuntu was shared libraries issues, so I installed

  • How do I install the Firestorm viewer for Second Life? 2012-06-24

    I am new to Ubuntu and trying to set everything up. I am VERY bad at doing that at the moment. In fact, I asked another question here only a few hours ago. Anyways, I am trying to get the Firestorm Viewer for Second Life. I followed instruction given

  • Has user testing found that the "three horizontal bars" for main menu on mobile is commonly understood? 2013-09-16

    "Three horizontal lines" aka "hamburger icon" seems to be becoming a convention for the "main menu" button, especially on mobile sites: Has anyone done any user testing of this convention, or got any A/B testing results? (i.e

  • A/B testing in user test 2014-02-12

    I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the interface is the layout. The informations are the same

  • User Testing Framework and Questions for comparative tests 2014-02-19

    I am planning to conduct a number of task-based moderated think aloud tests for a service that I am designing. I will be repeating the test with the same setup regularly throughout the year and I am looking for a framework of questions that I can use

  • How to allow for 'exploration' in your user testing? 2014-10-02

    In a lot of user testing sessions I've been in a common task is to test the effectiveness of a particular icon or navigation control. The typical scenario: tester: How would you accomplish task x? testee: They stare at the screen for a while because

  • User Testing: Painful- But Worth It 2015-07-09

    Perhaps this is a familiar story. You've built a new website or app, dogfooded it with your team, and now you're ready to test with real-life users. Excitedly, you gather a few people who are most likely to be your target audience, and begin the test

  • The Absolute Beginner's Guide to User Testing 2015-09-08

    This article is a companion piece to my newly launched incrementalux podcast and is based on my learning from interviewing people much smarter than I am. In Pursuit of the Perfect Product While user-testing has always had a place in the web developme

  • Should we ask users to play a role when performing user testing? 2015-10-26

    I'm actually working on a users-testing session, to test an expense report application. My question is: When I ask the user to do a task, should I ask them to play a role? Example: If we want users to declare a train ticket, should we say them: "You

  • Twitter Recruiting Members for User Testing 2007-06-19

    Like it or hate it, Twitter has caused a stir in the online social networking space. In a nut shell, Twitter allows users to pump out infinite amounts of (usually) ego-driven noise to the intarweb with a multitude of widgets, mashups and plugins that

  • Paying test subjects for user testing/beta testing 2010-11-19

    I've read that focus groups are a poor way of conducting user research as it attracts professional focus groupers who work the focus group "circuit" for a living. This obviously has major drawbacks as you're getting "expert" opinions i

  • What User Testing techniques do you use? 2011-01-10

    In producing dynamic maps for students/users I have found the time invested in user testing well worth it. I use a varient of think aloud protocol called hallway testing. I grab any 2 or 3 likely users for 20 minutes each and set them a sample task (

  • How to uninstall Second Life installer.sh 2011-01-18

    How do I uninstall Second Life I downloaded the Second Life viewer from install.sh --------------Solutions------------- If what you mean is that you downloaded the archive from the Secondlife web site, extracted the archive and ran the install.sh scr

  • What's the best way to test the battery life on a MacBook Air? 2011-02-06

    I just recently got a MacBook Air and noticed that its battery life is much lower than Apple's estimate of 7 hours. Since it is not the same, I would like to benchmark my battery life right now (while the battery is fairly new) so that in the future

  • User Testing and other best/most effective ways to implement UX in webpage 2011-04-11

    I am interested to know what some effective ways to determine and implement user experience would be. I think that user testing is a very effective one, i.e. getting people who are completely unfamiliar to the system using and testing it to determine

  • Isn't all user testing essentially a way of gathering how people think things should work? 2011-07-12

    When user testing is performed, users are often presented with a serious of variations of a product and observed to see where they excelled and where they had trouble. People will often click in areas that are not intended to be clicked on, or may tr

  • Plus icon for new tab not working in user testing. Anomaly or actually confusing for users- 2011-07-18

    For Context: So I am performing some user testing, and the interface is changing how tabs work with in the system. Currently tabs indicate the top level of different customer define hierarchies. The new interface is changing tabs to act like browser

iOS development

Android development

Python development

JAVA development

Development language

PHP development

Ruby development

search

Front-end development

Database

development tools

Open Platform

Javascript development

.NET development

cloud computing

server

Copyright (C) avrocks.com, All Rights Reserved.

processed in 1.105 (s). 13 q(s)