Lab publishes NUE test results. Greeters vs mentors: Greeters win
by Tateru Nino 
Oct 8th 2008 at 10:00PM
Filed under: News items, Second Life, Virtual worlds
Linden Lab recently attempted to evaluate the efficacy of its user-volunteer corps as a part of the new user experience (NUE) in Second Life. Two NUE areas were selected (Hanja and Korea), and baseline results established. Then testing was conducted in two phases using one area as a testing ground and the other as a control. The first incorporated themed tours by volunteer greeters, and the second involved general mentoring in conjunction with the original style of Orientation Island tutorials.
Phase one seemed to draw some very successful results, while phase two only showed a slight improvement. The phase two results look to be well within the margins of error, and essentially statistically insignificant. The sort of mentoring you might find at Help Island Public would, based on these results, seem to be broadly ineffective.
 |
Are you a part of the most widely-known collaborative virtual environment or keeping a close eye on it? Massively's Second Life coverage keeps you in the loop. |
It is pleasant (and somewhat surprising) to see some actual published results, though without clear definitions of the terms, the numbers given are of little objective use to third-parties and observers (which is, unfortunately, somewhat less surprising).
For example, among the assorted figures given are 'active first week' -- but since there's no definition of 'active' in this context, and the most two obvious definitions of active is already covered by other statistics in the chart, it is quite difficult to imagine just what the criteria for it might be.
Despite that, it seems fairly clear that the themed guided tours represent a significant short-term gain for the early new user experience over traditional mentoring.
'"Did the help from Second Life Greeters have any impact on your desire to stay in Second Live to learn and discover more?" came back with a resounding 91.7% response that the Greeters did increase the desire to stay in Second life!'
Whether this translates into practical longer-term retention is another matter, but it certainly seems to support the planned scheme to remove Second Life mentors from the NUE, and replace them with a more carefully, and rigorously selected and monitored group of volunteer greeters. The volunteer mentors, as they stand, do not seem to be providing significant benefits to the initial user-experience.
Reader Comments (Page 1 of 1)
Jacek Antonelli said on 10:49PM 10-08-2008
Wow, look at how much it went up! That's a huge spike! It's... oh. It's only a 3% increase? It looks much bigger when you ignore the numbers and just look at the pretty lines.
As you can tell, I'm rather skeptical of the way this rather limited data set is presented. It looks to me as if someone were trying to exaggerate the significance of their results in order to get more funding or support for their program.
If they want to make an *genuinely* convincing report, run each test longer than one week, show more than 1 data point per test, explain what the terms mean, and don't fool around with the axes so much.
Reply
Tateru Nino said on 1:33AM 10-10-2008
That 3% gain is severely exaggerated in the chart. Without definitions, there's no way of really knowing what it means. A 3% gain in *what* exactly?
It makes the graph and the numbers meaningless to anyone except Linden Lab, who are the only ones who know what the data means.
Reply