Massively

The problem I have with MMO beta testing, as it is currently practiced by many software companies, is that there is very little actual testing done. Many, so called, beta testers are just looking to score a free copy of a game before it hits release in order to decide whether or not they will buy the game. The "beta test" has become part of the marketing cycle. That's not why good developers beta test their code. This isn't the fault of the game developer as much as it is an indictment of the avid MMORPG fan who can't wait to play an upcoming release, especially if the "beta test" is free to play. It is important to distinguish between a "free trial" and a "beta test".

To be completely fair, the recent Age of Conan beta was a rather successful test of Funcom's infrastructure and I am reasonably sure that part of the "bloat" of the client involved snippets of code that provided feedback to Funcom in order to help them improve their product. That said, the bloat of the client, as well as rumors of "old client" usage floating around the gaming community might give potential players a bad first impression of an otherwise wonderful game. If I had been representing Funcom, I would have emphasized that this is a test phase and not a free preview. I'd mention the debugging code and add feedback mechanisms to the beta client in order to make it clear to "testers" that this isn't a free ride. Being upfront and honest about the fact that the beta client contains debugging code and other feedback elements would help soothe the expectation of "testers" who might get upset that their free preview is a bit laggy on their system.

As an example of how other companies have managed their beta test without having to worry about the high expectations of "free riders", Sony, with their beta release of Pirates of the Burning Sea , added feedback mechanisms to their beta client that asked the player for criticism during game play, even interrupting the game in order to do so. Players could complete a mission, but then would immediately have a full screen popup asking for feedback and criticism of their gaming experience. That could be frustrating to someone who is looking to play Pirates for free, but was invaluable to someone who was serious about testing Sony's product.

Another example of how to test an upcoming release, although better suited for games that have already been released and are in the process of adding new content, is the public test server. EverQuest used this model for a number of years, and World of Warcraft has used it as well. Sure, players get to play "for free", but the pool of testers is drawn from paid subscribers, thus eliminating the "free rider" that presents such a problem in an open beta. On test servers, players are well aware that content may be buggy, may change without warning, and that their characters may be reset or deleted with no warning. There is no expectation of a "release candidate" product, and players are encouraged to stress the game in whatever fashion they are capable of.

In the end, the problem isn't the company that stages an open beta test. The problem is usually the pool of testers. The average MMO player doesn't know the first thing about testing code. Sure, they can generate bandwidth and mash buttons with the best of them, but systematically testing and verifying functionality is a hard job, and one that the average "beta tester" has no interest in performing. Lately, it seems that MMO companies are just throwing open the doors to their game and hoping that the sample size of testers is large enough to produce some valid or relevant results. It would be more appropriate to call a public "beta" a "stress test" and set expectations with the testers accordingly.

Once again, if you have any questions that you would like to see answered in Ask Massively, feel free to drop us a line at ask AT massively DOT com or stop by our tipline. If you'll excuse me, I'm going to go play the Age of Conan beta.