Garriott says too many Beta testers hurt Tabula Rasa
Filed under: Betas, Sci-fi, Bugs, Events, in-game, MMO industry, Patches, Tabula Rasa
Gamasutra caught up with well-known designer Richard Garriott at the Independent Games Conference held late last month in Austin, TX. During the event he was asked what lessons he took away from the marketing of Tabula Rasa; his response was a bluntly negative appraisal of their Beta test. Too many testers were added too early, in his opinion, which created a current of negative views in the community and soured a lot of potential players on the game. The people who did participate in the beta, we've had to go back to and say 'look, look, we promise: we know it wasn't fun two months ago, but we fixed all that. Really, come try it again.' We've had to go out and develop free programs to invite those people back for free before they go buy it. So the beta process, which we used to think of as a QA process, is really a marketing process.
Massively, of course, was there as well. Check out Elizabeth Harper's great writeup of the keynote in full.






Reader Comments (Page 1 of 1)
12-05-2007 @ 11:56AM
Spars said...
I was a beta tester for TR, getting invited while visiting their booth at GenCon. I can honestly say, beta testing is what turned me off from TR. So I can understand their problem, and the sad part....it turned me off so much I have no plans on checking it back out, even with a "free" offer. First impressions....boy do they ever matter. :(
Reply
12-05-2007 @ 12:21PM
Gr1zz said...
Open beta has always been a marketing process used to test server stability after the game was finished. He should still know this from UO.
“Only about two or three weeks before launch did they(Guild Wars) do the ‘open it up for pretty much anybody to play,’ when the game was basically done," concluded Garriott.
Reply
12-05-2007 @ 12:41PM
Mike Schramm said...
I was also someone who wasn't impressed with the beta, but did they really change things that much? My biggest problem is that I was given a gun and told to shoot aliens, but standard shooter elements (headshots, strafing to dodge) weren't there at all.
And yes, gr1zz is exactly right-- open beta is a marketing tool. This may be worth a post on its own. :)
Reply
12-05-2007 @ 12:47PM
Schad said...
I think this happens quite a lot. It actually happened to me with DDO. The game absolutely sucked. I restarted playing roughly 3 months ago and I've never had a more fun experience. It'd be worth it to the major review sites to "re-review" nearly every MMO and see if it's any better than before.
Reply
12-05-2007 @ 1:22PM
Bildo said...
I was in the beta since it was early alpha. I never really liked the game, and it never really seemed to progress into something fun. I stopped testing because I had other games to test once the NDA lifted.
Now? I bought the game after a 3 day trial. I've played it every day for the past week and a half, and I have trouble pulling myself away from it sometimes.
Best thing ever? Nope. Much better now than in beta? Very true.
They need to hurry up with a 14 day free trial once the AHs are in and economy begins. Because I think everyone needs to try this game before the big duo (AoC and WAR) hit next year. Great fun.
Reply
12-05-2007 @ 1:26PM
Tom P said...
Unfortunately, as a late joiner (this weekend, approx 1 month after TR was released) I'm sad to say the game just isn't that good. Schad's comments regards the FPS/MMO hybrid are all very well, but I see these things as the main problems:
1. Huge FPS hits when the reason isn't clear. Graphically (poly count) the player models are well ahead of WoW, but they all look very similar, and sometimes the silliest thing cut FPS from 60 to 20 for no apparent reason. And yes, I have a VERY nice PC more than capable of running Crysis at 1680x1050 on High detail.
2. Everyone looks the same.
3. All the weapons are the same.
4. Gameplay is the same.
5. The Summoner/Necromancer class is borked.
6. The Healer class is borked - unnecessary until the endgame, although I accept they can be useful in instances.
7. Graphically too plain. Drab, boring colours - I'm not looking for a DayGlo OrcFest like WoW, but hell... there's only so many shades of grey and brown in the world and while I do like both colours, RG seems to have gone out of his way to put them ALL in the game.
In a nutshell, it's not delivering yet. Perhaps in the endgame or when some of the class abilities are fixed, but until then... shelved.
Reply
12-05-2007 @ 1:27PM
Bildo said...
On the point of reviews, game's actually getting quite decent ones, with most places putting it in the 8/10 range. So it's got critic support. But these days if you don't get a 9 you're fudged.
Reply
12-05-2007 @ 1:54PM
Brian McBride said...
I dunno... the release of TR is not that exciting either really. I think old Garrott is just looking for something to blame the lack of success he was hoping for.
WoW had an early beta, that game seems to be doing well. So have others. I dunno, I often have doubts about Garrott's development skills these days. Ultima Online was somewhat of a hit because it was so unique, but so many years later, Richard has not produced another hit at all.
I predict TR will fall into the same camp as Anarcy Online, living but also in a state of obscurity.
Reply
12-05-2007 @ 2:23PM
Malixu said...
Is release TR a lot better than beta? Sure seems it. Is it good? Err, not so sold, myself...
I'd mostly say that they should have kept the beta to hand-picked people, rather than letting for example Fileplanet subscribers in (which included me), until much later on in the beta.
Reply
12-05-2007 @ 2:31PM
Jarek said...
Sad that he would blame beta testing for his game issues. The problem was not that the game had too many testers. Hell thats what salvaged part of the game. If they had not had so many people screaming its not ready I fear they would have stuck to their original schedule and released to a complete disaster.
Beta testing is about tuning class balance, fixing quest bugs, tweaking client and server performance, etc. When a game is opened to closed beta testers the core gameplay and features should all be in place already. Major changes should only be still happening in the alpha stage where everything is in-house. And yes, open beta is mostly a marketing and stress test. Like duh?
Reply
12-05-2007 @ 2:50PM
Markymark said...
Lol if your game sucks it sucks.... don't blame beta testers who think the game sucks when it does for the downfall of the game.
Reply
12-05-2007 @ 3:00PM
Regis said...
Gamasutra? You didn't notice that Massive was there and wrote a great
live coverage from his speech? :)
http://www.massively.com/2007/11/30/live-at-the-independent-game-conference-richard-garriott-keynot/
Reply
12-05-2007 @ 5:39PM
Jeromai said...
Ridiculous to blame the beta testers. I bought the pre-order code to get into the test to both preview the game (by the time you have a preorder invite, your game should be pretty darned polished, no?) and to help it along in its development. I walked into their tutorial and nearly threw up.
I then spent an hour or two writing up in detail what seemed wrong with their first impression of the game and then summarizing it for their forums so that it could be cut and paste and easily sent to and read by the dev team. Things like clunky UI (which is still a complaint to this day), NPCs skittering about, rifle doing insufficient damage to feel heroic, etc. This was still when cover didn't work right and stuff shot right through walls at you.
I was actually quite pleased at the next time I tried their tutorial again, after their next patch, when they got rid of most of those distracting problems. (And got cover working reasonably well.)
Why am I not subscribed to the game?
1) Overhyped promises. The Cult of RG.
Sorry, TR is not revolutionary nor innovative by any stretch of the imagination. If you promised this before, I want to see it before I spend my money on you. I still don't see it. Pasting Garriot's name on top of the game doesn't make it a success either. Sure, the visual arrival of spawns is cute, but it's still in the same place. The ethical parables are neat, but few and far between and doesn't give me a sense of being in a meaningful world.
2) Comparative worth of a subscription.
For $15, I want CoX or WoW quality and quantity/depth of potential gameplay. For $9.99, I'm getting LOTRO. I wasn't able to justify spending $15 for AA's gameplay, nor can I justify spending it on a half-finished TR. I can get something similar (first person shooting, bugs and all) for FREE in Hellgate: London.
3) Dev team didn't appear to have their act together or interact well with their players.
I'm spoiled. CoX dev interaction with their community is stellar. I see Flying Labs doing good things with their PBoTS forums as well. Even WAR and AoC shares insights into their design and development process now and then. TR devs? Utter silence. No sharing of any reasoning as to why certain skills was made as such, and so on, resulting in zero confidence as to their ability to balance their classes properly. I hear they're still balancing with a sledgehammer... (lightning and shotgun was pretty much all you needed beta-wise. Now there's a rage nerf?)
Granted, they work fast since they're churning out heaps of needed fixes and polish after running over their marketing deadline to go gold... Maybe in six months, the game should be where it ought to have started.
4) Personal non-attraction to its gameplay.
Dunno, when someone says FPS-style MMO, I was looking for something that could let me swivel around as fast as an FPS. TR didn't achieve that, the third person rotation of its avatar just felt clunky. Hellgate feels faster to me (except when trying to render lots of entities in one place, then there's major lag.)
I also ended up doing a lot of spawn camping and waiting in line for overhunted mobs, and Fed Ex quests and jogging all over. Didn't someone use the word 'revolutionary' in conjunction with TR some time ago? Ha ha. At least LOTRO doesn't try to hide that it's a WoW clone, and charges less for its non-originality.
XP and loot distribution for grouping and shared kills also seemed wonky. I had the most fun just running off by myself to a deserted shard's Big Tree Hill and going for 5x , 6x combo kills on umpteen amounts of Bane.
That wasn't getting me any progression on the story though, nor could I hope to do something similar once all the shards got crowded. So yeah, I agree with the critics. TR isn't a bad game, but it's not a must-get game either. 7.5 out of 10.
Reply
12-05-2007 @ 6:53PM
Wildhammer said...
Richard Garriot just wanted an opportunity to have something with his name on it sold across the world. The man has an ego the size of Calcutta and then some.
Reply
12-05-2007 @ 9:28PM
Adele Machka said...
I played briefly prior to proper release and now avidly afterwards, I've rather enjoyed myself in both terms of story line, questing, and game play. The beta (albeit admittedly late beta) is part of what sold me on making not only a purchase, but the special edition purchase, and signing up for a full year subscription.
Should content continued to be added down the line, (as I can only imagine it will be ala CoX, WoW et al )l I can see myself resubscribing next year this time as well. I'm personally really enjoying it and ultimately thats what its all about.
Reply
12-05-2007 @ 10:13PM
killcount said...
I was one of those who was turned off from beta, but it was mainly because it was at a point where the memory leaks and lag was so bad I couldn't play. If any one has a trial key, hit me up at killcount@gmail.com, I'd lvoe to give it a second go
Reply
12-07-2007 @ 9:21AM
Green Armadillo said...
In fairness, he does have a point. Companies are opening their betas up far too early in the dev cycle, mainly because games are launching too early in the dev cycle. This isn't usually the fault of the actual developers - the people funding the project want to start making money, and the marketing people want beta buzz to help with the crucial launch day sales (though the buzz is often hindered by the fact that the product is so buggy that the NDA has to be kept in force until the game is almost out on the shelves).
The issue is that you really don't get a second chance to make a first impression. Hellgate: London really drove this point home by releasing a demo that a lot of people didn't like. I finally got into the beta via a paid pre-order key and decided not to go through with purchasing the game cause I just didn't like it. It may be way better now, but I'm not going to give them another chance when it costs me money. Then again, if you don't give me a free trial/beta of some sort, I may be less inclined to buy your game for fear that it'll be a $50 coaster if I hate it since you can't really resell the things.
The answer is simple - don't launch until you're done. Problem is, unless it's all your money, most companies don't have that option.
Reply
12-12-2007 @ 2:18PM
Elidroth said...
Actually the problem is not that beta is happening too early, but that beta has become nothing more than a marketing tool. People try to get into beta periods not to test, but to play for free and tell their friends they're in beta for . Why else would the gaming media give away beta slots as contest rewards?
Beta testing used to be about testing things you really couldn't properly evaluate with a small test group. People expected to find problems because that was their purpose for being included in the 1st place. Now if a game isn't absolutely ready for release at beta, it's chances of success are massively crippled as NDA breaks and general viral (word of mouth) marketing turns people away.
Reply
1-18-2008 @ 7:55PM
Draug said...
A gross excuse for the bad way their developing game options turned out to be.
I did beta test TR, and like me 90% of the beta testing forum posting comunity warnned them the game wasnt ready, the game didnt have enough depth to be even insteresting after the inicial rush, loads of stuff missing, loads of stuff badly implemented, lack of graphics, ( the lvl 40+ zones were not even open to beta testers before the last week before the release lol and were really buggy and almost empty of content ) etc etc and really some weird annoying bugs that went almost with the game till the release some of those server bugs relates with latency etc.
Everyone with 2 cents of inteligence and a minimum experience on the gaming industry could see it would fail and Richard G. knew what was coming.. he and Ncsoft just decided to take the easy road of making some fast bucks.
Reply