EVE Evolved: EVE Online's server model
Filed under: Sci-fi, EVE Online, Business models, Economy, MMO industry, PvP, Politics, Virtual worlds, EVE Evolved
Almost any time a discussion about EVE Online comes up, one way or another we end up talking about the server. EVE Online is unique among today's most popular MMOs for its single-server approach. While most MMOs deal with large number of users by starting up large numbers of separate servers with identical game universes, EVE maintains only a single copy of its game universe on a massive cluster of servers. CCP's decision to go with a server model that doesn't use any sharding or instancing whatsoever has had a major impact on in-game activities and how the game has developed.
Server woes:
Unfortunately for CCP, maintaining their vision of a single game universe has proven a lot more difficult and costly than anyone anticipated. Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry. Even with this massive power behind the EVE universe, there are still problems as CCP tries to keep the server upgraded ahead of its ever-expanding playerbase.
In this article, I discuss the unique gameplay that is possible thanks to EVE's server model, the problems the server currently faces and what CCP is planning to do about it.
With a single game world, players are free to flock to whatever solar system they like even if the server that solar system is on can't handle the load. Popular trade hub Jita and several popular mission-running systems such as Dodixie suffer badly from lag at peak play times on the weekend. Massive fleet battles over important territory often suffer from the same crippling lag, turning what should be an amazing sci-fi space battle into a poor slideshow.
Server structure:
Each of EVE's 5000+ star systems is loaded as a separate process onto any one of hundreds of IBM blade servers, with some high-load systems being given a server all to themselves and many low-load systems being combined and run on servers together. These "SOL Servers" are tied into EVE's main database server where changes to the game take place (where the magic happens).
Since players need to move between solar systems, they are connected to proxy servers which keep track of which SOL server the player is on. It's an ambitious system but has worked well for over five years with constant upgrades going on in the background to keep up with the increasing number of players entering EVE daily.
Effect on PvP:
You could be forgiven for thinking that an MMO's server model doesn't affect its gameplay significantly but EVE Online has proven this wrong for five years running. Putting all players together in one server drastically increases the opportunity for PvP. Instead of the MMO norm of less than 5,000 potential players for you to interact with and barely 1000 online at peak hours, EVE's server houses over 300,000 players with a peak concurrent user record of over 40,000. Additionally, since there's only one server for all players, there's no option to sign up to a non-pvp version of the game. This puts all players in the same world with the same rules whether they like it or not. If all you want to do is trade, mine and run missions you're just as vulnerable to PvP as everyone else and that's a major factor in defining the harsh feel of the EVE universe.
If EVE did offer a non-PvP server option, roles such as the pirate or corporate spy wouldn't really be possible any more because most potential targets would be playing on the non-pvp server. The players on the non-pvp server would also suffer from having a duller, less challenging game experience. We'd have one server full of hunters with no prey and one server full of prey with no excitement to their game.
Ultima Online experienced this issue in the Renaissance expansion when they released Trammel, a server where non-consensual PvP was no longer possible. With all the cut-throat villains separated from the general population, the villains had nothing to do and the remaining players lost their opportunity to be heroes.
Territorial conflict:
The lack of instancing in EVE Online's game universe has had an even more profound impact on PvP than the lack of a non-PvP server. When a solar system is depleted of resources, is becoming overcrowded or is being camped by pirates, there is no second instance of the system to switch to. The ability to pursue attackers from system to system successfully or to lock down a star system and prevent your enemy from passing through allows for piracy and very real territorial control that just isn't possible with another server model. Conflict over resources and space arises as a natural consequence of gameplay and not from a developer-defined game mechanic. Real player alliances are forged and broken every week in EVE with complex politics behind them.
Economic impact:
EVE is often lauded for its realistic player-based economy and real working markets but neither of these would function well on a sharded server. Throwing all of the players together in one place forces the markets to act based heavily on the rules of supply and demand. Without enough players driving both sides of the supply and demand curve, a single player could interfere with the global market very easily for a very long time. This has been done before in games like World of Warcraft to manipulate prices for a profit. It worked because with so few players on each server, one particularly rich player's effect on the market can be proportionally massive.
In EVE's safer systems, even major price manipulations tend to be balanced out by other players in a matter of hours, making price manipulation in trading hubs a very expensive and risky venture. It's said that the number of players in EVE caused the markets to hit critical mass long ago, reaching a point where high demand is almost always met by players with adequate supply within reasonable time-frames. As a result, the game's market hubs are always stocked full of whatever you might need.
Upgrades:
CCP recently encountered a problem they hadn't seen since late 2005. Certain server nodes were running out of memory, filling up with legitimate user data and crashing. Their response has been a controversial change to introduce player limits to star systems under heavy load. Although this was changed to only affect trade hub Jita for now, it highlights hardware inadequacy that CCP are meeting head-on with another round of server upgrades. The current server hardware uses impressive processors and advanced solid-state RAMSAN disks with the fast access speeds and large storage capacity that EVE's servers require.The bottleneck at the moment is getting data from one processor or ram disk to another and this is where their latest project comes in.
CCP aims to link the processors and RAM drives of every SOL server together with high-speed low-latency "Infiniband" technology, allowing data transfer at rates of several gigabytes per second. This will allow any processes which can be threaded to be split off and run on a processor that's not being used heavily at runtime, which should massively increase the server's load-balancing ability. The infiniband project poses a huge task for EVE's programmers, who are in the unenviable position of having to rewrite large portions of the core server code. If all goes well with their internal infiniband tests, these major changes in server architecture could eventually spell the end of laggy fleet battles and node crashes.
Summary:
With the problems CCP are constantly facing with their server and the cost of its upkeep, other developers seem reluctant to take EVE Online's server model on board. However, this model affects a lot more than its running costs and complexity and may be practically required for any successful next-generation PvP based MMO. It makes avenues of gameplay such as meaningful politics, piracy and real territorial warfare not just a possibility but an unavoidable consequence of group play. Could the single server approach become commonplace in the next generation of PvP-based MMO? I, for one, hope it does.
Server woes:
Unfortunately for CCP, maintaining their vision of a single game universe has proven a lot more difficult and costly than anyone anticipated. Working with IBM, the EVE server cluster is maintained in London and is currently the largest supercomputer employed in the gaming industry. Even with this massive power behind the EVE universe, there are still problems as CCP tries to keep the server upgraded ahead of its ever-expanding playerbase.
In this article, I discuss the unique gameplay that is possible thanks to EVE's server model, the problems the server currently faces and what CCP is planning to do about it.
With a single game world, players are free to flock to whatever solar system they like even if the server that solar system is on can't handle the load. Popular trade hub Jita and several popular mission-running systems such as Dodixie suffer badly from lag at peak play times on the weekend. Massive fleet battles over important territory often suffer from the same crippling lag, turning what should be an amazing sci-fi space battle into a poor slideshow.
Server structure:

Each of EVE's 5000+ star systems is loaded as a separate process onto any one of hundreds of IBM blade servers, with some high-load systems being given a server all to themselves and many low-load systems being combined and run on servers together. These "SOL Servers" are tied into EVE's main database server where changes to the game take place (where the magic happens).
Since players need to move between solar systems, they are connected to proxy servers which keep track of which SOL server the player is on. It's an ambitious system but has worked well for over five years with constant upgrades going on in the background to keep up with the increasing number of players entering EVE daily.
Effect on PvP:
You could be forgiven for thinking that an MMO's server model doesn't affect its gameplay significantly but EVE Online has proven this wrong for five years running. Putting all players together in one server drastically increases the opportunity for PvP. Instead of the MMO norm of less than 5,000 potential players for you to interact with and barely 1000 online at peak hours, EVE's server houses over 300,000 players with a peak concurrent user record of over 40,000. Additionally, since there's only one server for all players, there's no option to sign up to a non-pvp version of the game. This puts all players in the same world with the same rules whether they like it or not. If all you want to do is trade, mine and run missions you're just as vulnerable to PvP as everyone else and that's a major factor in defining the harsh feel of the EVE universe.
If EVE did offer a non-PvP server option, roles such as the pirate or corporate spy wouldn't really be possible any more because most potential targets would be playing on the non-pvp server. The players on the non-pvp server would also suffer from having a duller, less challenging game experience. We'd have one server full of hunters with no prey and one server full of prey with no excitement to their game.Ultima Online experienced this issue in the Renaissance expansion when they released Trammel, a server where non-consensual PvP was no longer possible. With all the cut-throat villains separated from the general population, the villains had nothing to do and the remaining players lost their opportunity to be heroes.
Territorial conflict:
The lack of instancing in EVE Online's game universe has had an even more profound impact on PvP than the lack of a non-PvP server. When a solar system is depleted of resources, is becoming overcrowded or is being camped by pirates, there is no second instance of the system to switch to. The ability to pursue attackers from system to system successfully or to lock down a star system and prevent your enemy from passing through allows for piracy and very real territorial control that just isn't possible with another server model. Conflict over resources and space arises as a natural consequence of gameplay and not from a developer-defined game mechanic. Real player alliances are forged and broken every week in EVE with complex politics behind them.
Economic impact:

EVE is often lauded for its realistic player-based economy and real working markets but neither of these would function well on a sharded server. Throwing all of the players together in one place forces the markets to act based heavily on the rules of supply and demand. Without enough players driving both sides of the supply and demand curve, a single player could interfere with the global market very easily for a very long time. This has been done before in games like World of Warcraft to manipulate prices for a profit. It worked because with so few players on each server, one particularly rich player's effect on the market can be proportionally massive.
In EVE's safer systems, even major price manipulations tend to be balanced out by other players in a matter of hours, making price manipulation in trading hubs a very expensive and risky venture. It's said that the number of players in EVE caused the markets to hit critical mass long ago, reaching a point where high demand is almost always met by players with adequate supply within reasonable time-frames. As a result, the game's market hubs are always stocked full of whatever you might need.
Upgrades:
CCP recently encountered a problem they hadn't seen since late 2005. Certain server nodes were running out of memory, filling up with legitimate user data and crashing. Their response has been a controversial change to introduce player limits to star systems under heavy load. Although this was changed to only affect trade hub Jita for now, it highlights hardware inadequacy that CCP are meeting head-on with another round of server upgrades. The current server hardware uses impressive processors and advanced solid-state RAMSAN disks with the fast access speeds and large storage capacity that EVE's servers require.The bottleneck at the moment is getting data from one processor or ram disk to another and this is where their latest project comes in.

CCP aims to link the processors and RAM drives of every SOL server together with high-speed low-latency "Infiniband" technology, allowing data transfer at rates of several gigabytes per second. This will allow any processes which can be threaded to be split off and run on a processor that's not being used heavily at runtime, which should massively increase the server's load-balancing ability. The infiniband project poses a huge task for EVE's programmers, who are in the unenviable position of having to rewrite large portions of the core server code. If all goes well with their internal infiniband tests, these major changes in server architecture could eventually spell the end of laggy fleet battles and node crashes.
Summary:
With the problems CCP are constantly facing with their server and the cost of its upkeep, other developers seem reluctant to take EVE Online's server model on board. However, this model affects a lot more than its running costs and complexity and may be practically required for any successful next-generation PvP based MMO. It makes avenues of gameplay such as meaningful politics, piracy and real territorial warfare not just a possibility but an unavoidable consequence of group play. Could the single server approach become commonplace in the next generation of PvP-based MMO? I, for one, hope it does.

















Reader Comments (Page 1 of 2)
Harmen said on 5:51PM 9-28-2008
... The infiniband project poses a huge task for EVE's programmers, who are in the unenviable position of having to rewrite large portions of the core server code....
s/unenviable/enviable/
Sounds like a fun project to me; quiet the opposite of unenviable :)
Reply
Brendan Drain said on 6:12PM 9-28-2008
If I'm being absolutely honest with myself, I'd like to be on a project like that too. I'll be doing a big project this year for the final year of a Masters degree in Computer Science and would love to get something like that.
The EVE server guys are constantly pushing into unexplored territory, no other MMO uses their single distributed server model.
Reply
Erbo Evans said on 9:06PM 9-28-2008
Heh...only now is CCP needing to employ networking technologies that we in the supercomputer cluster business have known about for years. :-)
Reply
Brendan Drain said on 12:11PM 9-29-2008
It's more that those technologies are economically viable for them to use now. One of the devs mentioned in an interview in late 2007 that they were going to focus on using technologies that were now becoming financially feasible such as infiniband. It's probably taken until now to get the project underway and make any real progress.
CrazyKinux said on 9:17PM 9-28-2008
The geek in me who dabbled and dibbled with Computer Science for 2 semesters rejoices at such article!
I just wish CCP would release a couple of Dev Bogs on the subject like they did more regularly a few years back.
Great article!
Reply
michael, St E said on 7:48AM 9-29-2008
Well, there was a Devblog released on Saturday, which described how Jita came to run out of memory. Classic case of the removal of one bottle-neck resulting in the discovery of another. :)
I popped into Jita Sunday afternoon; 710 in local, no major lag, station performance fine. Much improved!
CrazyKinux said on 11:32AM 9-29-2008
Mike, you're one brave (or crazy) capsuleer to venture out into Jita on a Sunday!! =)
Mythic said on 1:41AM 9-29-2008
Awesome article! I hope more MMO's go single server in the future.
Guild Wars developed a good compromise that made it easy to always play with your friends while minimizing overcrowding with dynamic instances of all multi-player zones.
The Eve system is even more amazing, and they are definitely on the bleeding edge of massive tech!
Reply
Sephirah said on 8:13AM 9-29-2008
Nice article!
Hope for more MMO related technology articles!
Reply
recursive said on 4:45PM 9-29-2008
I'll just take this moment to add myself to the nice article list.
Interesting to hear how they do it, though I don't think it's effects are just limited to PvP MMOs, any developer should probably at least take notice of the things CCP's run into during their rise to fame.
Reply
Coherent said on 5:11AM 9-30-2008
Heaven forfend EVE should create a PVE server shard... only to have the players wake up to the fact that there is essentially zero playable content in the game.
EVE replaces meaningful content with mindless conflict. Strangely enough, there are many who are comfortable with this - they're welcome to it.
Reply
Brendan Drain said on 11:18AM 9-30-2008
The thing is, it's not mindless conflict. Fighting a game of capture the flag for points or equipment as is done in many other MMOs would be mindless and arbitrary. EVE's PvP is a means to an end and results as a natural consequence of competition for resources. There are no pre-established rules or victory conditions, just armies marching under their respective banners for their own reasons.
EVE is a PvP-based MMO, I've always thought of the PvE portions as just being there to allow players to make isk with which to buy ships to PvP in. There are a lot of players who do nothing but missions and mining and such but they usually get bored and quit. Anyone playing EVE with no intention of joining in on the large-scale PvP is wasting their money. PvP is what the entire game is based around and is the most fun I've had in an MMO to date.
Benjin said on 12:06AM 10-02-2008
Loved the article. The single game world really appeals on so many different levels. How many times have I wished I was on server XYZ so I could play with my friends who I discovered were also playing?
@ Harmon and Brendan: CCP sounds crazy cool and they have lots of job openings on their website right now too with the possibility to relocate to Iceland or China! Almost makes me wish I hated my current job and didn't own a home so I could become a conceptual artist...
http://www.ccp.is/jobs.aspx
Reply
Dash said on 11:43AM 10-02-2008
As an EVE player, I realy liked this article. Yes, it is about the tech side of the game, but it talks about how it affects gameplay. It is true, EVE need it's one-world structure to maintain it's harsh gameplay and economic structure, which are the reasons people play it.
I'm (mostly) a miner there, but a non-pvp server would be the end of the game to me. Unlike most games, the AI enemies are not a problem, the danger comes from every single player (in fact there is no 100% safe place in the game). And there is no level grinding, no NPC buying your stuff, so any form of money making is also dependent on others (market, labs, pirating).
But is this model sustainable? It is costly and hard to keep lag free. Never heard of another MMO using the same system (and I played over 50). Anyone know any one that could use this system?
Reply
NIkudada said on 9:26PM 10-02-2008
Could their engine and server tech be bought by another company to make something akin to planetside? I mean, a mmofps like wwII online that doesn't look like it was designed in the late eighties would be pretty amazing.
Reply
Brendan Drain said on 10:02PM 10-02-2008
The general model is very difficult to apply to a first-person perspective. It's all about how EVE handles space. Space is divided into cubes around 1000km across called grids. This is so you only need to recieve information on what's happening on your grid rather than what's happening everywhere in the entire solar system. Although this does nothing for each server's processing load, it dramatically reduces its output network load and thus reduces lag.
Translating this system to other MMOs, the closest system would be the standard zone model that Everquest 2 uses (among others). Each zone is a separate, distinct area like a grid. The problem is that if the number of people in one zone increases beyond the maximum that one server can handle (either in CPU load or network output), problems like lag start to occur. Zones would need to be chopped up so that they are so small that there are less people in each zone to handle.
The issue with chopping zones up is that moving from one zone to another takes time and the smaller the zones are, the more annoying and impractical travelling becomes. EVE solves this problem by implementing almost seamless grid transition and their server model allows players to move from one solar system to another relatively quickly (under 15-30 seconds under ideal conditions). With infiniband, we could see this delay disappear or be reduced to a few seconds flat.
I can see how the server architecture could be applied to a first person game but the design challenges make it incredibly difficult to get right. It's not just a matter of a company buying the designs or engine from CCP, it would need to be completely gutted and redesigned for a first person perspective game. The games industry's major investors seem unwilling to risk their money on big changes like that, unfortunately, but it would make one hell of a game if someone managed it.
Nikudada said on 6:35PM 10-13-2008
I appreciate your detailed response. While i agree that it would be a hard sell you can't deny that there are a glut of investors willing to risk pretty serious capital on the MMO model. Funcom managed to secure an additional 30 million in private equity last year so I don think its too absurd. Really the biggest barrior from a financial perspective would be ensuring you market your potential game as something which would will, given time, WILL work. Risk, in this space, is inherent. I would think a project like the mmo Otherland would seem overy abitious to most investors but they apparently got somone to foot the bill.
I suppose, though, the question remains if it wouldn't just be smarter to build an engine from scratch since you would (and i'm going off what you wrote) have to maniuplate so much of EVEs engine that it would have the same number of bugs accociated as a new engine.
Frank said on 1:21AM 10-03-2008
The word "slideshow" is far to polite a term for the 45 minute blackscreen fleet grid-load odyssey, only to have the node crash 5 minutes later. After another 20 minute login, you arrive already dead.
Eve players would welcome "slideshow" quality in fleet battles. Heck, I'd pay $5 more a month in order to obtain " slideshow" quality in a fleet battle.
Reply
Tar-om said on 9:51AM 10-03-2008
"these major changes in server architecture could eventually spell the end of laggy fleet battles and node crashes. "
Sadly that is not the case. Although the changes will help run heavily loaded systems like Jita and the main mission hubs far more smoothly, it will do nothing for fleet combat.
CCP are not proposing to multithread individual grids (read comment 16 above) which means that since all fleet combat takes place in a single grid, we will still be in the situation that both fleets and their interaction is being handled by a single thread on a single processor core.
Although this is not the exact same situation at present where the entire solar system is handled by that single thread, it isn't far off since nearly everyone in the system will be in that single grid when the combat starts. Also, anyone warping to the combat will still have the horrific grid loading time that plagues now.
When challenged on this point, CCP said that the timing issues would be too great to resolve. Which is a shame, since the endgame in EVE is broken and, by CCP's admission, too difficult to fix.
Reply
Brendan Drain said on 2:05PM 10-03-2008
You'll have to dig your way through the forum threads on the issue to find this info but essentially CCP's long term goal does appear to be segmenting the combat occuring on one grid and delivering it to multiple CPUs. A developer mentioned separating the combat and the reporting of that combat to clients, for example. Currently these are done in the same thread but they could potentially be split off into two.
Unfortunately it is the case that there's only so much they can split up a fleet battle. All of the actual combat calculations (except random number generation) must be performed in the same thread to avoid the sync issues you mentioned. I still think a lot of people are underestimating the potential of the project to increase the size fleet battles can be before they lag. Although they're initially only going to be using the infiniband for dynamic run-time load balancing of entire star systems, the potential is there for everything threadable to be split up, spreading load out significantly.
My favourite possibility (that I seriously hope works) is for essentially seamless system-to-system transitions to work. If they can make jumping into another system pretty much instant, that means treating a single grid as its own system becomes possible while keeping the warp-in seamless. The results would be far far more pronounced in jita where pilots are spread around gates and stations than in 0.0 fleet battles where everyone's on the same grid but if they get that to work eventually I'll be so impressed.