Jump to: navigation, search


What is the strategy HeroEngine uses to scale?

Each Area is governed by a seperate server process called an Area Server. The Area Server processes are dynamically allocated to physical hardware servers based on a load-balancing scheme (least-burdened heuristic).

An Area Server is a monolithic process with a single-threaded HeroScript Virtual Machine. Area Servers can communicate with any other Area Server through asynchrnous remote function calls from HeroScript.

Because any number of Area Server processes can be spun-up, utilizing as much hardware as available, the broad-phase scalability is limited mostly by switching fabric bandwidth and the database back-end. The utilization of these resources is contingent on your particular game implementation, so care must be taken during the design phase. Further, the capacity of the switching fabric and the database store depends on your datacenter build-out.

Narrow-phase scalability is constrainted by the fact that a given Area process can become over-burdened and utilize too much CPU during processing game events, character movement packets, game mechanics, AI and so forth.

Particularly taxing game mechanics can be engineered (or re-engineered after they are discovered to be a bottleneck) into "services" by use of System Areas. A System Area is a process, created from HeroScript, that manages any type of processing required asyncronously to any particular geogrpahical area. In this way, it provides a "service" to the game system. For example, the management of guids, or quest assignment, perhaps even combat resolution. Because System Areas can be instanced many times to account for load, and each process can run on any physical hardware available, this provides a high-degree of scalability.

Like all systems that trade off communcation for parallelism, there is ultimately a limit at which the communication processing itself is more burdensome than the processing it was attempting to parallelize. This is also the fundemental problem is dynamic load balancing systems which, in their effort to create greater and greater parallelism, simply explode the resources required to manage the proxy traffic.

Therefore careful game and system design will always be a big part of any scalability strategy. HeroEngine can provide a solid foundation and toolset for this, but it is still important to realize that the game-specific implementations will be key to effective scaling.

How does the game design affect the scalablity?

It is critical to coordinate game design with the techncial realities of massive numbers of players and their behavior. How many users can participate in a given Area will be largely determined by the game mechanics that exist in your game design. For example, extremely complex combat logic in a high-frequency combat-heavy game design will necessitate the splitting of the world into smaller areas so that the load can be properly balanced.

An single Area can easily handled a thousand users in one game implementation, but maybe only handle a couple hundred in another. There is no way to determine this ahead of time, since HeroEngine is entirely generic and doesn't restrict how you might use it. If you write inefficient scripts, your results will be worse than if you write efficient ones. If you are smart about how much AI procesing creatures perform, then you will be able to handle more creatures in a given area than if you didn't. And so on.

The best way to deal with this is to identify the key aspects of your game design that are absolute requirements and which are flexible. Make sure to leave room for trade-offs, because in MMOs there are hard-and-fast physical limitations. Server hardware is only so fast at any given generation, database systems can process only so many transactions, etc. And remember that the "illusion" of doing a lot is better than actually doing a lot. This is, of course, the Black Art of MMO design. Our team will be glad to assist you in any planning to help you through this.

Can Areas automatically load balance by partitioning users into seperate processes?

Currently HeroEngine does not do this sort of dynamic load balancing.

It is attractive to think that a dynamic load balancing scheme will eliminate the need to worry about how many users congregate into a specific area. Having it dyanmic, the thinking goes, will make it so you don't have to try and design-manage or predict how users will behave.

However, the reality is that any scheme of sub-dividing the processing of users carries it's own resource burden in the form of proxy traffic. HeroEngine connects areas together through fixed feature points, where proxying is carefully controlled. But in a dynamic load balancing scheme the divisions happen arbitrarily and change constantly based on the heuristic used.

The problem with this is that the as the dyanmic cells shrink in size to achieve greater parallelism, the proxy traffic explodes. In a system with a syncronhous proxy mechanism (such as a stackless Phyton implementation), this can quickly lead to diminishing returns and ultimately the cells can no longer subdivide without it being a net negative impact on server performance.

Because of this, the "fools gold" of thinking a dynamic load-balancing system will save you from using careful world and game design methodologies could actually lead to disaster!

In either case, you need to carefully consider your game mechanics and world design to manage maximum player population densities. If you have regions of high desnity, then break them into smaller areas. Reduce the number of connections to prevent too much proxy burden, etc.

What if an Area server gets too many players anyway?

A: If a Area Server begins to use more resources than available on the physical hardware, then the system will attempt to migrate that process to a less burdened server (if available). However, if the process ends up alone on a server and is still taxing it beyond it's capabilities then no further automatic resolution can occur.

At this point, it would be wise to redesign the Area distribution. Break the offending area into more pieces, each as separate areas and publish this new configuration. One way to do this is to copy an area into another, then delete parts from each side so that it is effectively split in two. Then create the appropriate junctions and so forth. Some work will have to be done to adjust for players logging in so they go to the correct area (for those who logged out pre-split) but this can be taken into account from HeroScript events.

It is also possible that instancing might be appropriate for your game design. Simply have your area exist as multiple instances and cap the maximum number in each. You can even have it spin up more instances if those fill up automatically. This can all be done from HeroScript, including whatever mechanisms, heuristics or user interface you want to provide for picking what instance a player goes into. Instancing is a tremendously effective way of dealing with extremely over-burdened areas. You simply have to take some care to make sure members of a group (if your game has groups) all move into the same area together.

NOTE: A future feature of HE, called Vertical Scaling, is envisioned to allow for scalability even in this situation. The idea is that the Seamless World proxy capability could be leveraged to allow for the workload of a given Area process to be broken into multiple processes that could be on the same server or even different servers.

What is the maximum capacity of a single area server?

This depends on the hardware and the CPU budget for a user. For example, you can define the CPU budget on a per user basis to be 10 megahertz. If you have hardware that runs at 3 gigahertz, approximately 300 users could exist on the same area server. However, some of the scripting functions such as broadcast chat do not all scale linearly, so the actual capacity of the server would be somewhat less. If you have a game implementation that offloads some of the CPU budget to other servers, then more users will fit.

How does the client scale in respect to graphics?

HeroEngine supports the concept of managing many shader levels (all the way down to fixed function) with all key features able to be "nerfed" back by the end-user to match their CPU/GPU capabilities.

Going forward, we already support multiple cores in that some key capabilities run in separate threads (for example, physics).

See also

Personal tools