Release Planning

Developers should be careful not to run blocking operations in the main thread. This applies to the server components which run scripts, including the BaseApp, CellApp and DbMgr (when using class-customised data types). This is required to ensure the process can respond to its peers and manager process in an appropriate time. In an extreme case, a process that pauses for too long may be considered as dead by other server components and stopped via a signal from BWMachineD.

This time period is different for different processes. For example, by default, if a CellApp pauses for more than 3 seconds worth of inactivity, the CellAppMgr will kill that CellApp process (a common cause is because of an infinite loop in script). For BaseApps, by default, if they are non-responsive for longer than 5 seconds, the BaseAppMgr will kill that BaseApp process. These limits are configurable, see the Server Configuration with bw.xml.

One of the following solutions can be used for large or blocking operations:

It is recommended to review the server warnings during development and to give extra attention to warnings about loading resources in the main thread.

Please also note that operations which take a short time in a test environment might take more time in a production environment.

More information about blocking the main thread can be found in the Server Programming Guide.

Care needs to be taken to avoid situations where Python script makes assumptions that an entity is a Real Entity when it is possible for it to be a Ghost Entity or even an Entity Mailbox. When referring to other entities from script inside an entity, you should never assume that you have a real entity. It is possible for this entity to be a ghost entity or, in some situations, a mailbox to a remote entity.

Generally speaking, the self property will be a real entity but entities passed in as argument or found other ways could be ghost entities or mailboxes. The exception to this is when a method is called on a ghost entity that is not in the entity's def file.

To help identify these problems during development, you should enable the bw.xml option cellApp/treatAllOtherEntitiesAsGhosts and disable the option cellApp/shouldResolveMailBoxes during development. For further details please review Server Programming Guide's section Debugging.

You should validate any input coming from a client, as gamers might try to modify the game client and exploit the server by sending invalid data.

Arguments to defined methods that are typed as PYTHON are inherently risky and should not be used at all for <Exposed> methods that the client can call on the server. This is because a PYTHON argument might contain code which will run on the server.

PYTHON arguments are useful when developing game system prototypes, but they should be converted to another data type such as FIXED_DICT prior to production. You can class-customise FIXED_DICT and wrap the values with another Python object.

Another reason for this restriction is the performance cost of a PYTHON argument. A PYTHON argument requires pickling of the data type for every send and receive. This can be much slower than simply reading and writing primitive types, and also takes up more space on the network stream.

Ideally, developers should review and fix every WARNING, ERROR and CRITICAL message in the server logs. For those messages that cannot be fixed, the developer should have a good understanding of what the message means and why they are occurring.

In order to assist in reducing the noise when reviewing production logs, it is useful to remove any development / debugging log messages from game code / script. This should include any non-essential HACK_MSG, DEBUG_MSG as well as all non crucial print statements from entity scripts.

Multi-core brand name machines are becoming the market standard and are recommended. We recommend taking into account the specific game requirements and the cost effective market solutions when choosing the cluster machine specifications. In general, our customers are currently using 2-4 core machines with 2-4 GB of memory. One CellApp or BaseApp application should be run on each core. Depending on the usage requirement, each machine should be equipped with one or two 1Gb NICs and with enough disk space to host the server resources, the server binaries and system log files. Appropriate free disk space should be available on the disk where the server is installed. Redundant PSU's should be considered to ensure high reliability.

BaseApp processes using the secondary database feature require disk space to write to their SQLite databases. Attention should be paid to ensure that there is enough disk space for these processes.

The machines should have enough memory to avoid swapping as this can lead to poor performance and even processes being shut down. Swap space is still recommended as there is still a chance that these situations can be handled.

For sizing the cluster, please review the section Sizing the Cluster.

Please note that we strongly recommend that the processor affinity for the server processes be left to the operating system to manage.

The hardware required by your database will be heavily dependant on how much load your database is put under. As with most databases the general rule is to use multiple cores, large memory and high performance reliable hard disks. RAID solutions should be considered for the MySQL hard disk. These should be implemented by a MySQL expert.

Setting up the database machine should be done by an experienced MySQL DBA taking into account the performance measurements taken in the test environment.

The server tools components have greater disk requirements than normal cluster machines. You may wish to have a higher performance hard-disk in this machine to handle the load of logging for all server processes. It can also be a good idea to isolate logs to separate partitions. This ensures that if a single logging process runs away and consumes too much disk space, other processes still have some free disk space.

The general strategy used for long term storage of logs is to use logrotate to perform periodic log switches. Older log files should be archived in order to reduce the space requirements.

It is crucial that the master copy of the server binaries and resources and the MySQL database are stored on fault tolerant hard disks which are also backed up regularly.

BigWorld recommends against using NAS related solutions like NFS for the database as these solutions can cause increased network load and therefore degrade performance.

The BigWorld cluster has yet to be performance tested using SAN solutions but we expect these solutions to better fit the BigWorld cluster deployment.

When deploying the BigWorld cluster on separate machines without the usage of a SAN or a NAS solution, a solution should be used to distribute game resources between the cluster components. Any commonly used solution can be used for this issue. Some examples include:

BigWorld Technology supports multiple types of games. Some games include many NPC entities and complex server AI calculations while others include more player characters and less NPCs. The number of spaces can also vary between different implementations. This variety means that the cluster requirements will depend on the specific customer game implementation.

The best solution for sizing the cluster is to measure the BigWorld server instance requirements on a test environment and then to use these results to predict the required cluster hardware. BigWorld Technology allows adding machines to a cluster as well as changing the role of each machine. This allows easy adjustment of the cluster during both the testing and the deployment stages. Please note that extra resources should be allocated to any server to handle load spikes and unexpected events. The chapter Cluster Deployment Examples below contains examples based on real cluster deployments.

For long term deployment environments, BigWorld supports and recommends the usage of one of the following Linux distributions. Be aware that 32 bit Linux distributions are not supported.

While some effort should be put into tweaking server configuration for operational stability, it is also important to consider the load of your system during server startup. Spiky load during server startup may reduce the speed that your cluster will startup and, in the worst case, potentially prevent server startup completely due to overloaded processes.

In order to assist spreading load between server processes and increasing space load time, the following options should be configured in your game's bw.xml file.

While these options will assist in smoothing out your server startup, it is important to take into consideration that the CellAppMgr configuration options apply across the entire lifetime of the server, not just during startup. For this reason it may be useful for your game script to implement a loading phase and an active game phase. During the transition between these two phases the configuration options such as <cellAppMgr/maxLoadingCells> can be modified via a Watcher set request. This also applies to any other load balancing options or other configuration options you may choose to tweak for game startup. See the Server Operations Guide for more details on these options.

MySQL is quite well configured by default, however there are a few options that should be considered for tweaking to ensure optimal performance within your own production environment.

The cluster network should be carefully configured to provide both security and performance.

<root>
    <personality> FantasyDemo </personality>
    
    <parentFile> server/production_defaults.xml </parentFile>

    <externalInterface> 192.168.1.0/24 </externalInterface>
</root>

While BigWorld strives to make BWMachined versions interact with each other as seamlessly as possible, it is recommended to make sure that all the installed versions of BWMachined within your BigWorld network cluster have the same version number. The easiest way to check this is from within WebConsole. Navigate to the Cluster Control module's All Machines page. WebConsole will display differently versioned BWMachined installation in red and will show a warning at the bottom of the screen if multiple BWMachined versions are installed within the same cluster.

Another way to see the same information using command line tools is to use the control_cluster.py command cinfo. Differing or older versions of BWMachined have their version number displayed at the end of each line. For example:

$ control_cluster.py cinfo
dev01      10.40.1.01       0 processes   0% of 1000MHz (7% mem)
dev02      10.40.1.02       6 processes   7% of 2200MHz (77% mem)
dev03      10.40.1.03       0 processes   0% of 2405MHz (2% mem) (v41)
fedora6    10.40.1.06       1 process     0% of 1300MHz (5% mem) (v41)
bwtools    10.40.1.100      1 process     2%, 0% of 2133MHz (30% mem)
5 machines total

In this example we can see that the machines dev03 and fedora6 have an older BWMachined version of 41 than the rest of the cluster.

If you want to quickly check the version of BWMachined on your current host and don't have access to the BigWorld server tools, you can query the installed BWMachined binary directly using either the init.d script as root:

# /etc/init.d/bwmachined2 version
BWMachined (BigWorld 2.0.0 Hybrid64. 11:18:12 Jul  8 2010)
Protocol version 42

or query the installed BWMachined binary using the --version command line argument:

# /opt/bigworld/current/bwmachined/sbin/bwmachined2 --version
BWMachined (BigWorld 2.0.0 Hybrid64. 11:18:12 Jul  8 2010)
Protocol version 42

Newer BWMachined versions are developed to support running older instances of the server, so updating to a newer version of BWMachined will almost always still allow running older servers.