Table of Contents
Periodically, a complete copy of each cell entity is backed up on
the base entity. Only cell entities with an associated base entity are
fault tolerant. The CellApp backup period specifies how often cell
entities are backed up to their base entities, and is specified in the
bw.xml option
<cellApp/backupPeriod>.
Should a CellApp process become unavailable, the real entities located on the cells residing on that process will be restored by their corresponding base entities to other CellApps. The state of the cell data of the restored cell entities is the same state as was given from the most recent backup from the cell entity to the base entity.
The CellApp restoration process typically follows these steps:
-
A CellApp process becomes unavailable.
-
Base entities that have cell entities on the now unavailable CellApp process restore their corresponding real entities to other CellApps.
-
Restored cell entities have the
onRestore() callback called on them. Because the restored cell data is taken from the last time the cell entity backed up to the base, this copy can be up to twice the backup period. This callback should check that the entity's properties are in a consistent state. -
For player cell entities, their corresponding client-side player entities have the
onRestore() callback called on them.
The callback onRestore() is invoked on
the cell entity to inform it that it is being restored.
The code fragment below illustrates its implementation on the cell entity:
class SomeEntity( BigWorld.Entity ):
...
def __init__( self ):
# set up initial property values
...
def onRestore( self ):
# check that property values are consistent, and
# perform any cleanups that need to occur
...Example file
<res>/scripts/cell/SomeEntity.py
The above diagram shows a space divided into three cells, the top cell residing on CellApp 4, the bottom-left cell residing on CellApp 3 (not shown) and the bottom-right cell residing on CellApp 8 (not shown). In the cell that CellApp 4 has in this space, it has the cell entities with IDs 4156, 5712 and 2114, all within the same spatial region. The entities 4156 and 2114 have corresponding base entities that reside on BaseApp 1 and 2 respectively. Entity 5712 does not have a base entity, and is a cell-only entity.
Cell entities back up their data to their corresponding base
entities. So in this example, entities 4156 and 2114 send a copy of
themselves to their corresponding base entities. The rate at which they
do this can be configured by changing the backup period, using the
bw.xml option
<cellApp/backupPeriod> (see the CellApp Configuration Options for more details). When
a single CellApp goes down, cell entities are restored from the backup
data sent to their base entities. Any cell entities that do not have
corresponding base entities will not have been backed up, and so will
not be restored.
If CellApp 4 was to go down, the cell entities for 4156 and 2114 will be restored onto other CellApps from their base entities. Entity 5712 will not be restored, as it has no corresponding base entity.
It is important to note that when restoring cell entities, the
cell data that is used will be whatever data was present at the time of
the last cell entity backup. Thus, any modification to the cell entity
data since the last backup is lost, and the state of that cell entity
may be inconsistent when it is restored. For example, a backup of a cell
entity may be made in the middle of a multi-step transaction. The script
callback
Entity.onRestore() can
be used to check the state of outstanding transactions, and the decision
to either roll them back or continue them can be made in script.
With BaseApp fault tolerance, BaseApps back up the base entity data and cell backup data of all their base entities to other BaseApp processes periodically.
Should the primary BaseApp become unavailable, then all its entities
are restored from the backup process. In this case, the BaseApp invokes
the callback onRestore on the base or proxy
entity, in a process similar to the one for CellApp restoration. This
callback should ensure that all properties on the entity are in a
consistent state.
It should be noted that when a BaseApp fails unexpectedly, entities
that were on that BaseApp before the death might be restored to different
BaseApps. Care should be taken when writing scripts, to avoid assumptions
that a Base entity is local. When performing an entity backup, Base entity
properties are streamed using their description in the definition file (if
there is one). For properties that are not specified in the definition
file, these are pickled. Each entity is backed up individually, so if two
entities refer to the same object, on restoration they will likely each
have a copy of that object. An option is available to prevent properties
not specified in the definition file from being backed up. To disable
backing up undefined properties set the bw.xml option
<baseApp/backUpUndefinedProperties> to
false. For more information on this option refer to the
Server Operations Guide section BaseApp Configuration Options.
For more details regarding Fault Tolerance, see the document Server Operations Guide's chapter Fault Tolerance, and the document Server Overview's section Server Components → BaseApp → Fault Tolerance.