Each player must be able to query the status of another player by name:

Using BigWorld.lookUpBaseByName() causes a query to DBMgr (which causes a read on the primary database), while sufficient for many scenarios (and empirically works in many released BigWorld-based games), introduces a potential bottleneck. The discussion below outlines a design for a distributed mapping of player names to player mailboxes which effectively offloads this load to BaseApp game script, which can be scaled up by adding more BaseApps.

The idea is to have multiple PlayerRegistry Base entities that contain a distributed mapping of player names to player mailboxes. These PlayerRegistry entities have no geospatial representation, they exist only as a system service, and so generate no load with respect to AoI updates.

Each BaseApp has a corresponding PlayerRegistry entity - this spreads the PlayerRegistry entities out and protects against BaseApp failures. Having more than one PlayerRegistry entity per BaseApp does not add any additional redundancy benefit.

PlayerRegistry entity instances register themselves globally. Globally registered bases have their mailboxes registered under a string key in a global bases mapping that is synchronised across every BaseApp. Player names are hashed against the known number of player registries, and a particular PlayerRegistry instance is located via the Global Bases mechanism (see Global Bases).

When player entities are created, they add themselves to the distributed registry by hashing their own name to the appropriate PlayerRegistry entity, and registering their base mailbox with that PlayerRegistry entity. On logout, they contact that same PlayerRegistry to notify it of the logout, and this results in the removal of the mapping between that player name and that player mailbox. A scheme for rebalancing the player registry entries can be implemented which re-balances the entries across the PlayerRegistry entities when a new PlayerRegistry is added or removed, such that the hash scheme remains consistent.

Queries for a particular player name are done by first hashing the player name to be looked up to the appropriate PlayerRegistry, and then querying one of the multiple PlayerRegistry entities via a remote method, and a callback remote method with a mailbox. Requests for player lookup are asynchronous, and caller entities implement a callback method that is called back when the lookup is complete.

Each PlayerRegistry needs a persistent mailbox list for fault tolerance purposes, so that the registry is restored to another BaseApp along with the PlayerRegistry entity if the BaseApp it formerly resides on fails. In this case, it is likely that it will be restored to another BaseApp which already has its own PlayerRegistry, so re-balancing should be done and then the restored PlayerRegistry should be destroyed.

This system can be scaled up by increasing the number of BaseApps to handle queries. Tiered request schemes could also be used to avoid large numbers of globally registered base entities becoming a bottleneck.

Each player maintains a list of other players that they can use for the following purposes:

Assume that friendship relation is symmetric, so that if A is on the friend list of B, then B is on the friend list of A. A friends list can be implemented as an ARRAY of FIXED_DICT consisting of a STRING name property, and the MAILBOX of the player (or None if offline), and a UINT8 Boolean flag hasResponded indicating that this player has responded to our request to add that player as a friend.

Each player must have the ability to send mail to other players. This mail includes some text and optionally in-game items.

The scalability of SMTP/IMAP mail servers can be leveraged here. Note that these game mail servers are completely internal to the game - no public access would be allowed (though this would be up to the game design).

Each player has an associated email address. BaseApps can query IMAP servers asynchronously using a TCP socket registered with BigWorld, without blocking game script. Python has good support for communication with IMAP over a socket (see the chapter Non-Blocking Socket I/O Using Mercury)

Items can be gifted using special attachments or special email headers, depending on the item system used. Item data would never be directly sent via email, instead, gifted items over email would be held in escrow, as with AoI-based player item trading. See Inventory and Item trading below.

Assume a game inventory system with the following features:

Store a fixed amount of inventory item slots per-player on the player entity to limit the amount of inventory data that is associated with player inventory.

Some popular MMOs have the concept of banks where players must be in a specific area to access items stored at the bank. This could be a separate entity that is loaded on request when a player is accessing their bank, and then destroyed once they leave the bank. The capacity of the on-player inventory and the bank inventory could be tuned to optimise database load.

With the on-player inventory, this can be stored as a BigWorld ARRAY of item descriptors. Item descriptors themselves would be persisted as a BigWorld FIXED_DICT, but could be class-customised when loaded from the database so that items are represented in script as an arbitrary object type.

Player inventory changes are expected to be frequent. Per-element changes in a BigWorld ARRAY are propagated to the client with a description of the change path to that element and the new element value (i.e. the entire array is not sent from the server to the client each time an element is changed).

If each time the inventory is changed, the entity wrote its state out to the secondary database, then a bottleneck can occur, as this operation is expected to be frequent.

In this case, we rely on the fault tolerance mechanism for ensuring against data loss. This works by periodically saving out the state of the entity to another process. That other process is responsible for restoring the entity in the event of a process failure. For example, cell entity data is backed up to the corresponding base entity's BaseApp, and base entities are backed up to other BaseApps. This is the first level of fault tolerance, and the frequency of backups can be configured.

There is also a second level of fault tolerance, which is the periodic archiving of the base and cell entity state to the secondary databases. The frequency of this can similarly tuned to achieve optimum BaseApp load.

However, for important changes to the item inventory, for example, a quest item, game script can request a write to the secondary database and have that confirmed via an onWriteToDB() callback. For trading transactions between two players, see below.

Player entities in the same spatial vicinity must be able to negotiate trade of items that they own.

Each player makes an offer to each other, placing their offered items in escrow. Once both players accept the opposing player's offer, the trade succeeds and the items are traded. If one player cancels the trade, all offered items are returned to their respective players.

Item trading transactions must not result in duplicate items or item loss.

BigWorld can readily supply the base mailboxes of any player entity in a player's AoI. Otherwise, if trading with a specific person not in the player AoI, a player look-up is required.

Escrow entities are created for the lifetime of a transaction, and hold mailboxes to the two entities bartering. Escrow entities persist to the database. Trading consists of two stages, the negotiation stage and the transfer stage. Escrow entities are created on the least loaded BaseApp.

The negotiation stage is a series of offer operations made from a player entity to an Escrow entity, each of which is then forwarded to the opposing player entity.

If the server stops in the middle of a transaction, the Escrow entity has enough persistent information to cancel itself on restore and return items back to their owning player entities.

Player entities on the server offer items to the other player (in response to GUI interactions from their player client) in the form of remote method requests to the Escrow entity. In doing so, they transfer these items from their inventory to a special holding area on the player entity on the server. This holding area is not accessible for any other purpose by the player's client, other than to remove the item from the current offer, which moves that item back into their inventory.

Each transfer to/from their player inventory to the trade holding area results in:

If, for some reason (temporary or otherwise), the database write fails, the entire trade is cancelled, and the items are returned to the players via remote method calls, which are acknowledged via a remote method call by the players back to the Escrow entity. When the Escrow entity receives acknowledgements from the two players that the trade has been cancelled, it deletes itself from the database.

Each player can signal to the Escrow entity that it is willing to accept the trade as it stands. Once the Escrow entity receives positive notification for both parties, it transfers ownership of the items to the corresponding opposing players by signalling to the player the item data that they have traded.

Total database writes: 2 for each offer made, and at least 2 offers are made. 3 writes for the transfer stage.

This illustrates that trading can potentially be an expensive operation in terms of writes to disk. However, all the writes are distributed amongst the entities involved, and most would be written to the secondary database. Only one of the database writes, when the Escrow entity is destroyed, results in the primary database being utilised, in order to remove that Escrow entity from persistent storage.

Note that each participating entity in a trading transaction is not required to be on the same process. This scales well because there can be an arbitrary number of BaseApps, and players and Escrow entities would be uniformly distributed amongst the BaseApps. Recall that while CellApps have player distributions that map to where they are spatially, base entities on BaseApps do not have this spatial relation.

There is a cost to the primary database associated with the creation and destruction of each Escrow entity. This design can be improved by consolidating the escrow operations to target a pre-existing EscrowManager entity rather than creating and destroying Escrow entities. A similar scheme could be implemented to the PlayerRegistry entities by having an EscrowManager entity per BaseApp. Trading entities would nominate and agree on a random EscrowManager to use for their trading transaction.