bw logo

Chapter 7. Stress Testing with Bots

The Bots application is a process that can command arbitrary numbers of simulated clients to log into the server and perform activities. It is written in C++, but controlled through Python, like the BigWorld Client.

7.1. The Login Process

The bots are designed to log in a server just like a real client. By default, the Bots process locates a LoginApp by broadcasting a message requesting all LoginApps to respond, and then using the first one to reply. This is the simplest method to log in if there is only one instance of a BigWorld server running.

The script is located in folder bigworld/bin/$MF_CONFIG, and has the following syntax:

bots [-serverName <srv> [-port <port>]]
     [-username <user> [-password <pwd>] [-randomName]] [-scripts]

The options for invoking bots are described in the list below:

  • -serverName <srv>

    Server that the bot clients will login to (overrides the auto-locate method).

  • -port <port>

    Login port. Ignored if Bots is using auto-locate method.

  • -username <user>

    Username that the bot clients will log in as.

  • -password <pwd>

    Password for bot clients' login.

  • -randomName

    Randomises the login name based on the specified <username>.

  • -scripts

    Enables Bots to run scripts. This has a considerable performance cost.

The auto-locate procedure can be overridden by specifying the option -serverName <serverName> on the command line. If both methods fail, then the Bots process requests the server name on stdin (if attached to a terminal). If all these methods fail, then the process starts, but cannot create new bots until the server is set (via the Python method setDefaultServer).

If the option -scripts is specified then Bots looks for the entity definition files under folder <res>/scripts/entity_defs, and for scripts under folder <res>/scripts/bot.

Specific scripts can be created for Bots, but if a client script does not reference the client's C++ modules, then it can be reused. Note that the bots process does not support the BigWorld Client APIs, since it is intended to be lightweight.

In many operation scenarios the entity definition file for Bots could be either missing or different from the definition files for the server. The Bots would use an incorrect message digest for login; and hence the login will fail because LoginApp relies on the digest for verifying entity definition consistency. Previously, only way around this problem is turn the digest checking on the server off in <res>/server/bw.xml's dbMgr options section[11]. This option can also be changed on the fly via a watcher, using WebConsole's ClusterControl module; see ClusterControl). However, this is not a viable option, if you want to use Bots on production servers. You can now set custom MD5 digest in the <res>/server/bw.xml's bots section under option loginMD5Digest. This option can also be modified on the fly through WebConsole.

Bots can now realistically simulate general Internet networking environment by introduce artificial packet loss, network delays, sudden disconnections and frequent relogins. All this simulation can be programmed with Bots' personality script.

7.2. Python Interface

The Bots process was designed to be controlled programmatically via Python, but since Python can be used interactively, it is also a quick way to get some bots into the system and control them.

You can telnet to port 6075 to talk to the bots process, as illustrated below:

$ telnet 10.40.7.12 6075
Welcome to the Bot process
>>> BigWorld.getDefaultServer()
10.40.7.1
>>> BigWorld.setDefaultServer('server2')
>>> BigWorld.setDefaultTag('red')
>>> BigWorld.addBotsSlowly(500, 0.1)
>>> BigWorld.setDefaultTag('blue')
>>> BigWorld.addBotsSlowly(500, 0.1)
>>> BigWorld.addBotsWithName( [('Bot_01', '01'), ('Bot_02', '02')] )

Example of use of Python interface to Bots

7.2.1. Python Controller (bot_op.py)

BigWorld provides a Python program to simplify the management of large numbers of bots.

It automatically starts bot processes as needed, load balancing (based on CPU load) on all available machines not running any other BigWorld components.

The script is located in folder bigworld/tools/server and has the following syntax:

python bot_op.py [add [<number_of_bots>]]
                 [del [<number_of_bots>]]
                 [movement <controller_type> <controller_data> [<bot_tag>]]
                 [set (<watcher_name> <value>)+]
                 [run [<command>]]
                 [addprocs [<num_of_procs>]]
                 -u [username or uid]

The options for invoking bot_op.py are described in the list below:

  • add [<number_of_bots>]

    Adds the specified number of bots to the system. If <number_of_bots> is not specified, then one bots is added.

    Bots are added 16 at a time, up to a maximum of 256 per bots process.

    The script waits 1 second after each add, to check it a bot was effectively added. When all bots processes are full, it creates a new process on an available machine that is using less than 80% CPU (preferably one that is already running some bots processes).

  • addprocs [<num_of_procs>]

    Spawns the given number of bots processes on each machine that is considered eligible for running them, or 1 on each if no number is given.

  • del [<number_of_bots>]

    Deletes the specified number of bots from the system. If <number_of_bots> is not specified, then it defaults to 1. The number of bots will be ceiling clamped to the actual number of bots on the server.

  • movement <controller_type> <controller_data> [<bot_tag>]

    Changes the bot's movement controller. If option <bot_tag> is specified, then only bots matching it will be changed. Otherwise, all bots will be changed.

  • set (<watcher_name> <value>)+

    Sets the watcher to specified value on all bot applications.

  • run [<command>]

    Runs the specified Python command on all bots. If <command> is not specified, then it is read from stdin.

  • -u [username|uid]

    Manually specify the UID of the server that the script will take action on.

7.2.2. Methods and Attributes

The list below describes the bots' attributes and methods on ClientApp and BigWorld:

  • ClientApp Attributes

    • id (int) — Bot ID (read-only).

    • spaceID(int) — ID of the space the bot is currently in. 0 means not in any space (read-only).

    • loginName(string) — Bot's login name.

    • loginPassword(string) — Bot's login password.

    • tag (string) — Bot's user-specified tag. Allows the control of partial sets of bots.

    • speed (float) — Bot's speed.

    • position (Vector3) — Bot's position.

    • yaw (float) — Bot's yaw.

    • pitch (float) — Bot's pitch.

    • roll (float) — Bot's roll.

    • isOnline(bool) — Indicates whether the bot is connected to a server (read-only).

    • isDestroyed(bool) — Indicates whether the bot has been destroyed (read-only).

    • entities(PyObject)

      Dictionary-like attribute containing the list of entities that are in the simulated client's current AOI.

  • ClientApp Methods

    • logOn()

      Initiate log on process for the simulated client to connect to a BW server.

    • logOff()

      Make a simulate client disconnect from server gracefully. If the simulated client is not online, it will do nothing.

    • dropConnection()

      Make a simulate client drop the connection with the server. If the simulated client is not online, it will do nothing.

    • setConnectionLossRatio( float lossRatio )

      Set up network packet loss ratio for simulating unstable network environment. Range between 0.0 and 1.0. 0.0 means no packet loss. 1.0 mean 100 percent packet loss.

    • setConnectionLatency( float minLatency, float maxLatency )

      Set up simulated network data latency (in milliseconds).

    • moveTo( Math.Vector3 position )

      Set next destination position for the player avatar of the simulated client.

    • snapTo( Math.Vector3 position )

      Set the player avatar position for the simulated client.

    • stop()

      Stop the player avatar of the simulated client moving.

    • faceTowards( Math.Vector3 direction )

      Set the direction of avatar of the simulated client.

    • addTimer()

      Add a timer for the bot client. This function returns an integer timer id; otherwise it returns -1.

    • delTimer( int timerID )

      deletes an active timer corresponding the timerID.

    • onTick()

      This optional function is invoked every game tick, should it be defined in the player Avatar script.

  • BigWorld Attributes

    • bots

      Dictionary-like attribute containing the list of simulated clients. The key is the player's ID, and value is an instance of ClientApp.

    • bots [<player_id>]. entities

      Dictionary-like attribute containing the list of entities created under the simulated client specified by <player_id>. The key is the entity's ID, and the value is the reference to the entity.

      In non-script bots, only the player entity is created and stored in this attribute.

    • bots [<player_id>].entities[<ent_id>].clientApp

      Attribute referring to the ClientApp that owns the entity. An entity can access its owner ClientApp via this attribute. BigWorld.bots[<player_id>].entities[<ent_id>].clientApp equals to BigWorld.bots[<player_id>].

  • BigWorld Methods

    • setMovementController(string type, string data) — Context: ClientApp

      Sets a new movement controller for the bot. On failure, the controller is left unchanged. Returns true on success, false on failure.

    • setLoginMD5Digest(string MD5Digest) — Context: BigWorld

      Set the 32 character long MD5Digest (in hex readable form) for server login. If the input string length is not exactly 32 character, the MD5 digest will be reset to empty.

    • addBots( int numBots ) — Context: BigWorld

      Immediately adds the specified number of simulated clients to the application. If too many bots are added at once, then some of them may time out, depending on how long it takes to log in all of them.

    • addBotsSlowly(int numBots,float delayBetweenAdd,int groupSize=1) — Context: BigWorld

      Adds bots to the system in groups of groupSize, and add a delay between each group. This method returns immediately, adding the bots in the background, and is the best way to add bots to a system.

    • addBotsWithName( PyObject loginInfo )

      Immediately add a set of bots with login name and login password specified from loginInfo. The structure of the loginInfo is a list of tuples which consist of two strings. The first string is the login name and the second one is login password. In the event of incorrect loginInfo, a Python exception will be thrown.

    • delBots( int numBots) — Context: BigWorld

      Deletes the specified number of simulated clients from the application. The method onClientAppDestroy() of the personality module is called.

    • getDefaultServer() — Context: BigWorld

      Returns the server name that the bots try to log in.

    • setDefaultServer( string serverName ) — Context: BigWorld

      Sets the server name that the bots try to log in.

    • onTick() — Context: BWPersonality

      This optional function is invoked every game tick, should it be defined in the personality module in <res>/server/bw.xml.

    • onLoseConnection( int playerID ) — Context: BWPersonality

      If this callback function is defined in the personality module specified in file <res>/server/bw.xml, it will be invoked when a simulated client loses its connection with a server abnormally. The personality script can decide whether the client should be destroyed or remain dormant for later reactivation by return True or False respectively. NOTE: if this callback is not defined, bots that lose connection will be automatically destroyed.

    • onClientAppDestroy( int playerID ) — Context: BWPersonality

      If defined in the personality module specified in file <res>/server/bw.xml, this callback method is called for each destroyed bot client. The argument is the ID of the deleted player entity. A bot client may be destroyed due to either loss of server connection or by BigWorld.delBots() method call. If a bot client is destroyed by BigWorld.delBots() method call, it will be disconnected from the server gracefully. We recommend client should execute additional log off procedure in this onClientAppDestroy callback function. An example is as following:

      def onClientAppDestroyed( playerID ):
        bot = BigWorld.bots[playerID]
        if bot.isOnline:
          bot.entities[playerID].base.logOff()

      The implementation above assumes that the player entity on the server has an exposed base method logOff(), which will then destroy the player cell and base entities on the server. The script must to be placed under the script folder <res>/scripts/bot.

7.3. Controlling Movement

The default movement controller Patrol moves the bots along a graph read from a file.

The name of the file containing the movement graph is specified in file <res>/server/ bw.xml, on section <bots>, by the configuration option controllerData.

<root>
  ...
  <bots>
    <controllerData>   server/bots/test.bwp  </controllerData>
    <controllerType>   Patrol                </controllerType>
    <password>         passwd                </password>
    <username>         Bot                   </username>
    <shouldLog>        true                  </shouldLog>
    <serverName>                            </serverName>
    <randomName>       false                 </randomName>
    <shouldUseScripts> false                 </shouldUseScripts>
    <port>             0                     </port>
    <publicKey>        loginapp.pubkey       </publicKey>
  </bots>
  ...
</root>

Example file <res>/server/bw.xml

The file name might have the suffix random, in which case the start position of the bot is chosen randomly from the set of node positions in the graph.

The format specification for the movement graph file is illustrated below:

<controller>

  <nodeDefaults>
    ?NodeProperties
  </nodeDefaults>

  <nodes>
    *<node>
         <pos>   integer integer integer  </pos>
        ?<name>  string_node_name         </name>
        ?NodeProperties
        *<edges>
           <edge>  string_node_name  </edge>
         </edges>
     </node>
  </nodes>

</controller>

Grammar of movement graph files

The list below describes the tags in the movement graph file:

  • edge (section nodes/node/edges)

    Name of one of the nodes that this node borders. Must be the same value specified for a name section in the file.

  • edges (section nodes/node)

    Tag specifying all the nodes that this node borders.

  • name (section nodes/node)

    Name used to refer to the node. If this tag is not specified for a node, then you can refer to it by an integer sequential number, with the first node having an index of zero.

  • node (section nodes)

    Tag for section specifying a specific node in the graph.

  • nodeDefaults

    Tag for section specifying default values for all nodes. These values can be overridden on a per-node basis in section nodes/node.

  • NodeProperties (section NodeDefaults and nodes/node)

    For details, see NodeProperties Section.

  • nodes

    Tag for section specifying all the nodes in the graph.

  • pos (section nodes/node)

    XYZ position of the centre of the node. The node is the area around pos, extending for n metres around it (n being the value of NodeProperties' tag radius).

7.3.1. NodeProperties Section

The format specification for the NodeProperties section is illustrated below:

?<minStay>   float  </minStay>
?<maxStay>   float  </maxStay>
?<radius>    float  </radius>
?<minSpeed>  float  </minSpeed>
?<maxSpeed>  float  </maxSpeed>

Grammar of NodeProperties section

The list below describes the tags in the NodeProperties section:

  • maxSpeed

    Maximum speed of the bot.

  • maxStay

    Maximum number of seconds that the bot should stay in the node.

  • minSpeed

    Minimum speed of the bot.

  • minStay

    Minimum number of seconds that the bot should stay in the node.

  • radius

    Number of metres around the node's pos location to be considered as the node. When a bot is travelling between two nodes, it chooses a random point of arrival in the destination node. This way, the bots do not follow the exact same line. The point is chosen to lie within the radius of the destination node.

An example movement graph file is displayed below:

<patrolGraph>
  <nodeDefaults>
    <minStay>   5  </minStay>
    <maxStay>  10  </maxStay>
    <radius>   20  </radius>
  </nodeDefaults>
  <nodes>
    <node>
      <name>      Town1      </name>
      <minStay>   7          </minStay>
      <pos>       5000 0 0   </pos>
      <edges><edge>  Town2        </edge></edges>
    </node>
    <node>
      <name>      Town2        </name>
      <pos>       5000 0 5000  </pos>
    </node>
  </nodes>
</patrolGraph>

Example movement graph file

7.4. Extending Bots

7.4.1. Creating New Movement Controllers

The default movement controller is Patrol, but you might want to create one that better represents the expected traffic patterns and entity distributions of your game, or to stress test other aspects of the system.

To create a new movement Controller, derive its class from the MovementController class. Also derive a class from the MovementFactory class. The factory is used to parse the Controller data argument, and instantiate a new movement controller, while the movement controller moves the bot around.

The fragments below list the relevant methods in the base classes:

  • Class MovementController

    bool nextStep( float speed, float dTime, Vector3& position, Direction3D& direction)
  • Class MovementFactory

    MovementFactory(const char* name)
    
    MovementController* create(
      const std::string&  data,
      const Vector3&      startPosition)

7.5. Miscellaneous Bots Issues

7.5.1. Running out of File Descriptors

As the Bots process is designed to test a large number of client connections to a BigWorld server, it is possible that in doing so the Bots process may exceed the maximum number of open file descriptors allowed by the system configuration.

This issue will be identifiable by newly created bots not communicating with the LoginApp, as well as log output from the Bots process similar to the following:

Bots   ERROR   Endpoint::findDefaultInterface: if_nameindex returned NULL (Too many open files)
Bots   ERROR   NetworkInterface::recreateListeningSocket: Couldn't determine ip addr of default interface

If you encounter this issue, adjust the maxOpenFileDescriptors on General Configuration Options to increase the number of allowed open file descriptors.



[11] For details check allowEmptyDigest option in DBMgr Configuration Options.