Table of Contents
This section describes some of the debugging features of the BigWorld client.
Debugging is an important part of any development process, and the client has many features that facilitate finding bugs early, in-game debugging, remote debugging, and the immediate application of changes without restarting.
Many debugging features are accessed via a specially defined debug key. By default, the debug maps to the grave (tilde) key on the keyboard, however this can be re-configured by editing the engine configuration XML file.
BigWorld provides a number of features that aid in runtime debugging, such as the debug menu and the telnet Python service.
To remove these features from a final release build of the client, a number of compile guards exist. These are collectively controlled by the following define located in src/lib/ cstdmf/config.hpp.
#define CONSUMER_CLIENT_BUILD 0
If CONSUMER_CLIENT_BUILD is 0, then the development features will be compiled.
Individual features are encapsulated in their own individual compile guards, which can collectively be switched using the above define. Individual features may also be enabled, by setting the corresponding force enable define to a non-zero value.
This is illustrated in the example below:
#define CONSUMER_CLIENT_BUILD 0
#define FORCE_ENABLE_DEBUG_KEY_HANDLER 0
#define ENABLE_DEBUG_KEY_HANDLER (!CONSUMER_CLIENT_BUILD \
|| FORCE_ENABLE_DEBUG_KEY_HANDLER)
A watcher is an object that wraps a variable or function result in a program, and turns it into a string.
Watchers can be read-write or read-only, and can be viewed in various ways. Their hierarchy is usually organised by functionality. There are also watchers that can dynamically match changing data, such as the contents of a sequence or map.
The simplest watcher type is the DirectoryWatcher, which allows the creation of a directory of other watchers (including DirectoryWatchers). This is one way in which the hierarchy can be built up. For example, a float value for the amount of rain drawn by the rain system in the BigWorld client is specified at 'Client Settings/Rain/ amount'. This uses three DirectoryWatchers: the root directory, the Client Settings directory, and the Rain directory.
There are also templatised watchers for any data type that can be streamed onto a std::stream object. These come in two flavours, as DataWatchers of variables, and as MemberWatchers of the result (or sole argument) of member functions of a class.
The DataWatchers and MemberWatchers can also take a 'base object' argument, which the data they point to is considered a member of. This is very useful when combined with more complicated types of watchers, such as the SequenceWatcher.
The SequenceWatcher and MapWatcher appear to be the same as a DirectoryWatcher, but they watch any class that supports the STL sequence or mapping methods. They have just one client watcher, but they present as many children as there are elements of their wrapped container. The child watcher is called with its 'base object' set to the element of the container. To handle pointers in these containers, there is also the BaseDereferenceWatcher, which presents no extra hierarchy level, but rather dereferences its base object and calls a sole child watcher with that value. The children can be named either by a static list, or by making a request for a certain value in the child watcher (such as name).
From the types above, it can be seen that a watcher hierarchy can be set up to both expose simple settings and to track complex structures.
The simplest way to make a watcher is to use the MF_WATCH macro. This macro takes the path name for the watcher, and a variable or member function to get/set its value. It expands to code that adds the appropriate watcher.
To change the parsing or display of a value (as a string), it is not necessary to write a new watcher. Instead, you can use member functions that get and set the value as a string. This is how the 'Client Settings/Time of day' value works — there are simple accessors for timeOfDayAsString, which format the time of day as 'hours:minutes' and parse it back from the same format. Setting a watcher can also be used to trigger a command — for example, the server is also notified whenever the time of day is set (for debugging purposes only).
To track complex structures, the appropriate watcher must be created directly, and inserted into the watcher hierarchy. Classes that participate in such an arrangement often implement a getWatcher() static method that returns a watcher object that works when the 'base object' is set to an instance of their class. The same watcher can be shared by all instances of the class, and is usually of type DirectoryWatcher of DataWatchers and MemberWatchers.
This is done on the client with the map of entities maintained by the Entity Manager. The entities are named by their ID.
Most of the C++ state of the client is also available through explicitly added watchers, as they were needed for some stage of the debugging already.
The watcher console provides access to the watcher system from within the game. It can be brought up at any time, and overlays the game screen with a listing of the current watcher directory.
Directory-like entries can be followed down, and the value of any writable watcher can be set by selecting it, pressing Enter, then typing in a new value.
There are also keys for adjusting numerical values by 1, 10, 100, or 1000.
The watcher system is exported from the client using a very simple UDP-based watcher message protocol. The server uses this protocol extensively for its own debugging needs.
This is not related to Mercury, although it can be (and currently is) linked to its input loop. When the client starts up, it broadcasts a watcher nub notification packet, advertising the port its watcher nub is listening for requests on (it sends a similar message when it shuts down cleanly).
To access the watcher values, a number of tools may be used, but by far the most useful is a UNIX daemon called 'watcher'. This daemon listens for any broadcast watcher packets and adds or removes them from an internal list.
The other side of the daemon presents an HTTP interface to all the watcher nubs it knows. It inserts the name of the watcher nub (contained in the registration message, e.g., 'client of John', 'cell14', etc...) as the top level of a watcher super-hierarchy. It presents the individual watcher trees as branches of this top directory. The path in the request URL is translated into the watcher value request path. This interface can get and set values.
So by connecting to this daemon with any web browser, all the running clients on the local network can be seen, and then investigated or manipulated, right down to changing the value of a Python variable in an entity. Python variables can be set to arbitrary Python, which is executed on the client to find its value. The client can also send its watcher nub notification to another external address if so desired, so even remote clients can be accessed in this way.
The memory tracking system can be used to determine the amount of memory allocated to a class or module. The ResourceCounters class is the primary class used to perform the tracking.
This is a simple class that tracks the memory usage of a component based on a description string (the component allocating/deallocating the memory) and a memory pool enumeration (the memory pool to be used, i.e., system memory, managed video memory, or default video memory).
The two main worker methods of this class are add and subtract, which respectively track memory allocations and deallocations using the description-pool.
The ResourceCounters class maintains a map of the components it is tracking. Rather than calling the methods of ResourceCounters directly, two macros are exposed. These macros are defined away to nothing when memory tracking is disabled:
#define RESOURCE_COUNTER_ADD(DESCRIPTION_POOL, AMOUNT) / ResourceCounters::instance().add(DESCRIPTION_POOL, (size_t)(AMOUNT)); #define RESOURCE_COUNTER_SUB(DESCRIPTION_POOL, AMOUNT) / ResourceCounters::instance().subtract(DESCRIPTION_POOL, (size_t)(AMOUNT));
ResourceCounters also exposes toString-style methods that are used by the realtime resource-monitoring console to display the current memory usage statistics.
With respect to the memory tracking system, there are two basic types of memory allocation:
-
DirectX memory allocation
All DirectX resources are COM objects. COM objects use reference counting to manage their sharing among multiple objects. The Moo library provides the wrapper class ComObjectWrap for dealing with these objects.
Inside BigWorld, there are two reference types used to refer to DirectX objects: plain points (e.g., DX::Texture*) and ComObjectWrap pointers. For DirectX objects, the memory tracking is performed purely through the ComObjectWrap class. Therefore, any DirectX resource that needs its memory tracked must be refactored to use ComObjectWrap references if it does not do so already. For details on how the DirectX textures were instrumented, see DirectX textures.
-
Standard memory allocation
All other objects in BigWorld use standard stack or heap memory. Unlike the memory tracking for DirectX resources, there is no unified way of instrumenting a particular class or set of classes. For details on the main issues encountered when instrumenting arbitrary classes, see Binary blocks, Terrain height maps, and Terrain texture layers.
This section documents the instrumenting of four classes/types that make up the BigWorld system:
-
DirectX textures
-
Binary blocks
-
Terrain height maps
-
Terrain texture layers.
Each of these classes/types presents its own set of challenges. These examples highlight the main issues that come up during instrumentation.
Even with the memory-tracking functionality added to the ComObjectWrap class, COM objects are not memory-tracked by default, because the accounting for memory must occur after the resource has been created. Since there is no way of knowing when this happens automatically, memory tracking cannot be activated automatically for all COM objects.
In order to activate the memory tracking for a particular
DirectX resource, the programmer must make a call to
ComObjectWrap::addAlloc(<string
description>) after creating the DirectX
resource to activate the memory tracking for this resource. Note that
ComObjectWrap objects automatically handle the
deallocation of memory from the memory tracking system on destruction
of the last resource reference. Therefore there is no need for the
programmer to explicitly make the deallocation.
Below are excerpts of the lock_map.*pp files that support DirectX texture instrumenting.
-
src/tools/worldeditor/project/lock_map.hpp
Class LockMap { public: ... ComObjectWrap<DX::Texture> lockTexture() { return lockTexture_; } ... private: ... ComObjectWrap<DX::Texture> lockTexture_; ... };
-
src/tools/worldeditor/project/lock_map.cpp
LockMap::LockMap() : gridWidth_(0), gridHeight_(0), colourLocked_(Colour::getUint32(COLOUR_LOCKED)), colourLockedEdge_(Colour::getUint32(COLOUR_LOCKED_EDGE)), colourLockedByOther_(Colour::getUint32(COLOUR_LOCKED_BY_OTHER)), colourUnlocked_(Colour::getUint32(COLOUR_UNLOCKED)) { } ... void LockMap::setTexture(uint8 textureStage) { Moo::rc().setTexture(textureStage, lockTexture_.pComObject()); } ... void LockMap::updateLockData( uint32 width, uint32 height, uint8* lockData) { bool recreate = !lockTexture_.pComObject();
...
}
void LockMap::releaseLockTexture()
{
lockTexture_ = NULL; 
}
...
void LockMap::createLockTexture()
{
...
HRESULT hr = Moo::rc().device()->CreateTexture(
gridWidth_, gridHeight_, 1, 0, D3DFMT_A8R8G8B8,
D3DPOOL_MANAGED, &lockTexture_, 0 );
lockTexture_.addAlloc("texture/lock terrain");
...
}
Several functions require a reference to a DirectX texture resource, such as setTexture. A call to ComObjectWrap:: pComObject() returns such a pointer.
When releasing resources, the release call is handled automatically. It must not be called explicitly on release — ComObjectWrap should simply be set to NULL.
It is important that when ComObjectWrap objects are being used to access a DirectX resource that they be used everywhere to access that resource. ComObjectWrap objects automatically increment and decrement the internal reference count of the COM objects they reference when passing COM objects between themselves. Passing a COM object reference from a ComObjectWrap object to a regular pointer circumvents the automated memory deallocation and reference decrementing functionality.
The only situation when a DirectX resource should be exposed as a plain pointer is when passing the object to DirectX functions directly (see the above example in src/tools/bigbang/project/lock_map.cpp's function setTexture, when Moo::rc().setTexture is called).
In src/tools/worldeditor/project/lock_map.cpp's function createLockTexture, directly after the call to Moo::rc().device()->CreateTexture, a call to lockTexture_.addAlloc("texture/lock terrain") is made. This call accounts for the allocation of memory during the Moo::rc().device()->CreateTexture call for the lockTexture_ object. The" texture/lock terrain" memory resource pool will be incremented by the amount of memory used by lockTexture_. But how to is the amount of memory used by lockTexture_ determined?
ComObjectWrap is a templated class. A
different class implementation is created at compile time for each COM
object that makes use of this class. In the case of
DX::Texture, a
ComObjectWrap<Dx::Texture> class is created.
During the call
ComObjectWrap::addAlloc(<string
description>), calls to the functions
ComObjectWrap::pool and
ComObjectWrap::size are made. Each COM object can
implement its own version of these methods using template
specialisation. The
ComObjectWrap<Dx::Texture> versions
are:
template<> uint ComObjectWrap<DX::Texture>::pool() const
{
// Get a surface to determine pool
D3DSURFACE_DESC surfaceDesc;
pComObject_->GetLevelDesc(0, &surfaceDesc);
return (uint)surfaceDesc.Pool;
}
template<> uint ComObjectWrap<DX::Texture>::size() const
{
// Determine the mip-map texture size scaling factor
double mipmapScaler = 0.0;
for (DWORD i = 0; i < pComObject_->GetLevelCount(); i++)
mipmapScaler += 1 / pow(4.0, (double)i);
// Get a surface to determine the width, height, and format
D3DSURFACE_DESC surfaceDesc;
pComObject_->GetLevelDesc(0, &surfaceDesc);
// Get the surface size
uint32 surfaceSize = DX::surfaceSize(surfaceDesc);
// Track memory usage
return (uint)(surfaceSize * mipmapScaler);
}Excerpt — src/lib/moo/com_object_wrap.ipp
The ComObjectWrap<DX::Texture>::pool function determines what memory pool the texture resides in by examining the surface description of the surface at mipmap level 0. The ComObjectWrap<DX::Texture>::size function determines the size of the memory allocation by computing the mipmap scaling factor due to the mipmap levels, determining the memory footprint of the zero level mipmap level for the given texture format, and then multiplying the two.
Every DirectX resource that is to be memory-tracked will need to implement its own versions of these functions.
Memory tracking for BinaryBlock involves calls to the RESOURCE_COUNTER_ADD and RESOURCE_COUNTER_SUB macros in the constructor and destructor respectively. In the constructor, if the BinaryBlock is the owner of the data, the memory tracking call is:
RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( "binary block",
(uint)ResourceCounters::SYSTEM),
(uint)(len_ + sizeof(*this)))Otherwise, if another BinaryBlock owns the data, the call is:
RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( "binary block",
(uint)ResourceCounters::SYSTEM),
(uint)sizeof(*this))Note that in the first version, the size of the binary data allocation is accounted for, as well as the size of the BinaryBlock object, while in the second version only the size of the BinaryBlock object is accounted for. Similarly in the destructor, if the BinaryBlock owns the data, the memory tracking call is:
RESOURCE_COUNTER_SUB( ResourceCounters::DescriptionPool( "binary block",
(uint)ResourceCounters::SYSTEM),
(uint)(len_ + sizeof(*this)))Otherwise, the call is:
RESOURCE_COUNTER_SUB( ResourceCounters::DescriptionPool( "binary block",
(uint)ResourceCounters::SYSTEM),
(uint)sizeof(*this))As it stands, the memory usage for all BinaryBlock objects is placed in a single resource pool called "binary block". This is still a very high level of granularity — it is likely that BinaryBlock objects are used by many different objects to varying degrees. It would be useful to known their distribution throughout the system. For details on how this is done, see Terrain texture layers.
Like most classes, TerrainHeightMap2 has primitive types, pointers, and objects as member variables.
TerrainHeightMap2::TerrainHeightMap2(CommonTerrainBlock2 &terrainBlock2):
...
{
...
RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( "height map",
(uint)ResourceCounters::SYSTEM),
(uint)(sizeof(*this)))
}
TerrainHeightMap2::~TerrainHeightMap2()
{
RESOURCE_COUNTER_SUB( ResourceCounters::DescriptionPool( "height map",
(uint)ResourceCounters::SYSTEM),
(uint)(sizeof(*this)))
}Excerpt — src/lib/moo/terrain_height_map2.cpp
The calls to RESOURCE_COUNTER_ADD and RESOURCE_COUNTER_SUB account for the static memory used by the member variables, but do not account for the dynamically allocated memory used by our member pointers, the member pointers of our member objects, member pointers of member objects of our member objects, and so on.
Therefore, it is necessary to examine both the member pointers and the member objects to account for their dynamic memory usage, if any. With respect to the TerrainHeightMap2 class, there are three members of interest:
-
CommonTerrainBlock2* terrainBlock2_
-
TerrainQuadTreeCell quadTree_
-
Image<float> heights_
On construction of a TerrainHeightMap2 object, a reference to the parent CommonTerrainBlock2 objects is passed in. As such, the TerrainHeightMap2 is not responsible for the memory usage of this class (in fact, the opposite is true — if memory tracking is added to CommonTerrainBlock2, then it is likely that the memory allocated by TerrainHeightMap2 should be allocated into the CommonTerrainBlock2 resource counter). Simply accounting for the terrainBlock2_ pointer in the constructor's call to RESOURCE_COUNTER_ADD is sufficient.
The TerrainQuadTreeCell class must be inspected to check whether it or any of its members result in any dynamic memory allocations. The inspection is recursive until all members and their allocations are accounted for. For the TerrainQuadTreeCell class we have the following expansion:
TerrainQuadTreeCell
std::string allocator_; // Need to account for this
std::vector<TerrainQuadTreeCell> children_; // Need to account for this
BoundingBox boundingBox_; // Need to expand
static BoundingBox s_insideOut_; // For simplicity we do not
// account for static members
Outcode oc_; // Is of primitive type uint32
Outcode combinedOc_; // Is of primitive type uint32
Vector3 min_; // Only contains primitives
Vector3 max_; // Only contains primitivesFor TerrainQuadTreeCell, the dynamic memory used by member variables TerrainQuadTreeCell::allocator_ and TerrainQuadTreeCell::children_ must be accounted for.
-
TerrainQuadTreeCell::allocator_
allocator_ is a std::string that records the resource pool being used. Since we want TerrainHeightMap2's TerrainQuadTreeCell instances to use TerrainHeightMap2's resource pool, we need a way to configure TerrainQuadTreeCells resource pool. This is done by passing the resource pool string to the TerrainQuadTreeCell constructor.
TerrainQuadTreeCell::TerrainQuadTreeCell(const std::string& allocator);
Excerpt — src/lib/moo/terrain_quad_tree_cell.cpp
allocator_ is then set to the resource pool passed as parameter, so that the correct resource pool will be used. It is a simple matter to account for allocator_'s additional memory usage in the constructor's memory-tracking call to RESOURCE_COUNTER_ADD:
TerrainQuadTreeCell::TerrainQuadTreeCell(const std::string& allocator); { ... RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( allocator_, (uint)ResourceCounters::SYSTEM), (uint)( sizeof(*this) + allocator_.capacity() )) ... }A similar call is made in the destructor.
-
TerrainQuadTreeCell::children_
children_ is a std::vector containing zero or four TerrainQuadTreeCell objects. Vectors use dynamic memory allocation, but since the objects are of type TerrainQuadTreeCell, that means we are already accounting for the memory of each allocation. The only catch with having a vector of TerrainQuadTreeCell objects is that the call to children_.resize(4) in TerrainQuadTreeCell::init would create four TerrainQuadTreeCell objects using the default constructor. These objects must use the same resource pool as their parent TerrainQuadTreeCell object. This is achieved by passing a prototype instance to the resize method with the appropriate resource pool:
void TerrainQuadTreeCell::init(...) { ... TerrainQuadTreeCell childPrototype(allocator_); children_.resize(4, childPrototype); ...Excerpt — src/lib/moo/terrain_quad_tree_cell.cpp
At this point all memory has been accounted for in class TerrainHeightMap2.
The Image<float> class must be inspected to see if it or any of its members result in any dynamic memory allocations. Like the inspection performed on TerrainQuadTreeCell, the inspection is recursive until all members and there allocations are accounted for. The member variables of the Image<PixelType> class are:
Image<PixelType> std::string allocator_; // Need to account for this PixelType* pixels_; // Need to account for this PixelType** lines_; // Need to account for this uint32 width_; // Primitive uint32 height_; // Primitive bool owns_; // Primitive size_t stride_; // Primitive
For Image<float>, the dynamic memory used by the member variables Image<float>::allocator_, Image<float>::pixels_, and Image<float>::lines_ must be accounted for.
-
Image<float>::allocator_
Just like TerrainQuadTreeCell::allocator_ (for details, see TerrainQuadTreeCell quadTree_), Image<float>::allocator_ stores the resource pool for this object. Its memory is accounted for in the constructor's memory tracking call to RESOURCE_COUNTER_ADD:
RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( allocator_, (uint)ResourceCounters::SYSTEM), (uint)( sizeof(*this) + allocator_.capacity() ))A similar call is made in the destructor.
-
Image<float>::pixels_
The dynamic memory allocations assigned to Image<float>::pixels_ can be determined simply by searching through the Image class. Every direct or indirect memory allocation and deallocation must be accounted for. The following is an example of an indirect dynamic memory allocation to the Image<float>::pixels_ member variable.
template<typename PixelType> bool Image<PixelType>::createBuffer(uint32 w, uint32 h, PixelType* & buffer, bool& owns, size_t& stride, bool& flipped) { ... // Track memory usage RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( allocator_, (uint)ResourceCounters::SYSTEM), (uint)(sizeof(PixelType) * w * h)) buffer = new PixelType[w*h]; ... } template<typename PixelType> void Image<PixelType>::init( uint32 w, uint32 h, PixelType *pixels, bool owns, size_t stride, bool flipped) { ... pixels_ = pixels; width_ = w; height_ = h; owns_ = owns; stride_ = (stride == 0) ? w*sizeof(PixelType) : stride; createLines(width_, height_, pixels_, stride_, flipped); } template<typename PixelType> inline void Image<PixelType>::resize(uint32 w, uint32 h) { ... PixelType *newBuf = NULL; // Creates a PixelType pointer bool owns, flipped; size_t stride; // newBuf points to newly created pixel buffer createBuffer(w, h, newBuf, owns, stride, flipped); // pixels_ now points to newly created pixel buffer init(w, h, newBuf, owns, stride, flipped); } -
Image<float>::lines_
Use the same technique described above for Image<float>::lines_.
The final instrumenting case study is for class TerrainTextureLayer2. The techniques described in the earlier case studies can be used to account for all but three of TerrainTextureLayer2's member variables:
CommonTerrainBlock2* terrainBlock_; // Same as CommonTerrainBlock2* terrainBlk2 std::string textureName_; // Need to account for this TerrainTextureLayer2::ImageType blends_; // Same as Image<float> heights_ uint32 width_; // Primitive uint32 height_; // Primitive Vector4 uProjection_; // Primitive Vector4 vProjection_; // Primitive size_t lockCount_; // Primitive BaseTexturePtr pTexture_; // Need to account for this State state_; // Primitive BinaryPtr compressedBlend_; // Need to account for this
TerrainTextureLayer2's member variables
-
std::string::textureName_
textureName_ can be accounted for in a similar manner to TerrainQuadTreeCell::allocator_ (for details, see TerrainQuadTreeCell quadTree_) and Image<float>::allocator_ (for details, see Image<float> heights_), except that the allocation occurs in several places in terrain_texture_layer2.cpp. Each can be handled similarly by subtracting from the resource pool before an assignment and adding to the resource pool after an assignment, as illustrated below:
RESOURCE_COUNTER_SUB( ResourceCounters::DescriptionPool( "texture layer", (uint)ResourceCounters::SYSTEM), (uint)textureName_.capacity()) textureName_ = filename; RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool( "texture layer", (uint)ResourceCounters::SYSTEM), (uint)textureName_.capacity())Excerpt — src/lib/moo/terrain_texture_layer2.cpp
-
BaseTexturePtr::pTexture_
BaseTexturePtr is a smart pointer to the abstract class BaseTexture. The first case study above already accounted for all texture usage in the system, therefore we need a way to override the call to ComObjectWrap::addAlloc(std::string description) so that for this particular texture the "terrain texture layer/texture" memory resource pool is used.
First, a search is performed to find where pTexture_ is being assigned:
pTexture_ = TextureManager::instance()->get( textureName_ );
The desired memory pool needs to be added to this parameter list to override the default pool used for TextureManager's created textures. The call now looks like this with all the default values for the parameters filled in:
pTexture_ = TextureManager::instance()->get( textureName_, true, true, true, "terrain texture layer/texture" );The last parameter must have a default value assigned so that the interface for the method is not broken. The allocator pool must be passed through to the construction of the AnimatingTexture and ManagedTexture classes. This requires the addition of the allocator pool description string to the constructors of these classes (and any other classes that inherit from BaseTexture that need more accurate memory tracking). The constructor of the BaseTexture class is also changed to accept this allocator string in its constructor and store it in a member variable. This member variable can then be used by all classes that inherit from BaseTexture to account for the memory allocated in the call to CreateTexture.
-
BinaryPtr::compressedBlend_
As mentioned above, the BinaryBlock class accounts for all its memory usage. Therefore, if we wish to account for the compressedBlend_ dynamic allocation in the TerrainTextureLayer2 memory pool we must override the memory tracking in BinaryBlock. This can be achieved similarly to how the resource pool was overridden when creating a texture. The desired pool must be passed through to the BinaryBlock constructor. This required adding the default valued allocator string to the end of the following methods:
inline BinaryPtr Moo::compressImage( Image<PIXELTYPE> const &image, std::string allocator) { ... return compressPNG(pngData, allocator); } ...src/lib/moo/png.hpp
BinaryPtr Moo::compressPNG( PNGImageData const &data, std::string allocator) { ... BinaryPtr result = concatenate(outputBuffers, allocator); ... } ... BinaryPtr concatenate ( std::vector<BinaryPtr> const &buffers, std::string const &allocator) { ... BinaryPtr result = new BinaryBlock(data, totalSize, allocator() ); ... }src/lib/moo/png.cpp
BinaryBlock::BinaryBlock( const void * data, int len, const char *allocator, BinaryPtr pOwner ) { ... RESOURCE_COUNTER_ADD( ResourceCounters::DescriptionPool(allocator_, (uint)ResourceCounters::SYSTEM), (uint)sizeof(*this)) }src/lib/resmgr/binary_block.cpp
The memory tracking console can be displayed in the client, World Editor, Model Editor and Particle Editor. To display the console in the client, press Ctrl+DEBUG+F5, and to display it in the tools, press Shift+Ctrl+F5.
The user can cycle through the realtime memory usage information by pressing Space. There are three levels of granularity:
-
All memory — one pool.
-
System memory, video memory, and miscellaneous memory — three pools.
-
System memory, managed memory, default memory, and miscellaneous memory — four pools
Note that video memory is a combination of managed and default memories. Also, miscellaneous memory is a catchall for any memory that does not fall into the system, managed, or default pools (e.g., DirectX Scratch memory).
The implementation of a game using the BigWorld client can potentially involve many scripts. These scripts will interact and have real-time requirements and dependencies, since they are part of a network game — in some cases, the game cannot be simply stopped and stepped through without affecting the behaviour of the client being debugged.
Scripting must therefore be approached in the same way as any other coding, and not as a poor alternative to C++ coding. Scripts must be designed properly, and consideration must be given to the appropriate use of class hierarchies and other language concepts.
The Python console can be brought up at any time during the game.
This is a console that looks and acts exactly like a Python interpreter, so arbitrary Python can be entered into it. All C++ modules can be accessed, so the environment is the same as that in which the scripts are executed.
Python errors and exceptions that are not caught by the script itself (before they reach the C++ call-in point) are output to the console, and therefore are errors from commands entered into the Python console itself. The console is the first place to access when an error occurs in a game.
The Python console supports multi-line commands the same way that the standard Python interpreter does, and it also supports macros like a UNIX shell. Macro invocations begin with a dollar sign ($), and their expansions are contained in a dictionary in the personality file ( see the fantasydemo personality script for an example). It implements the BWPersonality.expandMacros() callback function.
The list below describes the line editing shortcuts supported by the Python console:
-
Backspace
Deletes the character to the left on insertion point. Same as Ctrl+H.
-
Delete
Deletes the character to the right of insertion point. Same as Ctrl+D.
-
End
Moves insertion point to the end of line. Same as Ctrl+E.
-
Home
Moves insertion point to the beginning of line. Same as Ctrl+A.
-
Ctrl+Backspace
Cuts to clipboard all text between insertion point and beginning of word. Same as Ctrl+W.
-
Ctrl+Delete
Cuts to clipboard all text between insertion point and end of word. Same as Ctrl+R.
-
Ctrl+Insert
Pastes the content of clipboard after insertion point. Same as Ctrl+Y.
-
Ctrl+Left arrow
Moves insertion point to the beginning of word.
-
Ctrl+Right arrow
Moves insertion point to the end of word.
-
Ctrl+A
Moves insertion point to the beginning of line. Same as Home.
-
Ctrl+D
Deletes the character to the right of insertion point. Same as Delete.
-
Ctrl+E
Moves insertion point to the end of line. Same as End.
-
Ctrl+H
Deletes the character to the left of insertion point. Same as Backspace.
-
Ctrl+K
Cuts to clipboard all text between insertion point and end of line.
-
Ctrl+R
Cuts to clipboard all text between insertion point and end of word. Same as Ctrl+Delete.
-
Ctrl+U
Cuts to clipboard all text between insertion point and beginning of line.
-
Ctrl+W
Cuts to clipboard all text between insertion point and beginning of word. Same as Ctrl+Backspace.
-
Ctrl+Y
Pastes the content of clipboard after insertion point. Same as Ctrl+Insert.
The console also offers the functionality of auto code completion. By pressing Tab, it automatically completes the current word by trying to match it against the attributes defined for the referenced context. If a unique match is found, then it is added after the insertion point. If more than one match exists, then pressing Tab cycles through all of them.
The services provided by the Python console are also available remotely over the network.
The client accepts connections using the telnet protocol on TCP port 50001. A number of telnet options are supported, such as the emulation of line editing.
All scripts can be reloaded by pressing Caps Lock+F11.
Existing entity class instances are updated so that they are instances of the new classes, without losing their dictionary. Entity script classes can provide a reload method to refetch stored function references that have changed after reloading. For example, the reload function in the player script recalculates its key bindings, which would otherwise continue to reference functions in the old version of the class.
When the server sends an update to a script, only that script is reloaded and only its instances are updated.
Some common scripting errors are listed below:
-
Could not find class
<entity>There is a Python error in the file
<res>/scripts/client/<entity>.py. Check the file python.log in the current working folder.
-
App::init: BigWorldClientScript::init() failed
There is a mismatch between Python script and
<res>/scripts/entity_defs/<entity>.def.
-
EntityType::init: EntityDescriptionMap::parse failed
An undefined data type was used in file
<res>/scripts/entity_defs/<entity>.def file, or in<res>/scripts/entity_defs/alias.xml.
For details on entity definition and Python script files, see the document Server Programming Guide's sections Directory Structure for Entity Scripting → The Entity Definition File and Directory Structure for Entity Scripting → The Entity Script Files, respectively.
You can run BigWorld Client as a Python extension module, instead of as an executable. This will allow you to use third party interactive debuggers (such as Wing IDE and Komodo), or Python's built-in debug module (pdb) to debug your game's scripts.
To run the client as Python extension module, follow these steps below:
-
Create the Python extension module by building the client using the PyModule_Hybrid configuration. The resulting binary will be located at
bigworld/bin/client/bwclient.pyd.Note
This configuration requires the environment variable PYTHONHOME to be set to the top-level folder of a standard Python installation (e.g., C:\Python26).
-
Copy the example stub script from the fantasydemo project folder (
fantasydemo/launch_fantasydemo_pyd.py) into your own project folder (e.g.myproject/launch_myproject_pyd.py). -
If you have scripts located within additional resource paths, you will need to modify the stub script to add these paths to
sys.path. To do this, modify the resourcePaths list just before the list is assigned intosys.path.
Using the script above, you can start the client from the command prompt (for example, with the command python.exe launch_fantasydemo_pyd.py.) or from Visual Studio. Now, from anywhere in the scripts or from the Python console, you can invoke the standard Python debugger:
import pdb pdb.set_trace()
For more information on how to use the pdb module, please refer to the Python Manuals.
You can also use the start-up script to run the client using a third party interactive Python debugger/IDE. This should allow you to set breakpoints and inspect variables. For more information on how to use an interactive debugger, please refer to your package manuals.
Note
The client will be disconnected from the server if the execution of the main thread is interrupted by more than a few seconds, as it usually happens when Python scripts are debugged interactively.
Note
When running the client as an extension module, it is possible to quit the Python interpreter before exiting the client (by, for example, invoking exit() from pdb prompt).
Because the client will still try to execute Python commands during its shutdown process, this will cause the client to crash.
CAT provides a graphical interface to change watcher values and run Python console commands on a remote or local client. It does so via the Remote Python Console service provided by the client. For more details, see Remote Python Console.
In order to connect to the client, you have to provide the computer name and/or the IP address where it is running.
Client Chooser dialog
If you provide both the computer name and the IP address, CAT first tries to connect using the computer name, and if that fails, then it uses the IP address. Specify localhost as the computer name or 127.0.0.1 in the IP address to connect to the local client.
CAT automatically reconnects to the last client when it is started.
CAT searches for scripts in folder
<res>/../tools/cat/scripts,
where <res> is one
of the entries in the resources folders list (for details on how
BigWorld compiles this list, see the document Content Tools Reference Guide's chapter Starting the Tools).
For example, if one of the entries in
<res> is
C:/mf/fantasydemo/res, then CAT will look for scripts
under the folder C:/mf/fantasydemo/tools/cat/scripts.
It will then present a tree view of the scripts, as illustrated in the
picture below:
Tree view of scripts
CAT scripts are specialised Python scripts. In the image above,
the CAT script
<res>/../tools/cat/scripts/Render/Performance.py
is selected. It defines the GUI controls to the right of the tree. When
you select a different CAT script from the tree, different controls will
be presented to the right of the tree.
CAT scripts allow the viewing and manipulation of client watcher values as well as the execution of commands in the client's Python console. See the examples provided in folder fantasydemo/tools/cat/scripts for information on how to create your own scripts.
The list below describes the toolbar buttons and its functions:
-
Reconnect
Reconnects to the client, e.g., after you restart the client.
-
Disconnect
Disconnects from the client.
-
Auto update
Refreshes the data on the current tab once every 2 seconds.
-
Update
Refreshes the data on the current tab e.g., if watcher values are changed from the within client.
CAT looks for scripts in folder
<res>/../tools/cat/scripts,
thus enabling two separate sets of CAT scripts to exist:
-
BigWorld system scripts
-
Scripts defined by you that are specific for your game.
CAT scans this directory hierarchically, and generates a folder tree akin to Windows Explorer's one, using the folder names as branches and script files as leaves.
The picture below illustrates how a folder structure will be displayed in CAT:
Example folder structure vs. Resulting structure in CAT
The basic structure of the CAT script files is illustrated below:
from controls import * envSetup = \ """ … """ args = \ ( ... ) commands = \ ( ... )
Skeleton of CAT script files
CAT script files are divided in three blocks, which are described below:
-
envSetup
Block where variables local to this CAT script are created and initialised.
-
args
Represent various controls for variables, which can be edited. They can either represent variables local to the CAT page, or watchers that mirror values on the client.
-
commands
Executable commands that appear on the CAT page as buttons.
The following sections discuss the blocks args and commands.
There are several widget elements available in the args block of the CAT pages.
In order to add multiple elements, they should be placed in a list with comma delimiters. Any underscores in names are ignored and replaced by spaces.
The basic elements of args control the layout of CAT pages.
-
Divider()
This element causes CAT to display a horizontal divider between elements, allowing pages to be visually divided.
-
FixedText( "
<text>" )This element displays any static text segments such as headings. Multiple lines of text can be specified by using the newline character "\n".
It is possible to set up variables that are local to a specific CAT page.
These can also have an associated Python variable tied to them, to allow for updates from the client application.
Below is a list of commands to create variables in CAT:
-
StaticText ( "<name>" [, default = "<value>"] [, updateCommand = "<python_update _source>" ] )
This widget creates a non-editable field with the specified name and default value (if default is present).
If updateCommand is specified, then
<python_update_source>will be used as the Python value from the client that will be displayed.For example:
StaticText( "Position", updateCommand = """ location = $B.findChunkFromPoint( $p.position ).split( '@' ) print "(%.1f,%.1f,%.1f) @ %s @ %s" % ( $p.position.x, $p.position.y, $p.position.z, location[0], location[1] ) """ )Creation of field
-
CheckBox( "<name>" [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" ] ] )
This widget creates a check box called with the specified name.
If updateCommand is specified, then until the check box is edited, it will use <python_update_source> as the Python code to get the value that will be displayed.
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
For example:
CheckBox( "Use Dash", updateCommand = "$p.isDashing", setCommand = "$p.switchToDash(Use_Dash)" )Creation of check box
-
Int ( "<name>" [, default = <value> ] [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" [, minMax = (<min>,<max>) ] ] ] )
This widget creates an integer field with the specified name and default value (if default is present).
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
If updateCommand is specified, then <python_update_source> will be used as the Python value from the client that will be displayed.
If minMax is specified, then the field will have the specified (inclusive) value range.
For example:
Int( "Health", default = 100, updateCommand = "$p.healthPercent", setCommand = "$p.healthPercent = Health; $p.set_healthPercent()", minMax = ( 0, 100 ) )Creation of integer field
-
Float ( ( "<name>" [, default = <value> ] [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" [, minMax = (<min>,<max>) ] ] ] )
This widget creates a float field with the specified name and default value (if default is present).
If updateCommand is specified, then <python_update_source> will be used as the Python value from the client that will be displayed.
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
If minMax is specified, then the field will have the specified (inclusive) value range.
For example:
Float( "Speed Multiplier", default = 1.0, updateCommand = "$p.speedMultiplier", setCommand = "$p.speedMultiplier = Speed_Multiplier", minMax = ( 1.0, 10.0 ) )Creation of float field
-
FloatSlider ( ( "<name>" [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" [, minMax = (<min>,<max>) ] ] ] )
This widget creates a float slider with the specified name and default value (if default is present).
If updateCommand is specified, then <python_update_source> will be used as the Python value from the client that will be displayed.
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
If minMax is specified, then the field will have the specified (inclusive) value range.
For example:
FloatSlider( "Camera_Distance", updateCommand = "BigWorld.camera().pivotMaxDist", setCommand = "BigWorld.camera().pivotMaxDist=Camera_Distance", minMax = (0.01, 200.0) )
Creation of float slider
-
List ( "<name>", ("<option1>", "<option2>", ...), [, default = "<option9>" [, updateCommand = "<python_update_source>" ] ] )
This widget creates a drop-down menu with the specified name, containing the entries specified in the list passed as the second argument.
If default is specified, then the value entered must be present in the list passed as the second argument.
In a similar fashion to the Int and Float widgets, the drop-down menu can be given an associated Python variable.
-
Enum ( "<name>", ( ("<option_1>",<value1>), ("<option_2>",<value2>), ...), [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" ] ] )
This widget creates a drop-down menu with the specified name, containing the entries specified in the list of 2-tuple passed as the second argument.
Each entry will be associated with the value specified on the second element of its tuple.
If updateCommand is specified, then <python_update_source> will be used as the Python value from the client that will be displayed.
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
For example:
Enum( "Mode", ( ("Walk",0), ("Run",1), ("Dash",2) ), updateCommand = "2*$p.isDashing + $p.isRunning", setCommand = """ if Mode==0: $p.switchToRun(1); $p.switchToDash(0) if Mode==1: $p.switchToRun(0); $p.switchToDash(0) if Mode==2: $p.switchToRun(1); $p.switchToDash(1) """ )Creation of drop-down menu with associated numeric values
-
Bool ( "<name>", [, updateCommand = "<python_update_source>" [, setCommand = "<python_set_source>" ] ] )
This widget creates a drop-down menu with the specified name, with just one entry having two possible values: true or false.
If updateCommand is specified, then <python_update_source> will be used as the Python value from the client that will be displayed.
If setCommand is specified, then when the field is edited, <python_set_source> will be executed.
For example:
Bool( "Walking", updateCommand = "1-$p. isRunning ", setCommand = "$p. switchToRun (Walking)" )
Creation of drop-down menu with true/false entry
Watchers allow you to view the value of specific variables in a live system.
These values can be accessed from the Debug (Watchers) Console in the FantasyDemo (accessed by pressing DEBUG+ F7). However, CAT gives you a much simpler way to manipulate them.
Whenever adding a CAT control, it is essential to ensure that there is a matching MF_WATCHER that corresponds to the value in question. The best way to check that is by looking for the value in the watcher console within the game.
The value is retrieved from and saved to the value that matches the value path MF_WATCHER.
Below is a list of commands that allow you to manipulate watchers:
-
WatcherCheckBox( "<name>", "<value_path>" )
Shows a check box with the specified name.
-
WatcherFloat( "<name>", "<value_path>" [, minMax = (<min>,<max>)] )
Shows a float field with the specified name. If minMax is specified, then the field will have the specified (inclusive) value range.
-
WatcherFloatEnum ( "<name>", "<value_path>", ( ( "<option1>", <value1> ), ( "<option2>", <value2> ) ) )
Shows a dropdown menu with the specified name, containing the entries specified in the list of 2-tuple passed as the third argument.
Each entry will be associated with the value specified on the second element of its tuple.
-
WatcherFloatSlider( "<name>", "<value_path>", minMax = (<min>,<max>) )
Shows a float slider with the specified name, and with the specified (inclusive) value range.
-
WatcherInt( "<name>", "<value_path>" [, minMax = (<min>,<max>)] )
Shows an integer field with the specified name. If minMax is specified, then the field will have the specified (inclusive) value range.
-
WatcherIntSlider( "<name>", "<value_path>", minMax = (<min>,<max>) )
Shows a slider with the specified name, and with the specified (inclusive) value range.
-
WatcherText( "<name>", "<value_path>" )
Creates a field with the specified name.
These are commands that appear as buttons in CAT, shown below the elements specified in the block args.
They execute a given Python script. Each command to be added takes the following format:
( "<description>", "<script>" ),
where
<description> is
the button's caption, and
<script> is the
Python script to be executed.
For example:
from controls import *
...
commands = \
(
( "Hide_GUI", "BigWorld.hideGui()" ),
( "Show_ GUI ", "BigWorld.showGui()" )
)Example implementation of block commandsExample implementation of block commands
The BigWorld libraries provide a number of timing tools for analysing game performance. One of the most important of these is the DogWatch timer class.
When a DogWatch is created, it is registered in a global list. Whenever that timer is started, it is automatically entered into a timing statistics hierarchy, based on what other timers are running when it is started. The same timer can have multiple instances in the hierarchy if it is run more than once in different contexts.
The root timer is called Frame, and is always running for the entire frame time at every frame. All other timer instances are children of this timer's instance. Care is taken at the frame boundary to ensure that no time goes unaccounted for. Timers can even be running across the frame boundary. If this is the case, then they are momentarily stopped, and then restarted in the next frame's time slice.
A view of the DogWatch timer hierarchy is displayed in a debug console, which shows the average of the DogWatch times over the last second. The user can navigate and drill down in the hierarchy, which is displayed using indentation on a single console page. Any of the timer instances can also be graphed, and there can be as many graphs displayed simultaneously as desired. By default, the client keeps 120 frames of DogWatch timer history.