bw logo

Chapter 2. Operation

2.1. MakePatch

This tool is located at bigworld/tools/misc/offline_patcher/make_patch.py.

MakePatch builds patches which can be used for ApplyPatch to patch targets, and supports multi-versioned patch files (for details on ApplyPatch, see ApplyPatch).

Both MakePatch and ApplyPatch have their implementations as part of the patcherlib.command_line module.

2.1.1. Command-line usage

The syntax for invoking MakePatch via the command line is described below:

make_patch.py [options] <source-path> <dest-path> <patch-output-path>

The three required arguments (one source path for each source version that the patch will support) are described below:

  • <source-path>

    The source tree path. This can be either a directory or a ZIP file archive.

  • <dest-path>

    The destination tree path. This can be either a directory or a ZIP file archive.

  • <patch-output-path>

    The patch output path.

The available options are described below:

  • -h, --help

    Displays the help page.

  • --debug-level=LOGLEVEL

    Specifies the debug level of MakePatch - LOGLEVEL must have one of the following values: DEBUG, INFO, WARNING, CRITICAL, or ERROR.

  • --source-version=SOURCEVERSION

    Specifies the label for the source version - the default value is the basename of the <source_path> argument.

  • --use-checksums-if-unmodified

    Specifies that MakePatch should use checksums for unmodified files as well as modified files - the default is to not use checksum for unmodified files.

  • --ignore-patterns=IGNOREPATTERNS

    Specifies the patterns to ignore when traversing the directory hierarchy - the default is /CVS$|/\.svn$|/\.#[^/]+$.

    Uses commas to delimit the patterns, and use the backlash character to escape the comma character in a pattern (i.e., '\,').

    By default, certain file patterns are not considered when traversing the hierarchies looking for changes. Files matching the ignored patterns are not considered when computing the difference between the two trees. If directories match the ignored file patterns, then the entire tree is culled from the search space. These file patterns (as regular expressions) are:

    • /CVS$

      This pattern matches CVS control directories.

    • /\.svn$

      This pattern matches Subversion control directories.

    • /\.#[^/]+$

      This pattern matches .# files that are a result of CVS updates.

2.1.2. Patch archives

The MakePatch tool generates a patch archive, which is a tarfile with the following:

  • The changes between the source directory tree and a destination directory tree.

  • A manifest file.

The directory tree versions are given names indicating their version (e.g., bw_res-1.9.0). The only requirement for a version is that they be a string - they do not have to conform to any convention, but it may be wise to have some scheme where the target trees are identified by name and a version string appended to the target name.

Currently, you can specify more than one source version to provide changes against the destination version. This could be used to make a patch multi-versioned, that is, supporting multiple released versions by combining their changes from each supported source version into a single patch archive. However, this is discouraged, as it is much better for the CheckVersion script or equivalent to handle selection of individual patches to download from the versions file (for details on CheckVersion, see CheckVersion).

The changes for a particular version consist of the following:

  • New files added in the destination version.

  • Binary delta patches to existing files.

The manifest file contains instructions on which files to add, modify or delete, as well as checksum information. By default, checksums are generated for modified files only. However, you can use the --use-checksums-if-unmodified option to have checksums generated for every file encountered in the source directory tree. If checksums are present in the manifest file, then they will each be checked.

Note

ZIP file archives inside ZIP archives have their files extracted and the deltas generated based on the contents of the files contained inside the ZIP archive, rather than on the ZIP archive files themselves.

2.1.3. Multi-source support

MakePatch supports patch files that contain changes for multiple sources, intended against either multiple targets and/or versions.

This means that for a group of targets, changes for each target from one version to another can be bundled into the same patch archive.

This also means that for a particular target the changes for multiple versions of that target can be contained within the same patch archive. Typically, however, it is more advantageous to have a single-version patch files for each released version, and rely on VersionCheck and the versions file to drive the patching client to download a single-version patch. This is more flexible and saves on download bandwidth.

Changes for multiple sources can be added at the archive creation, or they can be added to an existing patch archive by re-running MakePatch with the same patch archive as the patch output path.

2.1.4. Example usage

2.1.4.1. Creating a single-source patch archive

The example below creates a patch archive between the source tree with path 1.8.0/bigworld/res (labelled bw_res-1.8.0) and 1.9.0/bigworld/res (labelled bw_res-1.9.0), and outputs it to a patch archive file called bw_res-1.8.0-1.9.0.patch.

$ python make_patch.py --source-version=bw_res-1.8.0 \
  1.8.0/bigworld/res     1.9.0/bigworld/res     bw_res-1.8.0-1.9.0.patch

2.1.4.2. Creating a multiple-source patch archive

The example below creates a patch archive between source trees:

  • Path 1.8.0/fantasydemo/res (labelled fantasydemo_res-1.8.0) and path 1.9.0/fantasydemo/res

  • Path 1.8.0/fantasydemo/game (labelled fantasydemo_game-1.8.0) and path 1.9.0/fantasydemo/game

These two sets of changes are packaged into a patch archive file called fantasydemo-1.8.0-1.9.0.patch.

$ python make_patch.py --source-version=fantasydemo_res-1.8.0
  1.8.0/fantasydemo/res     1.9.0/fantasydemo/res     fantasydemo-1.8.0-1.9.0.patch
...

$ python make_patch.py --source-version=fantasydemo_game-1.8.0
  1.8.0/fantasydemo/game    1.9.0/fantasydemo/game    fantasydemo-1.8.0-1.9.0.patch

2.2. ApplyPatch

This tool is located at bigworld/tools/misc/offline_patcher/apply_patch.py. It complements the MakePatch tool by applying patches made by it to a directory tree or ZIP archive. The ApplyPatch tool relies on functions exposed by the patcherlib.patch_apply module (for details, see The patch_apply module).

Both MakePatch and ApplyPatch have their implementations as part of the patcherlib.command_line module.

2.2.1. Command line usage

The syntax for invoking ApplyPatch via the command line is described below:

apply_patch.py [options] <patch-path> <target-path> <target-version>

-h, --help            show this help message and exit
--debug-level=LOGLEVEL one of DEBUG, INFO, WARNING, CRITICAL, ERROR

The three required arguments are described below:

  • <patch-path>

    The path to the patch archive.

  • <target-path>

    The destination tree path.

  • <target_version>

    The target version before patching.

The available options are described below:

  • -h, --help

    Displays the help page.

  • --debug-level=LOGLEVEL

    Specifies the debug level of ApplyPatch - LOGLEVEL must have one of the following values: DEBUG, INFO, WARNING, CRITICAL, or ERROR.

2.2.2. Example usage

2.2.2.1. Patching against a directory tree with a patch file

The example below applies a patch archive on to a source tree with path ./res and version bw_res-1.8.0, and outputs it to a patch archive file at downloads/bw_res-1.8.0-1.9.0.patch.

$ python apply_patch.py downloads/bw_res-1.8.0-1.9.0.patch bigworld/res bw_res-1.8.0 

2.3. CheckVersion

CheckVersion is a command line tool that implements a minimal patching client. As input, it takes the following configuration files:

  • A state file.

    Contains the state of targets for that end-user's distribution. The default is state.xml.

  • A versions file.

    Contains upgrade paths for possible release versions. This is retrieved from a remote server via HTTP. An example is supplied in bigworld/tools/misc/offline_patcher/examples/versions.xml.

  • A general game configuration file.

    Contains the URL to retrieve the versions file from. The default is patcher_config.xml, and an example is supplied in bigworld/tools/misc/offline_patcher/examples/patcher_config.xml.

CheckVersion gets from the general game configuration the URL for the versions file, then retrieves its contents and parses it. It then compares its local version from the local state with this remote version specified in the versions file, and determines an upgrade path if necessary from the versions file. It then downloads each patch archive, and applies them using the functionality provided in the patcherlib.patch_apply module (for details on this module, see The patch_apply module).

This script is intended as a bare-bones update client, and it is expected that game developers implement their own custom launcher application incorporating an update client that may reuse the functionality provided in the CheckVersion script. For example, a UI update client could be implemented which utilises progress bar widgets, HTML display of release notes, and so on.

The CheckVersion script is driven by the retrieved versions file (for details on this file, see Versions file).

2.3.1. Command line usage

The syntax for invoking CheckVersion via the command line is described below:

usage: check_version.py [options]

options:
-h, --help            show this help message and exit
-c CONFIGPATH, --config=CONFIGPATH
                      The patcher configuration file that contains the
                      versions.xml URL. Defaults to "patcher_config.xml".
-s STATEPATH, --state=STATEPATH
                      The state file. Defaults to "state.xml".
--debug-level=DEBUGLEVEL
                      The logging debug level. Defaults to "INFO".

2.3.2. Versions file

This file contains upgrade paths for past released versions to upgrade to the current version. Functionality for parsing this XML document is supplied via the VersionsXML class (for details, see The versions module).

The versions file is downloaded from a URL contained within the configuration file, defaulting to a patcher_config.xml in the current working directory, and is parsed using the simple Config class in check_versions.py. Developers may wish to change this scheme when developing their own update client.

An example versions file is supplied in bigworld/tools/misc/offline_patcher/examples/versions.xml.

The example file below describes an upgrade path from a past released version of a theoretical game from an initial version 1.0 to a new current version 1.1. In the example, there are three targets for which there are patches for:

  • your_game_res

    Refers to a ZIP archive of the packed game resources (e.g.., res-packed tree of the contents of your_game/res).

  • your_game_bin

    The game executable directory, designed for patching against the game executable and any other binaries, such as DLLs needed by the executable, as well as configuration files needed by the game, such as paths.xml.

  • bw_res

    Refers to a ZIP archive of the packed BigWorld base resources (i.e., res-packed tree of the contents of bigworld/res).

<?xml version="1.0"?>
<root>
  <currentVersion>1.1</currentVersion>
  <property name="baseURL">http://update.yourgame.com/downloads</property>
  <supportedVersions>
      <version>
          <name>1.0</name>
          <property name="updateInfoURL">info/test.php</property>
          <targets>
              <target>
                  <name>your_game_res</name>
                  <path>../your_game_res.zip</path>
                  <sourceVersion>1.0</sourceVersion>
                  <destVersion>1.1</destVersion>
                  <patch>
                      <transferType>http</transferType>
                      <name>your_game_res.1.0-1.1.patch</name>
                      <property name="url">patches/your_game_res.1.0-1.1.patch</property>
                      <property name="md5sum">00112233445566778899aabbccddeeff</property>
                  </patch>
              </target>
              <target>
                  <name>your_game_bin</name>
                  <path>.</path>
                  <sourceVersion>1.0</sourceVersion>
                  <destVersion>1.1</destVersion>
                  <patch>
                      <transferType>http</transferType>
                      <name>your_game_bin.1.0-1.1.patch</name>
                      <property name="url">patches/your_game_bin.1.0-1.1.patch</property>
                      <property name="md5sum">00112233445566778899aabbccddeeff</property>
                  </patch>
              </target>
              <target>
                  <name>bigworld_res</name>
                  <path>../bw_res.zip</path>
                  <sourceVersion>1.9.3.0</sourceVersion>
                  <destVersion>1.9.4</destVersion>
                  <patch>
                      <transferType>http</transferType>
                      <name>bw_res.1.9.3.0-1.9.4.patch</name>
                      <property name="url">patches/bw_res.1.9.3.0-1.9.4.patch</property>
                      <property name="md5sum">00112233445566778899aabbccddeeff</property>
                  </patch>
              </target>
          </targets>
      </version>
  </supportedVersions>
</root>

Example versions file

Note that version 1.1 of the overall game uses 1.9.4 BigWorld resources over version 1.0's usage of 1.9.3 BigWorld resources. This illustrates that the versioning of the individual targets is independent from the overall game versioning.

This is only an example layout for a theoretical game. Game developers must take into account the requirements of their game when partitioning their resources into suitable resource trees.

2.3.2.1. Properties

Properties are used to store specific details about some part of the patching process. For example, you might have a URL to a web page for each upgrade path containing the release notes that you wish to have displayed while the patching process is running. Custom transfer handlers can utilise the property mechanism to get details about how to retrieve a patch. Properties can be defined at the root level, at the upgrade path level (that is, under the version element), or at the patch level (under the patch element).

Properties are key-value pairs, described as in the following:

<property name="key_name">value</property>

The combined properties from root-level, upgrade path-level to patch-level are passed to the constructor of any transfer handler as keyword arguments. Properties at lower levels override properties at higher levels.

For example, the versionslib.transfer module defines the HTTP transfer method in the class HTTPTransferHandler. The constructor to HTTPTransferHandler has a baseURL property which, in the example above, is described at the root level. However, it could also be described per-upgrade path, so that each upgrade path has a different base URL.

2.3.3. State file

The state file specifies for each target which version is currently installed on the end-user's distribution. This is only a simple implementation - developers may choose to implement persistence of target state in another way as part of their launcher process (e.g., Windows Registry keys). Other implementations should implement the Python State interface for reusing the rest of the CheckVersions script and the versionslib library.

An example state file is supplied in bigworld/tools/misc/offline_patcher/examples/state.xml. It corresponds to the supplied example versions.xml versions file.

Functionality for parsing the state XML file is supplied in the form of the StateXML class defined in the versionslib.state module (for details on this class, see The state module).

2.3.3.1. Example state file

<?xml version="1.0"?>
<root>
  <currentVersion> 1.0 </currentVersion>
  <targets>
      <target name="your_game_res"> 1.0 </target>
      <target name="your_game_bin"> 1.0 </target>
      <target name="bigworld_res"> 1.9.4.0 </target>
  </targets>
</root>

Example state file

2.3.4. Deployment notes

The CheckVersion script relies on patcherlib for applying patches to targets. Deploying this may involve compiling all the .py files to .pyc or .pyo files, then adding them in a ZIP archive and placing in the Python library path.

The game launcher must run script methods for checking against a versions file to verify that its version is up-to-date. This requires that a Python interpreter be involved at some point in the game launch process, which implies that the launcher is itself a Python script or an executable that is linked against the Python runtime library.

2.3.5. Patch transport schemes

There is currently only one method of transport for patches: HTTP. However, the versions file syntax can allow for more transport schemes (such as BitTorrent, for example). Transport schemes are specified as part of the delivery element in the versions document - e.g., consider the following example versions file fragment:

<patch>
 <transferType>           http                             </transferType>
 <name>                   bw_res-1.9.4.0-1.9.x.ypatch         </name>
 <property name="url">    patches/bw_res-1.9.4.0-1.9.x.y.patch </property>
 <property name="md5sum"> c166772f145bcc59ac714b7995e52c2a </property>
</patch>

The transferType tags describes how to download a patch file from a URL via HTTP. This transfer handler supports resuming of partial downloads with a small amount of rollback, that is, resuming will occur at a point (by default 4K) before the end of the received file fragment. In order to detect possible file corruption, the newly downloaded portion corresponding to the rollback amount is checked against what was downloaded previously, and the download will restart from the beginning if corruption is detected.

Properties for a patch transfer are specified using the property tags. The HTTP transfer type defines the following properties:

  • url

    The URL from which to retrieve the patch file. If a relative URL is given, then it is relative to the base of the version file URL.

  • md5sum

    The MD5 sum of the retrieved patch file.

You can add your own transfer type by specifying properties that describe how to retrieve a patch file. You will need to modify the CheckVersions script and create and register a new PatchTransferHandler class.