Local Repository Cache
|
Interaction with clients
- HeroEngine game clients can download files from the remote Repository to their Local Repository Cache.
- HeroEngine editing clients (such as HeroBlade, via its Repository File Selector) can both download items from the Repository to their local cache, and also in certain limited circumstances can upload certain files to the Repository, for example to make certain text files available to HSL scripts. See Repository Data Storage.
- The Repository Browser (separate from HeroBlade) is used by artists to upload art from their local systems to the Repository. It shares the Local Repository Cache with HeroBlade and other tools.
Infrastructure
In terms of the LRC's place in the Engine infrastructure, it works like this: Whenever the FireStorm API is started up, a worker thread is created on the client, whose primary duty is to accomplish the time-expensive portions of storing and retrieving resources to disk, while the main thread continues to display the game on the client. When a client (such as HeroBlade or the Player Client) identifies that it needs an asset from the Repository, this communication is done via the remote Repository Server process, which is what communicates with the actual Repository database. When the art is downloaded from the Repository to the Local Repository Cache, it is stored on local disk, and then announced as delivered, to the client that requested the resource.
To observe this process in action, open the Repository Requests panel to see current outstanding requests. Those that appear and disappear rapidly from the panel are probably being filled from the LRC, while those that stay on the panel for longer are probably waiting for response from the Repository Server.
File size
As a developer builds the game, more and more art will be stored in their Local Repository Cache, which could cause it to become quite large (hundreds of megabytes). There is not currently any mechanism to automatically delete unused or out-of-date items, so in some circumstances, such as to save disk space or possibly troubleshoot some corrupted art, a developer may wish to purge the Local Repository Cache. However, be aware that doing this will purge all art from the cache, and require re-downloading of all art assets.
Location
The LRC location is determined by two configuration values; one specified by the OS installation and one by a client configuration in Master Control. Windows specifies where program data should be stored via an environment variable named ProgramData. The client configuration CacheFilePath, which by default is set to \HeroEngine\Repository\, dictates the path beneath the location specified by the OS environment variable.
The current setting for a Windows environment variable can be displayed by running a command prompt and issuing the command echo %ProgramData%.
The LRC's location can vary based on those configuration values, but typically can be found (depending on the Operating System) in:
Windows XP (Example path, the actual path may vary)
C:\Documents and Settings\All Users\Application Data\HeroEngine\Repository
Windows Vista/7 (Example path, the actual path may vary)
C:\ProgramData\HeroEngine\Repository
Depending on when your HeroCloud world was setup your path may look something more like:
C:\ProgramData\"Your World Name"\Repository
There are valid reasons to have repositories for the different worlds of you project to be found in different locations so that they have unique LRCs. The most common reason to do this would be to separate worlds that can (and should) share a common repository of assets such as production worlds from worlds that ideally should be separate in case potential issues arising from development.
Client Installation and the LRC
There are essentially three different ways to handle the client installation for production:
- Streaming Installation (just the client 10MB or so) with all content streamed on the fly
- Full Installation which includes client and all content in the repository
- Hybrid Installation where the client and an optimized subset of content is installed. The subset of content is created using what is called "guided repository cache population".
The production client by default falls in the category of streaming install and consequently functions identically to HeroBlade when the LRC is empty. The remaining types of installations require the creation of a primed cache that is then included in the client installation.
The primed cache for a full installation is generally generated utilizing the RepositoryBrowser (see below Populating the local cache). The disadvantage of a priming the cache in this manner is that you may include content that is not actually needed or used in the content available to your customers.
Guided repository cache population is accomplished in one of two different ways, both of which allow you to use your knowledge of how your game is played to choose exactly what goes into the installer. This helps ensure the installation is as small as possible while having a better initial user experience than purely streaming installs.
- Generating a Local Repository Cache, which is then included in the client installer.
- In the future, we will also support creating the Local Repository Cache from a list of files. We expect this list to be generated from the files utilized during the initial experience (character selection, tutorial area, etc).
Basic steps for a Guided Repository Cache that contains all Game Content:
- delete the LRC on the machine you intend to use to generate the guided repository cache
- using HeroBlade, log into the world and run through the content you want included (i.e. basically play your game):
- visit game play areas
- view UIs
- preview all game FXs
- load an area that contains a sample of each character specification (causing base textures and animations to load)
Contents
The folders are organized by Game ID, which should be unique across worlds (if not, this can cause GUID conflicts). These numbered folders contain specific worlds' filename/guid/content mapping information. The content mapping is a file #, row # link to the _assets folder (a specific denormalization from the hash-size key due to measured performance issues).
The _assets folder contains the actual content-addressable file data. Content-addressing is built from a file-size file-hash pair, which results in a 20-byte binary key. Assets that share the same content-addressing information are coalesced in the assets databases.
Populating the local cache
It is possible to make a copy of a local cache (called a "Fully Populated Cache" or a "Primed" local cache). This is a cache that has been initialized on your local system, and which may then be copied to other computers. This saves time so that no longer does each client need to painstakingly download all of the files needed. The pre-generated local cache will then have all of the resources available from the Repository, and could then be packaged up and used in the install process to initialize the LRC of other clients.
To fully populate the local cache:
- Start the appropriate Repository Browser for the world that you are dealing with
- Choose File / New Local View, and navigate to anywhere on your local hard-drive
- In Configure / Settings, check the "Prime Local Cache" checkbox, and press OK
- In the "Transfers" section of the Repository Browser, set Max Downloads to maximum (100)
- Carefully drag the root folder of the Repository down to the letter of the local hard-drive
- Note: It is not necessary to create a specific location folder for the copy operation, because in Prime Local Cache mode, the files will not actually be copied to where you drag them. Instead, this will be updating your configured location (see above section on Location for more)
- Wait for all of the downloads to complete, by waiting for when 100% of the downloads have been accomplished, and the number above the "Transfers" tab says "0/0".
- The local cache is now fully primed, and located in the usual location for the LRC (see Location).
To copy this file to another client, simply copy two folders:
- The numbered folder with the appropriate Game ID number
- The "assets" folder
If a developer accesses multiple worlds, they may need multiple clients, but all worlds will share the same Local Repository Cache. The only differences will be fairly small, such as a separate .map file stored in each Local Repository, for each world.
For shared repository data on the server side, see Sharing repositories.
Removing LRC files
Removing the _assets folder REQUIRES removal of all of the world-specific folders.
Removing a world-specific folder does not require removing the _assets folder, as the local repository cache code will "re-attach" the same content to the entry already in the assets folder.
As the underlying database is a straight port with one patch of SQLite, tools such as "SQLiteSpy" can browse the contents of the Local Repository Cache database files. The corresponding code in FireStorm is isolated to $/firestorm/internalcode/clientrepository/ LRRepositoryMapStorage.cpp and LRBlobStorage.cpp, coordinated by LRStorage.cpp, with utilities in LRStorageUtils.cpp.
To purge the LRC, see Purging the Local Repository Cache
Repository Corruption
The LRC can only become corrupt (aside from user action, such as doctoring the data in place) whenever there is a sudden shutdown of the system without time for the operating system to write all pending data to disk. IE: a power failure, or a blue-screen of death.
The solution is to "move aside" the LRC, and allow the client PC to recreate the empty LRC and re-download all of the necessary assets. Obviously, the first time entering will take longer than usual, and may discover errors in the client-side script code that made assumptions about asynchronicity and timing of data being available. Typically, the user then restarts HeroBlade and "it just works".
