A fast content pipeline is crucial for today’s engines, allowing the user to quickly iterate on features, assets, etc. Molecule’s content pipeline enables hot-reloading of every asset the engine understands (textures, models, shaders, …), while retaining blazingly fast loading times.

Having worked with a rather tedious workflow in the past (quit engine, compile asset, start engine again) with somewhat long loading times during development because intermediate data formats had to be parsed/loaded, I wanted Molecule to support two major features:

• No loading of non-binary data. The engine should only load and understand binary data compiled for the platform it’s currently running on.
• Hot-reloading of every asset the engine understands – be it textures, models, shaders, or scripts, or even configuration files.

The benefits of such an approach are manifold:

1. If the engine only ever reads binary data, no intermediate data formats (jpg/tga/png, fbx/obj) have to be parsed, greatly reducing loading times for the whole team. I’ve seen lots of hours of precious development time being spent on converting assets locally, or trying to load intermediate formats like .tgas and .xmls, which additionally cause bad memory behaviour (e.g. temporary allocations). Even if it’s just 5min. each day for each team member, it adds up during a 6 month development cycle.
2. Loading binary data only leads to much cleaner, shorter and faster code paths.
3. Checking for bad assets is done at asset compile time, not at run-time, again leading to cleaner code.
4. Hot-reloading obviously enables people to work and iterate much faster. Once you’re used to that feature, going back to the old quit/compile/restart workflow certainly feels like living in the stone age.

The way the content pipeline in Molecule works is quite straightforward:

• Every asset to be added to a project gets assigned an option preset which tells the content pipeline how that asset should be processed. An option preset is nothing more than a simple list of key-value pairs parsed by the corresponding stage of the content pipeline (e.g. an option preset for textures has vastly different values compared to a preset for converting meshes). You can create as many presets as you want.
• Each time a file is added/removed/changed, the content pipeline verifies whether that file belongs to the project, and has any option preset assigned to it. If so, it processes the file via various stages in the pipeline, and writes a compiled binary file if the conversion process is successful. The latter is crucial, because that means that faulty assets will no longer break the build, because no binary file is ever written for such assets.
• Of course, the above works for option presets, too. If any preset is changed, all the assets affected by the change will be recompiled automatically. This means that e.g. if any option of the preset for converting diffuse texture maps is changed, all affected textures will automatically be compiled and reloaded by the engine. You can literally watch the new assets getting streamed in.
• Whenever a new file is available for the engine to be loaded, the content pipeline sends a notification via TCP/IP to the running engine. This allows assets to be hot-reloaded on consoles, too. This also works for files coming from external sources, e.g. Perforce/CVS/SVN – you can sync from Perforce, and the engine will automatically pick-up and reload new files (if you want it to).

The following would be an example of two different option presets for compiling textures (diffuse maps and light maps):

TexturePresets:
{
	DiffuseMap:
	{
		ScaleX = 0.5
		ScaleY = 0.5
		Degamma = 2.2
		Gamma = 2.0
		Format = DXT
 	}

	LightMap:
	{
		ScaleX = 1.0
		ScaleY = 1.0
		Degamma = 2.2
		Gamma = 2.2
		Format = RGBA
	}
}

Most of the files stored by the content pipeline/editor are just simple text files somewhat similar to JSON/other human-readable formats – no fancy XML or anything. As stated above, if you were to change the gamma setting of the diffuse map option preset, all affected textures would automatically get recompiled again.

In addition, Molecule’s content pipeline offers some additional handy features:

• Direct loading of .psd Photoshop files. You can save your work in Photoshop, and the content pipeline/engine will automatically reload the asset. No need for intermediate formats like .tga or similar if you don’t need them.
• Direct loading of .mb Maya files. On machines having a Maya license installed (useful for artists on a team), the engine can directly compile assets from Maya binary files. Additionally, assets can be exported/compiled from within Maya/Max/XSI with a simple button-click.

The following video shows some of the above features – the scene itself is the well-known Sponza Atrium (made publicly available by Crytek), textured with a simple texture for testing purposes. The video shows hot-reloading of textures, shaders, and meshes (make sure to enable HD quality):

http://www.youtube.com/watch?v=BQPpZkRk6y4

Having worked with such a workflow during the past few months, I can honestly say that I never want to go back to anything different.