Today’s post is less of an insight into how Molecule works, and more of an announcement about an upcoming feature we are very proud of!
Molecule Engine’s scripting system uses runtime-compiled C++ code as a scripting language, and you can see the system in action here (please make sure to watch the video in original quality).
This allows the engine to leverage the full performance potential of native C++ code, while providing designers and scripters with extremely short iteration times, commonly only experienced when using traditional scripting languages such as lua, python, or others.
Scripters won’t have to deal with internal engine details, and don’t need to worry about pointers or other low-level language stuff. They only work with a pure C-interface and opaque structs, as can be seen in the video. But programmers can easily dive in and feel right at home with the whole engine available to them in native C++-code.
Furthermore, programmers can aid scripters easily by using their favourite debuggers and IDEs for debugging and development. Scripters will love certain IDE features such as IntelliSense, completion listboxes, and other things a modern IDE provides!
Let us know what you think in the comments!
Molecular Musings 
Hi there, we’ve been using this kind of optimization on our software) for some time and it works pretty well. Instead of supporting directly C/C++ scripts we have a language that gets compiled and transformed to C++ before execution begins (and we can run our scripts in native language if needed too). We are using mingw for the compilation process, what about you? Can you point some more technical information about your implementation? Risks and challenges, etc? Thanks!
The C/C++ code is used directly, and we use the native toolchain for compilation.
I can share a few more technical details in the future!
Interesting news indeed! Some time ago I checked this one: http://runtimecompiledcplusplus.blogspot.com. Do you use this one as a base of your implementation or have you rolled your own implementation?
I rolled my own implementation.
With all of the infrastructure already available in Molecule (separate content pipeline that talks to the engine over TCP/IP, generic resource hot-swap, auto asset-reload, etc.) it took me only 2 days to implement this, from start to finish.
I’m assuming this is just dynamic library hot-reloading right? How do you deal with references in the scripts to engine specific interfaces of C++ objects given the issues of toolchain specific name mangling conventions across the various platforms? I’d like to know how you deal with this from a portability perspective or is it just PC-specific as your engine tools exist only on that platform?
Yes, shared library hot-swapping is the gist of it.
Maybe I understood your question wrong, but I don’t see the problem with name mangling? The “script” code (it is C++ code after all) is compiled using the native toolchain, and compiled for each platform.
The main entry points (so to say) for the scripts are two pure-C functions, which are exported in the shared library. From there, the engine works against an abstract interface, and calls into the (virtual) script functions.
Did you mean something else?
Thanks for the response Stefan,
Your comments in response to name mangling considerations makes sense, I was thinking about the problem the wrong way, I.e. I didn’t consider the fact that as you’re compiling the scripts directly you can just link against your engine libraries statically which means you don’t need to worry about run-time symbol issues that can be a problem when dealing with dynamically loaded C++ code.
Can you clarify the point on engine -> script interaction though, i.e. what do you mean by “virtual” script functions? do you mean literally virtual functions i.e.
// engine-side
class MyCallback
{
public:
virtual CallMe() =0;
};
// script-side
class MyScriptEngineCallback : public MyCallback
{
// override CallMe here…
};
… or something else?
Ah, that’s what you meant! Yes, that could definitely lead to problems.
Yes, I literally mean virtual functions. The base class for a script offers abstract methods for startup, shutdown, update, etc. A script implements this interface, but all of this behind-the-scenes is of course hidden away. The engine runtime then calls the script’s virtual functions inside a try/catch block.
The pure C functions are required for creating/destroying the script (or rather an instance of the actual implementation of the interface), and setting up the script environment.
Hello,
Nice work 🙂
Do you do everthing (unload/recompile/reload) in one single libary for the entire game code, or there are several libraries ?
Hi Francois,
I’m not sure whether you meant the engine libraries, or the scripts.
The engine consists of several libraries, each one takes care of hot-reloading binary data itself.
The scripts are all compiled into individual files, so changing a script means unloading/recompiling/reloading just this file, as long as it doesn’t have any dependencies that also need to be compiled in order for it to work.
As Dotun said, would love to know how you deal with the portability issues, and will there be any development towards other platforms?
Hi, a few questions:
– Do you have any problems with compile times? I am using C because I am afraid that it can take too long to compile c++.
– Can you attach a script to any game object? If so, does it mean every game object with a script has a separate dll? Is “update” virtual function called for each of these objects?
– What happen with script member variables during hot reload? It seems their values are kept as the light in the video continued from the same position when its color is changed.
– Do you have access to script variables directly from the game editor (in a way similar to Unity)?
– Don’t you have problem with scripters causing hard to find bugs, memory leaks, crashes, …?
Thanks for this blogpost, I have never liked the idea of an interpreted scripting language for game scripts, now I can see there are programmers out there using compiled languages for the task.
Hi Mikulas,
No, not at all. Even if I make the whole engine available to a script, compiling it takes less than a second, even when the content pipeline spawns a new process, sets up the compilation environment, and compiles the script. When compiling several scripts at once (e.g. for a full asset build), you only need to spawn the process once, and can compile several scripts in parallel.
The key is to use pre-compiled header files, and have a lightweight codebase which really only includes what it needs. Additionally, I use a separate header-file which pulls in everything a script needs, and make sure that this file does not pull in unnecessary includes.
Yes, a script is a ScriptComponent in the entity-component-architecture that Molecule uses, and can be attached to any entity.
In development builds, each script has a separate shared library in order to support fast hot-swapping of individual scripts. In retail builds (or any bundled asset build), several scripts belonging to the same resource package can be put into the same shared library, saving memory and yielding faster load times.
Yes, Update() is virtual and is called for each ScriptComponent, with additional safety nets to not make the script code crash under any circumstances.
Yes, that’s right. The state of a script is serialized to memory, the script reloaded, and then the state is serialized back from memory into the component.
Not at the moment, but I would like to change the manual serialization step into a system based on a lightweight reflection mechanism. This could be used to expose variables (both globals and members) to the editor, and do the serialization automatically.
A script can never crash the engine. Scripts are executed using custom exception handlers which take care of that.
Regarding memory leaks, the whole engine is built around handles and IDs (see the corresponding blog post for more information), and scripters should only use the C-like interface (free functions). Scripters are never exposed to pointers or any low-level struct, but rather opaque structs (Entity, ID, etc.) which they use as identifiers to certain objects.
Of course, if anybody wants to access certain engine parts directly and juggle with raw pointers, there’s always a possibility for memory leaks. I would assume that mostly programmers helping out scripters would do such things, but I understand that using C++ as a scripting language can be both a blessing and a curse in that regard. However, I do like the prospect of using a good IDE and a professional debugger when trying to find script bugs. No matter which language you choose, there will be tons of bugs (mostly logic-related ones generally) anyway :).
Thank you very much for your answer, I really appreciate it. Clearly your scripters are different than the ones we have here 🙂
Small update regarding compilation times: I have further optimized the setup and compilation step, and the script seen in the video is compiled in ~0.3s, including spawning a process and copying the compilation output.
Pingback: OpenGL Roundup, September 19, 2013 | Learn OpenGL ES
Hi, I assume you use GetProcAddress (or something equivalent on other platforms). Does this work with ps3 (there is nothing like GetProcAddress as far as I know)?
On PS3, there is something similar to dynamically loaded/shared libraries: .prx files.
But don’t you have to link against *_stub.a to use .prx?
I haven’t tested it on PS3 yet (no access to a dev-kit at the moment), but I’m pretty sure it can be done if you get “creative”. You can always patch v-table entries or similar.
Pingback: Using runtime-compiled C++ code as a scripting language: under the hood | Molecular Musings
Pingback: Video: RCC++ at the 2012 Develop Conference | Ragnarok Connection
Hi there, I started working on my own engine for school and I saw this and wanted to implement this immediately, but for some reason, I did everything the same as on Github. It will not compile at all. Is there some configurations I need to do in order to let it see file changes?? completely lost. I also checked the properties of visual studio also nothing there either..
Hi,
I think you’re mixing up my blog post with somebody else’s implementation, because I did not host anything on Github.