Using runtime-compiled C++ code as a scripting language

Today’s post is less of an insight into how Molecule works, and more of an announcement about an upcoming feature we are very proud of!

Molecule Engine’s scripting system uses runtime-compiled C++ code as a scripting language, and you can see the system in action here (please make sure to watch the video in original quality).

This allows the engine to leverage the full performance potential of native C++ code, while providing designers and scripters with extremely short iteration times, commonly only experienced when using traditional scripting languages such as lua, python, or others.

Scripters won’t have to deal with internal engine details, and don’t need to worry about pointers or other low-level language stuff. They only work with a pure C-interface and opaque structs, as can be seen in the video. But programmers can easily dive in and feel right at home with the whole engine available to them in native C++-code.

Furthermore, programmers can aid scripters easily by using their favourite debuggers and IDEs for debugging and development. Scripters will love certain IDE features such as IntelliSense, completion listboxes, and other things a modern IDE provides!

Let us know what you think in the comments!

Advertisements

22 thoughts on “Using runtime-compiled C++ code as a scripting language

  1. Hi there, we’ve been using this kind of optimization on our software) for some time and it works pretty well. Instead of supporting directly C/C++ scripts we have a language that gets compiled and transformed to C++ before execution begins (and we can run our scripts in native language if needed too). We are using mingw for the compilation process, what about you? Can you point some more technical information about your implementation? Risks and challenges, etc? Thanks!

    • I rolled my own implementation.
      With all of the infrastructure already available in Molecule (separate content pipeline that talks to the engine over TCP/IP, generic resource hot-swap, auto asset-reload, etc.) it took me only 2 days to implement this, from start to finish.

  2. I’m assuming this is just dynamic library hot-reloading right? How do you deal with references in the scripts to engine specific interfaces of C++ objects given the issues of toolchain specific name mangling conventions across the various platforms? I’d like to know how you deal with this from a portability perspective or is it just PC-specific as your engine tools exist only on that platform?

    • Yes, shared library hot-swapping is the gist of it.
      Maybe I understood your question wrong, but I don’t see the problem with name mangling? The “script” code (it is C++ code after all) is compiled using the native toolchain, and compiled for each platform.

      The main entry points (so to say) for the scripts are two pure-C functions, which are exported in the shared library. From there, the engine works against an abstract interface, and calls into the (virtual) script functions.

      Did you mean something else?

      • Thanks for the response Stefan,

        Your comments in response to name mangling considerations makes sense, I was thinking about the problem the wrong way, I.e. I didn’t consider the fact that as you’re compiling the scripts directly you can just link against your engine libraries statically which means you don’t need to worry about run-time symbol issues that can be a problem when dealing with dynamically loaded C++ code.

        Can you clarify the point on engine -> script interaction though, i.e. what do you mean by “virtual” script functions? do you mean literally virtual functions i.e.

        // engine-side
        class MyCallback
        {
        public:
        virtual CallMe() =0;
        };

        // script-side
        class MyScriptEngineCallback : public MyCallback
        {
        // override CallMe here…
        };

        … or something else?

      • Ah, that’s what you meant! Yes, that could definitely lead to problems.

        Yes, I literally mean virtual functions. The base class for a script offers abstract methods for startup, shutdown, update, etc. A script implements this interface, but all of this behind-the-scenes is of course hidden away. The engine runtime then calls the script’s virtual functions inside a try/catch block.

        The pure C functions are required for creating/destroying the script (or rather an instance of the actual implementation of the interface), and setting up the script environment.

      • Hello,

        Nice work 🙂

        Do you do everthing (unload/recompile/reload) in one single libary for the entire game code, or there are several libraries ?

      • Hi Francois,

        I’m not sure whether you meant the engine libraries, or the scripts.
        The engine consists of several libraries, each one takes care of hot-reloading binary data itself.
        The scripts are all compiled into individual files, so changing a script means unloading/recompiling/reloading just this file, as long as it doesn’t have any dependencies that also need to be compiled in order for it to work.

  3. Hi, a few questions:
    – Do you have any problems with compile times? I am using C because I am afraid that it can take too long to compile c++.
    – Can you attach a script to any game object? If so, does it mean every game object with a script has a separate dll? Is “update” virtual function called for each of these objects?
    – What happen with script member variables during hot reload? It seems their values are kept as the light in the video continued from the same position when its color is changed.
    – Do you have access to script variables directly from the game editor (in a way similar to Unity)?
    – Don’t you have problem with scripters causing hard to find bugs, memory leaks, crashes, …?

    Thanks for this blogpost, I have never liked the idea of an interpreted scripting language for game scripts, now I can see there are programmers out there using compiled languages for the task.

    • Hi Mikulas,

      – Do you have any problems with compile times? I am using C because I am afraid that it can take too long to compile c++.

      No, not at all. Even if I make the whole engine available to a script, compiling it takes less than a second, even when the content pipeline spawns a new process, sets up the compilation environment, and compiles the script. When compiling several scripts at once (e.g. for a full asset build), you only need to spawn the process once, and can compile several scripts in parallel.
      The key is to use pre-compiled header files, and have a lightweight codebase which really only includes what it needs. Additionally, I use a separate header-file which pulls in everything a script needs, and make sure that this file does not pull in unnecessary includes.

      – Can you attach a script to any game object? If so, does it mean every game object with a script has a separate dll? Is “update” virtual function called for each of these objects?

      Yes, a script is a ScriptComponent in the entity-component-architecture that Molecule uses, and can be attached to any entity.
      In development builds, each script has a separate shared library in order to support fast hot-swapping of individual scripts. In retail builds (or any bundled asset build), several scripts belonging to the same resource package can be put into the same shared library, saving memory and yielding faster load times.
      Yes, Update() is virtual and is called for each ScriptComponent, with additional safety nets to not make the script code crash under any circumstances.

      – What happen with script member variables during hot reload? It seems their values are kept as the light in the video continued from the same position when its color is changed.

      Yes, that’s right. The state of a script is serialized to memory, the script reloaded, and then the state is serialized back from memory into the component.

      – Do you have access to script variables directly from the game editor (in a way similar to Unity)?

      Not at the moment, but I would like to change the manual serialization step into a system based on a lightweight reflection mechanism. This could be used to expose variables (both globals and members) to the editor, and do the serialization automatically.

      – Don’t you have problem with scripters causing hard to find bugs, memory leaks, crashes, …?

      A script can never crash the engine. Scripts are executed using custom exception handlers which take care of that.
      Regarding memory leaks, the whole engine is built around handles and IDs (see the corresponding blog post for more information), and scripters should only use the C-like interface (free functions). Scripters are never exposed to pointers or any low-level struct, but rather opaque structs (Entity, ID, etc.) which they use as identifiers to certain objects.

      Of course, if anybody wants to access certain engine parts directly and juggle with raw pointers, there’s always a possibility for memory leaks. I would assume that mostly programmers helping out scripters would do such things, but I understand that using C++ as a scripting language can be both a blessing and a curse in that regard. However, I do like the prospect of using a good IDE and a professional debugger when trying to find script bugs. No matter which language you choose, there will be tons of bugs (mostly logic-related ones generally) anyway :).

      • Thank you very much for your answer, I really appreciate it. Clearly your scripters are different than the ones we have here 🙂

    • – Do you have any problems with compile times? I am using C because I am afraid that it can take too long to compile c++.

      Small update regarding compilation times: I have further optimized the setup and compilation step, and the script seen in the video is compiled in ~0.3s, including spawning a process and copying the compilation output.

  4. Pingback: OpenGL Roundup, September 19, 2013 | Learn OpenGL ES

  5. Hi, I assume you use GetProcAddress (or something equivalent on other platforms). Does this work with ps3 (there is nothing like GetProcAddress as far as I know)?

  6. Pingback: Using runtime-compiled C++ code as a scripting language: under the hood | Molecular Musings

  7. Pingback: Video: RCC++ at the 2012 Develop Conference | Ragnarok Connection

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s