Reflection

I think the standard choice for a collider shape of a player controller nowadays remains firmly in the court of a capsule. It can represent a vertical body with rotational symmetry, quite like a cylinder, but without involving much more complicated math. Its most important advantage over shapes with flat bottom walls is that a capsule can scale slopes and obstacles much more smoothly than its counterparts.

Because of that, when prototyping Hosper in Godot, I decided to use it for a player controller as well. It served me well during testing of different forms of geometry for maps. I decided to go ahead and add support for capsule collisions in Chavelleh, but despite already having dealt with spheres and arbitrary hulls, this exercise still took me some time and left me unconvinced regarding the reliability of my implementation. Inclined to gather more insight into collision data, I quickly put together some visualizers and started testing collision scenarios involving different sets of bodies. While this exercise immediately helped me identify and iron out some bugs, the approach of describing every such scenario in a single scene, in C++, quickly revealed itself to be irritating - it took a lot of code to construct them and compiling and verifying every small change to either positions or behaviour took quite a bit of time.

These problems could be easily avoided were I to use physics-sandbox, my physics testing tool with a capability to define basic scenes using JSON. However, the scenarios I had in mind involved the tested bodies performing arbitrary motions in space; that could not be easily represented just by a set of parameters in a JSON file. There was little doubt in my head at that point, that this situation called for finally attacking the problem of introducing scripting capabilities to the engine - custom logic interpreted at runtime fits two use cases that are important to me perfectly, those being testing and mapping/modding.

There was a major obstacle directly ahead, however. In order to fully integrate an interpreted language, something like Lua for example, with the engine, majority if not all of its interfaces, declared and implemented in C++, would have to be exposed to the interpreter as well. Otherwise, the usefulness of interpreted scripts would be vastly limited. I very much didn’t want to introduce a ton of glue code binding the two languages together, especially considering a ton of redundancy that would generate. Thank the heavens, I didn’t spend another couple weeks thinking of how to work around that problem, but rather simply downloaded the trunk of the GCC repository, compiled and plugged it into the build system, enabled the newest standard, added the -freflection flag and went to town.

The following is a very rough insight into how I managed to get automatic C++-to-Lua binding generation using the new and shiny reflection, which is coming in GCC 16, along with support for a lot of other features of the C++26 standard. The two fundamental references that I used throughout this endeavour were the original proposal, P2996, found here, and the user manual for sol2, a C++ library implementing a sensible idiomatic C++ interface on top of the Lua interpreter (either PUC Lua - the original - up to version 5.4, with trivial changes necessary for supporting 5.5, or LuaJIT 2.1), which can be found here.

I’m not sure exactly how to format this knowledge dump, so let me introduce you to basic concepts of the proposal in a form of a list of bullets:

• Most, if not all, things related to reflection, i.e. stuff found in the std::meta namespace, is consteval. consteval is a qualifier primarily used to mark functions and means that its result must be known at compile time; this stands in contrast with constexpr, which allows a function to be used both at runtime and at compile time. The two can be easily mixed up here, as in some cases values of some arguments may only be known at runtime (which they can’t be required to with consteval), or some values may unexpectedly be required to be calculated strictly at compile time. C++26 also standardized the concept of “poisoning” statements involving function calls to also become consteval (such expressions are known as immediate-escalating expressions). I don’t fully understand the nature of some of the errors related to consteval that I have encountered along the way, but resolving them involved a slight bit of refactoring and thinking about when each piece of information is available or needed exactly. At worst, mess around with adding consteval/constexpr here and there.
• The pain points arising from the aforementioned distinction become especially prominent when constructing wordy operations on reflected types. For a reason that is unknown to me, you can’t explicitly require function arguments to be consteval - this means that you can’t pass intermediate processing results between different helper functions easily. At best, you can use (capturing) lambdas.
• The proposal introduces a new reflection operator, ^^, replacing ^ and reflexpr from previous experiments, as well as the splicing operator [: X :]. The reflection operator is used on types (and any named construct, for that matter!) to be reflected as arguments to std::meta functions. For example, if one were to want to obtain a list of all members of a particular class or structure, one could invoke std::meta::members_of(^^T, std::meta::access_context::current()).
• The proposal also defines a structure, std::meta::info, that to my knowledge is implementation-specific and I’m not sure if it can be inspected. Most std::meta functions return either one instance of this structure or a vector of them, for iteration. Each such structure refers to, or serves as a descriptor of a specific language construct, e.g. a value, a variable, a function, a type, etc.
• In order to refer to this construct as an {l,r}value, one can use the splicing operator, [:x:], which the compiler will then substitute for the construct being referred to.
• While vectors of std::meta aren’t themselves consteval, their contents need to be known at compile time in order to be able to generate code based on the information from reflection. For this purpose, another proposal introduced a mechanism of define_static_array, which, when passed such a quasi-compile-time vector, creates a proper array that then can be iterated over using a new language syntax construct, template for. It intuitively works like a regular for loop, except it generates the body of the loop for each instance of the std::meta structure being iterated over (and only permits the container interation syntax).
• Some of the std::meta query methods also allow specifying the access context in which the query should take place, meaning that one can limit the retrieved list of constructs to include only those accessible from the call site (e.g. only public ones if invoked from outside of a structure/class/method definition), or all of them. Those are outlined in the proposal.
• P2996 is, in general, written in a half-referential, half-insightful manner, without any serious language legalese. I highly recommend going ahead and reading it yourself.

All these in tandem form a surprisingly expressive, if a somewhat verbose, mechanism for generating code targetting specific parts of a particular type, based on just its identifier. For example, one can now obtain a list of fields and methods of a structure, iterate over them, obtain their addresses or offsets within said structure and create bindings based off that.

Sounds exactly like my use case.

(Notabene: One major caveat that was explicitly not addressed in the original proposal is reflection of constructors and destructors. There’s more “magic” involved with them than with regular methods, so the authors of the proposal have decided to work on resolving their remaining issues in scope of a future one - meaning, support for their reflection is coming, most likely, in C++29 at the earliest.)
(Another notabene: If you decide to use the trunk before GCC 16 is properly released and encounter linking errors to the tune of <insert_math_func_name_here>@GLIBC<version> being undefined, liberally sprinkle -lm in the linking arguments of your project and its dependencies. Do make sure that -lstdc++ is added there too and that the exact right collection of tools is used for compilation and linking - ensuring that the correct binaries of C++ tools outside of /usr are invoked can be a bit of a pain.)

A small snippet of the method I cooked up to perform the bindings follows. It uses sol2/sol3 for the purpose of interfacing with the Lua interpreter itself.

template<typename T, typename... Ctrs>
void RegisterType()
{
    // Create an instance of sol::constructors using the list of constructors 
    // from the template.
    constexpr auto solConstructors = sol::constructors<Ctrs...>();

    // Create a new usertype for T.
    constexpr std::string_view typeName = std::meta::identifier_of(^^T);
    sol::usertype<T>& newType = _lua.new_usertype<T>(typeName, solConstructors));

    // Map members of T to the new usertype.
    template for (constexpr auto member :
                  std::define_static_array(std::meta::members_of(
                      ^^T, std::meta::access_context::current())))
    {
        // Methods and non-static fields.
        if constexpr (std::meta::has_identifier(member)
                      && !std::meta::is_static_member(member)
                      && !std::meta::is_constructor(member)
                      && !std::meta::is_type(member)
                      && (std::meta::is_function(member)
                          || std::meta::is_nonstatic_data_member(
                              member)))
        {
            constexpr std::string_view memberName
                = std::meta::identifier_of(member);
            newType[memberName] = &[:member:];
        }

        // Anonymous unions.
        else if constexpr (std::meta::is_type(member)
                           && std::meta::is_union_type(member)
                           && !std::meta::has_identifier(member))
        {
            template for (constexpr auto datum :
                          std::define_static_array(
                              std::meta::nonstatic_data_members_of(
                                  member,
                                  std::meta::access_context::
                                      current())))
            {
                if constexpr (std::meta::has_identifier(datum)) {
                    constexpr std::string_view datumName
                        = std::meta::identifier_of(datum);
                    newType[datumName] = &[:datum:];
                }
            }
        }
    }
    
    // [...]
}

The snippet leaves out some bits pertaining to template discovery and binding operator overloads, which I think is a good exercise to leave for the reader to solve. In any case, it presents the basic mechanisms used for the purpose: the two new operators, the whole shebang with template for and define_static_array, as well as exactly how much of the code needs to be marked constexpr - it appears to me that certain if constexprs actually modify some internal compiler assumptions related to the constructs being reflected on and matching those assumptions directly is crucial to successful compilation (e.g. in the // Anonymous unions condition, try replacing is_union_type with is_complete_type - in the current trunk (as of the time of writing), this will trigger compilation errors complaining about reflected types not being complete classes, which makes no sense on the face of it). There are some what I believe to be temporary quirks in GCC’s implementation of the new features, but all in all everything is usable, if requiring a bit of elbow grease.

I think Herb Sutter referred to reflection as the new engine of the language that will be used to propel it into the next decade. After the last couple days of experimentation, I can’t agree more - while still far from the usability that e.g. Jai will or Zig offers now, if you are the kind of a person who does this for the love of the struggle, this is indeed a feature to beat.