Hot-reloading a Bevy app

Development with Rust can be a lot of fun and the rich type system in combination with good tooling such as rust-analyzer can provide helpful hints and guidance continuously while writing code.

This, however, does not fully compensate for slower workflows that come from edit-compile-run cycles that require full program restarts plus the time it takes to recompile the entire program. Rust provides incremental compilation support that reduces the time for rebuilds often down to few seconds or for lightweight programs even less. But relying on, for example, macro-heavy code or types that make ample use of generics, compile times can go up quickly. This makes development with Rust quite different from lively environments and dynamic languages that might not just come with less build time but that might even provide feedback at the tip of a key.

Whether or not this impedes the ability to build software depends of course on what is built and on the approach a developer generally takes. For use cases such as games, UIs, or exploratory data analysis, quick feedback can certainly be an advantage — it might even allow you to make experiments that you would not have tried otherwise.

To enable such dynamic workflows when programming Rust, I have long wanted a tool that makes immediate feedback possible and hopefully easy to use. This blog post presents my take on the matter by introducing the hot-lib-reloader crate.

Table of Contents

• Where this idea comes from
• “Hello World” example
• Caveats and Asterisks
  • Global state
  • dlopen/dlclose behavior
  • Function signatures
  • Supported platforms
• How to use it then?
  • Use features to toggle hot reload support
  • If necessary, create indirections
  • How to use it with Bevy?
  • Hot-reload Template for cargo-generate
• Comments and feedback

Changelog

• 2022-08-15: Update for hot-lib-reloader v0.6
• 2022-08-09: Update for hot-lib-reloader v0.5

Where this idea comes from

None of this is new of course. Lots of languages and tools were created to allow code changes at runtime. “Live programming” in Smalltalk and Lisps is an essential part of those languages and which is enabled by reflection APIs the languages possess as well as by tools that¹. Also Erlang’s hot code swapping plays an important role for the reliability of Erlang programs — upgrading them is not an “exception”, it is well supported out of the box.

“Static” languages have traditionally much less support for the runtime code reload. An important part of it is certainly that optimizations become much harder to achieve when assembly code needs to support being patched up later². But even though modifying code arbitrarily is hard, dynamically loaded libraries are an old idea. Support for those is available in every (popular) operating system and it has been used to implement hot code reloading many times.

For C / C++, this mechanism has been made popular by the Handmade Hero series which presents this approach in detail (1, 2, 3). And also in Rust there have been posts and libraries around this idea (for example Hot Reloading Rust: Windows and Linux, An Example of Hot-Reloading in Rust, Live reloading for Rust, dynamic_reload).

Dynamic library (re-)loading is also what I am using in the approach presented here. The fundamental implementation is quite similar to the work linked above. Where this differs somewhat is in how you can interface with the reloader. By using a macro that can figure out the exposed library functions, it is possible to avoid a lot of boilerplate. In addition I will later show how this can be further customized to play nice in the context of frameworks such as Bevy.

“Hello World” example³

The simplest example will require a binary with some kind of main loop and a library that contains the code to be reloaded. Let’s assume the following project layout where the binary is the root package of a Cargo workspace and the library is a sub-package:

$ tree
.
├── Cargo.toml
└── src
│   └── main.rs
└── lib
    ├── Cargo.toml
    └── src
        └── lib.rs

The root Cargo.toml defines the workspace and the binary. It will depend on the hot-lib-reloader crate that takes care of watching the library and reloading it when it changes:

[workspace]
resolver = "2"
members = ["bin", "lib"]

[package]
name = "bin"
version = "0.1.0"
edition = "2021"

[dependencies]
hot-lib-reloader = "^0.6"
lib = { path = "lib" }

The library should expose functions and state. It should have specify dylib as crate type, meaning it will produce a dynamics library file such as liblib.so (Linux), liblib.dylib (macOS), and lib.dll (Windows). The lib/Cargo.toml:

[package]
name = "lib"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["rlib", "dylib"]

lib/lib.rs is providing a single public and unmangled function do_stuff that accepts some state parameter:

// When using the source_files: ["path/to/lib.rs"] auto discovery
// reloadable functions need to be public and have the #[no_mangle] attribute

pub struct State {
    pub counter: usize,
}

#[no_mangle]
pub fn do_stuff(state: &mut State) {
    state.counter += 1;
    println!("doing stuff in iteration {}", state.counter);
}

And finally src/main.rs where we define a sub-module hot_lib using the hot_module attribute macro:

// The value of `dylib = "..."` should be the library containing the hot-reloadable functions
// It should normally be the crate name of your sub-crate.
#[hot_lib_reloader::hot_module(dylib = "lib")]
mod hot_lib {
    // Reads public no_mangle functions from lib.rs and  generates hot-reloadable
    // wrapper functions with the same signature inside this module.
    hot_functions_from_file!("lib/src/lib.rs");

    // Because we generate functions with the exact same signatures,
    // we need to import types used
    pub use lib::State;
}

fn main() {
    let mut state = hot_lib::State { counter: 0 };
    // Running in a loop so you can modify the code and see the effects
    loop {
        hot_lib::do_stuff(&mut state);
        std::thread::sleep(std::time::Duration::from_secs(1));
    }
}

Above is the part that matters: The hot_lib module embeds a hot-reloadable version of do_stuff(&mut State). It has the same signature as the original defined in lib/lib.rs but it’s implementation will load it from the latest version of the dynamic library. This means we can just use it like the original function but it will be automatically updated when the library file changes! If you want to see the generated code that does all this you can run cargo expand.

OK, now we can actually run the app. Start two cargo commands (this assumes you have cargo-watch installed):

• cargo watch -w lib -x 'build -p lib' which will recompile the lib when it changes
• cargo run which will start the app with hot-reloading enabled

Now modify the do_stuff function, e.g. by changing the printed text or adding additional statements. You should see the output of the running application change immediately.

How it works

hot-lib-reloader uses the libloading crate which provides a unified and safe(r) interface around functions provided by the operating system to load and unload dynamic libraries (dlopen, dlclose, LoadLibraryEx, etc.). It watches the library file or files you specify using notify. When one of those changes it will mark the library as having a pending change. LibReloader::update will then use libloading to close the previously loaded library instance and load the new version. The module generated with the hot_module attribute macro above is a wrapper around a static instance of LibReloader::update. Actual calls to library functions are then made by looking up a symbol with the matching function name, assuming it resolves to an actual function with the signature assumed, and then calling it.

For various reasons the reloader will not actually load the library file that Rust created but will create a copy and load that. Among other things this prevents from holding a file lock on Windows, allowing the library file to change.

In addition to that, the hot_module macro takes care of some boilerplate. When declaring and using a reloader. A desugared version of the function do_stuff(&mut State) shown above will look like:

pub fn do_stuff(state: &mut State) {
    let lib_loader = __lib_loader();
    let lib_loader = lib_loader.read().expect("lib loader RwLock read failed");
    let sym = unsafe {
        lib_loader
           .get_symbol::<fn(&mut State)>(b"do_stuff\x00")
           .expect("Cannot load library function do_stuff")
    };
    sym(state)
}

In particular, without the macro it is necessary to keep the function signatures between where they are declared and where they are used in sync. To avoid that boilerplate, the macro can automatically implement the correct interface by looking at the exported function in the specified source files. In addition, LibReloader actively watches for file changes and runs lib updates in its own thread.

Caveats and Asterisks

This approach comes unfortunately with several caveats. The runtime of the program that the lib reloader operates in is a compiled Rust application with everything that involves. The memory layout of all types such as structs, enums, call frames, static string slices etc is fixed at compile time. If it diverges between the running application and the (re-)loaded library the effect is undefined. In that case the program will probably segfault but it might keep running with incorrect behavior. Because of that, all use of the reloader should be considered unsafe and I would highly advise against using it in situations where correctness and robustness of your application is required. This means, don’t use it for production environments or anything critical. The approach shown here is meant for development and fast feedback in situations where a crash is OK and nothing is at risk.

Global state

Because of that, what can actually be changed while maintaining the dynamic library re-loadable is limited. You can expose functions with signatures that shouldn’t change or static global variables. If exposing structs and enums, changing their struct fields is undefined behavior. Using generics across the dynamic library border isn’t possible either as generic signatures will always be mangled.

But state can get tricky in other ways as well. If global state is used by parts of a program that get reloaded, the state will no longer be accessible by the new code. This for example can be observed when trying to reload the macroquad / miniquad game frameworks. Those initialize and maintain OpenGL state globally. When trying to call reloaded code that uses the OpenGL context such as emitting render instructions, the program will panic. Even though it would be possible to externalize and transfer this state on reload and using the reloader with those frameworks won’t work, at least when directly accessing OpenGL related behavior. For strategies around this see How to use it then? below.

dlopen/dlclose behavior

Furthermore, as pointed out in this really great blog post from Amos the behavior of dlclose on Linux in combination with how Rust deals with thread local storage can actually prevent a library from properly unloading. Even though the approach presented here will actually allow you to load the changed behavior into your program, memory leaks cannot be prevented. So while you reload on Linux you will leak some (tiny) amount of memory on each library update.

Function signatures

The function names of reloadable functions need to be not mangled, so a #[no_mangle] attribute is mandatory. This in turn means that functions cannot be generic (with the exception of lifetimes). Assuming the rustc version of the code that compiles the binary and library are identical, the functions can be loaded even if not declared with extern "C". This in turn means that passing “normal” Rust types works fine. I do not know enough about how the Rust compiler chooses the memory layout of structs and enums to know if the same (non-generic) struct / enum definition can be laid out differently between compilations when the code remains unchanged. I assume not but if you run into problems making using extern "C" and #[repr(C)] might be worth trying.

Supported platforms

The current version of hot-lib-reloader has been tested successfully on Linux, Windows, and Intel macOS.

How to use it then?

2022-08-15: Also check the usage tips section of the hot-lib-reloader readme.

The above mentioned limitations restrict the possibilities of what changes can be made to running code.

What typically works well is to expose one or a few functions whose signatures remain unchanged. Those can then be called from throughout your application. If you want to split up your code further, using multiple libraries and reloaders at the same time is no problem. The reloaders should regularly check if watched libraries have changed, for that purpose they provide an update method that returns true if the library was indeed reloaded. This gives you control over when reloads should happen and also allows you to run code to e.g. re-initialize state if the library changed.

For example, for developing a game a simple approach would be to expose a update(&mut State) function that modifies the game state and a render(&State) function that takes care of rendering that. Inside the game loop you call the update function of the lib reloader to process code changes and optionally re-initialize the game state.

For frameworks such as Bevy where the main loop is controlled elsewhere, you can normally register callbacks or event handlers that will be invoked. For dealing with Bevy systems in particular, see below.

Use features to toggle hot reload support

Since only few parts of the application need to differ between a hot reloadable and a static version, using Rust features is the recommended way to selectively use hot code reloading and to also produce production code without any dependencies on dynamic library loading.

The reload-feature example shows how you can modify the minimal code example presented above to achieve this.

If necessary, create indirections

In cases like miniquad where global state prevents reloading a library you can fall back to creating an indirection. For example if you want to have a render function that can be changed, you could use a function that decoratively constructs shapes / a scene graph that then gets rendered in the binary with calls that depend on OpenGL.

When you have full control over the main loop

Also, frameworks that follow a more “functional” style such as Bevy can be supported. For Bevy in particular there exists specific support in hot-lib-reloader, see the next section for details.

How to use it with Bevy

Bevy manages the programs main loop by default and allows the user to specify functions (which represent the “systems” in Bevy’s ECS implementation) which are then invoked while the game is running. It turns out this approach allows fits a hot-reloading mechanism really well. The system functions can be seamlessly provided by either a statically linked or dynamically loaded library. All input they need they can specify by function parameters. Bevy’s type-based “dependency injection” mechanism will then make sure to pass the right state to them. In addition, Bevy’s resource system is a good fit to manage the lib reloader instance. A separate system can deal with the update cycle.

You can find a ready-to-use example in the examples section of hot-lib-reloader. I will go through it step by step below.

The project structure can be identical to the example show above. We will rename lib to systems to make our intentions explicit:

$ tree
.
├── Cargo.toml
├── src
│   └── main.rs
└── systems
    ├── Cargo.toml
    └── src
        └── lib.rs

Bevy app binary

src/main.rs then defines a systems_hot module which loads the functions provide by systems/src/lib.rs that would normally be available through the systems module. In particular, systems and systems_hot are 1:1 replaceable when you export the static items from systems, like we do below by adding use systems::*; inside mod systems_hot {/**/}. A reload feature is used to allow switching between static and hot-reload version.

use bevy::prelude::*;

#[cfg(not(feature = "reload"))]
use systems::*;
#[cfg(feature = "reload")]
use systems_hot::*;

#[cfg(feature = "reload")]
#[hot_lib_reloader::hot_module(dylib = "systems")]
mod systems_hot {
    use bevy::prelude::*;
    use systems::*;
    hot_functions_from_file!("systems/src/lib.rs");
}

We can then define the main function that does not differ from other Bevy apps.

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_startup_system(setup)
        .add_system(player_movement_system)
        .add_system(bevy::window::close_on_esc)
        .run();
}

Bevy systems

The two systems used in the example that are located in systems/src/lib.rs are run-of-the-mill bevy system functions that I’ll abbreviate here to save some room. The only notable piece here is the #[no_mangle] attribute of player_movement_system — and the absence of such an attribute with setup.

pub fn setup(mut commands: Commands) { /*...*/ }

#[no_mangle]
pub fn player_movement_system(
    keyboard_input: Res<Input<KeyCode>>,
    mut query: Query<&mut Transform, With<Player>>,
) { /*...*/ }

Only public functions with #[no_mangle] will be recognized as reloadable functions by the hot_module’s hot_functions_from_file! macro above.

To build both the executable and reloadable systems library you can start two cargo processes. Building the library:

# watch ./systems/, build the library
$ cargo watch -w systems -x 'build -p systems'

Building the executable (for the reason why Windows needs special treatment see this note):

# watch everything but ./systems and run the app
$ cargo watch -i systems -x 'run --features reload'
# Note: on windows use
$ env CARGO_TARGET_DIR=target-bin cargo watch -i systems -x 'run --features reload'

Using cargo watch for the excutable is not strictly necessary but it allows you to quickly change it as well when you make modifications that aren’t compatible with hot-reload.

Hot-reload Template for cargo-generate

To quickly setup new projects that come with the stuff presented here out of the box, you can use a cargo-generate template available at: https://github.com/rksm/rust-hot-reload.

Run cargo generate rksm/rust-hot-reload to setup a new project.

Comments and feedback

I just started to use hot-lib-reloader and so far I am surprised how well it works. It is not bringing Rust on-par with Lisp or Smalltalk but with code that requires lots of twiddling and changes it is enjoyable to use! Let me know what your impressions are either directly or via the discussions of this article on

• Hackernews
• Reddit
• Lobsters

Also, if you find bugs or run into problems I’m watching the Github issues.

Over and out.