Take non-Send resources out of World

[maniwani]

This PR builds on top of #9202 (I separated what could be separated from this PR into that one).

Objective

The aim of this PR is to be the "final" fix to our messy problems involving thread-local data (see #6552 for context).

Bevy only deals with a handful of thread-local resources (that mostly come from low-level OS APIs or libs that talk to those), but that handful has resulted in a ton of internal complexity (in bevy_ecs, bevy_tasks, and bevy_app). The way we currently store them makes it either difficult or impossible to implement some nice-to-have features (e.g. generalized multi-world apps, framerate-independent input handling, etc.).

Solution

tl;dr Separate non-Send data from the ECS entirely. It will remain indirectly accessible through an event loop running on the main thread. Then, move app updates into a separate thread (on permitting targets) so they don't block the event loop.

This setup should be familiar to any audio plugin developers (or async runtime developers). If a system on thread B needs to do something with data that can't leave thread A, it can just send a command to A's event loop and wait for A to run it (and send any result back).

Some unique benefits of separating Send and !Send data like this are:

• Res and ResMut can be removed in favor of Ref and Mut since they're redundant without NonSend and NonSendMut.
• World becomes unambiguously Send, so a world can be dropped on any thread.
• The multi-threaded executor is able to run any system on any thread (including exclusive systems).
• The multi-threaded executor is able to run any system without spawning a task for it (especially exclusive systems).
• Any system from any world on any thread can remotely access the local resources in the main thread.
• Input events could be timestamped (with Instant::now()) with virtually no delay/aliasing (the event loop isn't blocked).

---

Changelog

TODO: finish this section

• NonSend and NonSendMut have been removed.
• Moved storage of non-Send resources to an external ThreadLocals container. It's normally owned by the App.

Migration Guide

If you were doing this before:

fn foo(x: NonSend<X>) {
    /* ... */
}

you're doing this now:

fn foo(mut thread: ThreadLocal) {
    // This closure will run on the thread that owns the TLS.
    // `run` blocks until the closure is completed.
    thread.run(|tls| {
        let x = tls.resource::<X>();
        /* ... */
    });
}

For better or worse, the handling that used to be buried inside the scheduler is now the responsibility of whoever writes an App runner. It's not too much boilerplate though IMO. Here's an example of a basic runner that runs the app once.

fn run_once(mut app: App) {
    let (mut sub_apps, tls, send, recv, _runner) = app.into_parts();
          
    // Move sub-apps to another thread.
    let thread = std::thread::spawn(move || {
        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
            sub_apps.update();
            send.send(AppEvent::Exit(sub_apps)).unwrap();
        }));
        
        if let Some(payload) = result.err() {
            send.send(AppEvent::Error(payload)).unwrap();
        }
    });
    
    // Run an event loop in this thread.
    loop {
        let event = recv.recv().unwrap();
        match event {
            AppEvent::RunTask(task) => {
                task();
            }
            AppEvent::Exit(_) => {
                thread.join().unwrap();
                break;
            }
            AppEvent::Error(payload) => {
                std::panic::resume_unwind(payload);
            }
        }
    }
}

Spawning a thread like this is basically mandatory unless you're targeting an environment (i.e. wasm32) or are enabling a bevy compiler flag that forces completely single-threaded system execution. Otherwise, not doing so will risk running into a deadlock.

Unfortunately, the channel between the main and app threads can't always be a std::sync::mpsc::channel. The winit runner spawns an event loop that can only be woken up by winit events, so we have to use their special EventLoopProxy sender. That's the reason for the ThreadLocalTaskSender trait. I still had to box the channel as a trait object because bevy_ecs and bevy_app are separated.

[hymm]

This is just a quick skim with comments on trying to shrink this pr a bit. Will dig more into the meat of the pr later.

[SkiFire13]

    // Insert channel resource into every sub-app.
    let (send, recv) = std::sync::mpsc::channel();
    sub_apps.main.world.insert_resource(ThreadLocalAccessor::new(&mut tls, send.clone());
    for sub_app in sub_apps.sub_apps.values_mut() {
        sub_app.world.insert_resource(ThreadLocalAccessor::new(&mut tls, send.clone());
    }

FYI this example is unsound, each &mut tls technically invalidate any pointer obtained from the previous &mut tls.

[hymm]

what's the advantage of the approach here vs building an abstraction around a global thread_local!()?

[maniwani]

The main benefit is this approach is siloed within an App. You could prepare multiple App instances on the same thread without them conflicting on the same global variable.

If thread_local! (or #[thread_local]) has any other notable caveats, they're not our problem. Maybe it'll have issues on console platforms and we just don't know about them yet.

Keep in mind that this setup is about making the same store of thread-local data "available on all threads". I think thread_local! could at most replace the mutex¹. The rest of the implementation wouldn't change much. We still need the channel, etc.

Footnotes

1. But this mutex is only used to ensure soundness. It cannot be contended, so any performance impact is negligible. ↩

[maniwani]

I changed it to use thread_local! after all.

This has an annoying caveat where we kind of need to drop the non-send resources before exiting the app (or the user/plugin needs to do it) because of how thread_local! handles destructors.

[alice-i-cecile]

I think we should swap this to the 0.13 milestone now. We're drawing close and this is still in draft.

[maniwani]

Let me see if I can have this ready for review this weekend. I've already rebased (just haven't pushed) and only the winit runner is unfinished (winit runner code is the bulk of this PR).