Add profiler option to SCons builds, with support for tracy and perfetto.

[Ivorforce]

This is a minimal implementation to support the tracy profiler and the perfetto profiler.

• Related to Integrate Perfetto Profiler into Godot Engine godot-proposals#5346

If this is accepted, future PRs could expand the granularity of frame tracking to include the various servers, steps etc.
Future PRs could also add support for other profilers (e.g. Perfetto).

Why?

Tracy is a popular profiler for games and other "real time" software. "Profiling" means tracking performance over time, to try to figure out the cause of lag spikes, low frame rates, high RAM use, and others. The ultimate goal of course being to eliminate those problems.

It is already possible to profile Godot games. For one, Godot has a small profiler built-in. However, it's lacking features for serious profiling tasks, especially to profile c++ code.

There are guides to use external C++ profilers on the docs as well. These are capable of a lot, but there are features offered by tracy that are not supported by other profilers - for example per-frame profiling, which requires injection of a high precision callback to be worthwhile.

Here are the supported features per platform, stolen from the current tracy manual:

Alternatives

I am not familiar with any profilers except Instruments (including tracy). This PR is a first step to get familiar with them.
The idea is to establish a framework that allows injection of any useful intrusive profilers. For example, Perfetto may be integratable too, if desired by someone.

It is currently also possible to integrate tracy support with the Godot Tracy Module (cc @AndreaCatania). This is a great effort, and I've referenced this for my implementation.
The reason I'm still proposing to integrate this into the engine itself is that intrusive changes to the codebase are required to support frame tracking, which is arguably the most useful feature of tracy. It would be nice for the engine to 'just' support this.

Implementation notes

This is the first module I am working with / proposing. Chances are the approach is suboptimal. Please suggest improvements! :)

In particular, my idea was that you could include profiling.h and use generic macros to mark things to profile, regardless of the actual profiler used (in assumption that others work similar to tracy in this regard).

License

Tracy uses the BSD license, which is compatible with ours (thanks @AThousandShips and @Calinou!)

[AThousandShips]

You need to add this to COPYRIGHT.md and to thirdparty/README.md

The BSD license is compatible, we use several libraries with it, just needs to be attributed

[AThousandShips]

This would also be best tracked in a dedicated proposal, especially to compare alternatives and options

[Ivorforce]

This would also be best tracked in a dedicated proposal, especially to compare alternatives and options

Yeah, agreed. I wanted to implement it first to get a feeling for the amount of effort required, and to try it out directly with Godot. I'll likely write up a proposal once I get a better feeling for the features of tracy.

[clayjohn]

This is awesome! I actually looked at integrating Tracy earlier this month, but then decided not to continue because of all the required build system stuff to make it work.

Ideally this module would function in a profiler agnostic way. Perfetto is really nice and is also open source (its only 2 files IIRC so may be a better fit for a default profiler). A lot of people also like Pix which is Microsoft's tool for PC/Xbox.

I have discussed with a few people over the last year or so the need to be able to drop in a profiler (or have an integrated profiler with a way to easily replace markers with something else). That way we can have timestamps/markers that stay in the engine and we can track performance over time, and users of the profilers don't need to constantly re-add markers with every engine update.

[Calinou]

Tracy uses the BSD license.
I think it's compatible, but I'm no expert. So please weigh in if you know for sure!

Yes, 3-clause BSD is compatible 🙂

[kiroxas]

I do not see a way to annotate code that would not be scoped. Scope block do 90% of the work when you annotate code, but sometime you want to annotate portions of code that you cannot easily scope (it declares variables that will be used outside of the scope for example), and then you need to change the function to be able to annotate it correclty, which can change the behaviour of what you are profiling. You should not have to change surrounding code to profile it, this is why we often want to be able to add manual scope (ie BEGIN_ZONE / END_ZONE) manually for those cases. So I wonder if this is possible using Tracy, or we should at least have stub functions to be able to do it, because if people start using it, it will come up.

[Ivorforce]

BEGIN_ZONE / END_ZONE

I was looking for something like this, I haven't found it in tracy so far. Currently, I'm just wrapping code in plain scopes, which will not change function behavior, but it does incur some (perhaps unnecessary) whitespace changes. At the same time, usually you'll just want to annotate a full function, so it's probably fine?

It may also be possible to create manual start and end macros for tracy by wrapping its RTTI behavior and manually destructing the object. Perhaps even REPLACE_ZONE for as a convenient end + begin.

[kisg]

As the original author of the related proposal (and a 3.x perfetto-only version) thank you for your work on this! I hope it gets merged soon for everyone's benefit.

[Calinou]

The SCons option descriptions we have elsewhere don't end with a period:

Suggested change

	"Profiler framework to use compile into the binary.",
	"Profiler framework to compile into the binary (leave empty to disable)",

[Calinou]

Suggested change

	"Profile random samples application-wide using a callstack based sampler.",
	"Profile random samples application-wide using a callstack based sampler",

[Calinou]

While compiling with a symlink, I got this warning:

WARNING: Failed to redirect "/home/hugo/Documents/Git/wolfpld/tracy/public/TracyClient.linuxbsd.editor.x86_64.o"

I double-checked my symlink resolves correctly. The file exists and is 2.6 MB large.

I've never used Tracy before, so I don't know how you start its GUI on Linux. I've tried self-compiling it via CMake and the default options, but I don't see any program binary being compiled, only a static library (or a dynamic library when Godot compiles Tracy from SCons). The manual isn't really explicit on how you're supposed to start a profiling-enabled session, collecting data then viewing it. I'm much more used to the HotSpot workflow in comparison.

PS: Should we enable Tracy support in official binaries? As I understand it, debug symbols are not a strict requirement for Tracy to work (even if it's not as precise without debug symbols). This would allow users to troubleshoot some performance issues without needing to self-compile Godot.

[Ivorforce]

While compiling with a symlink, I got this warning:

WARNING: Failed to redirect "/home/hugo/Documents/Git/wolfpld/tracy/public/TracyClient.linuxbsd.editor.x86_64.o"

Oh, interesting!
I suppose this is somewhat of a regression of #101641 (cc @Repiteo). I think it should be fixable if we don't resolve symlinks in emitters.

I've never used Tracy before, so I don't know how you start its GUI on Linux. I've tried self-compiling it via CMake and the default options, but I don't see any program binary being compiled, only a static library (or a dynamic library when Godot compiles Tracy from SCons).

Yea, I had trouble compiling it from scratch too. In the end I just used brew install tracy. Perhaps there's pre-built binaries for Linux somewhere too?

The manual isn't really explicit on how you're supposed to start a profiling-enabled session, collecting data then viewing it. I'm much more used to the HotSpot workflow in comparison.

Once you have the GUI it's pretty easy; you just compile with scons profiler=tracy and run Godot. On the tracy side, you just run tracy, and in the GUI, click the connect button.

PS: Should we enable Tracy support in official binaries? As I understand it, debug symbols are not a strict requirement for Tracy to work (even if it's not as precise without debug symbols). This would allow users to troubleshoot some performance issues without needing to self-compile Godot.

Technically tracy is so low overhead that it would probably be OK to have it in official builds by default. On the other hand, there's cons like needing to include tracy source code after all, matching the tracy version, and some people would probably argue perfetto should be the default.
At the very least it's probably something we'd want to discuss after having the framework in place.

[Calinou]

Perhaps there's pre-built binaries for Linux somewhere too?

Unfortunately, upstream isn't willing to build redistributable Linux binaries and the Linux packaging situation is pretty dire (no Fedora or Ubuntu LTS package). That said, if the build instructions were more self-explanatory, I'd be able to build it from source just fine.

[Repiteo]

Shoot, I didn't even consider symlink shenanigans in the emitter. I've been meaning to revisit the emitter anyway to remove hardcoded paths, but now I have a real motivator!

[Ivorforce]

Perhaps there's pre-built binaries for Linux somewhere too?

Unfortunately, upstream isn't willing to build redistributable Linux binaries and the Linux packaging situation is pretty dire (no Fedora or Ubuntu LTS package). That said, if the build instructions were more self-explanatory, I'd be able to build it from source just fine.

The build instructions are said to be in the tracy manual. I think the folder you want to build is profiler, described in a section called "How to use CMake".

I was able to build it piecing the info together with homebrew's configuration:

cmake -B profiler/build -S profiler -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON  -DCMAKE_INSTALL_PREFIX=build
cmake --build profiler/build --config Release --parallel
cmake --install profiler/build

... or at least, almost, since I'm getting linker errors because it didn't build alongside glfw. But perhaps you'll be more lucky :)

Edit: I was actually able to build it now, using the GitHub workflow as reference! Try this:

cmake -B profiler/build -S profiler -DCMAKE_BUILD_TYPE=Release -DNO_ISA_EXTENSIONS=1
# -DNO_ISA_EXTENSIONS=1 May not be required for you
cmake --build profiler/build --parallel --config Release
profiler/build/tracy-profiler

[Calinou]

@Ivorforce Thanks! That works as long as I also add -DLEGACY=1 to the first cmake command, so that it can start on X11 (otherwise, only Wayland support is enabled).

[Calinou]

It works now, but I get a link error if I build without profiler=tracy after building with it once:

scons: Reading SConscript files ...
SCU: Generating build files... (max includes per SCU: 8)
Using linker program: mold
WARNING: System-provided icu4c or harfbuzz cause known issues for GDExtension (see GH-91401 and GH-100301).
Building for platform "linuxbsd", architecture "x86_64", target "editor".
Checking for C header file mntent.h... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
Linking Program bin/godot.linuxbsd.editor.x86_64 ...
mold: error: undefined symbol: tracy::GetProfilerv
>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced 63 more times

mold: error: undefined symbol: tracy::rpmallocm
>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(tracy::tracy_malloc(unsigned long))>>> referenced by scu_servers_rendering_1.gen.cpp
>>>               bin/obj/servers/libservers.linuxbsd.editor.x86_64.a(scu_servers_rendering_1.gen.linuxbsd.editor.x86_64.o):(tracy::tracy_malloc(unsigned long))>>> referenced by main.cpp
>>>               bin/obj/main/libmain.linuxbsd.editor.x86_64.a(main.linuxbsd.editor.x86_64.o):(tracy::tracy_malloc(unsigned long))>>> referenced 2 more times

mold: error: undefined symbol: tracy::InitRpmallocv
>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(tracy::tracy_malloc(unsigned long))>>> referenced by scu_servers_rendering_1.gen.cpp
>>>               bin/obj/servers/libservers.linuxbsd.editor.x86_64.a(scu_servers_rendering_1.gen.linuxbsd.editor.x86_64.o):(tracy::tracy_malloc(unsigned long))>>> referenced by main.cpp
>>>               bin/obj/main/libmain.linuxbsd.editor.x86_64.a(main.linuxbsd.editor.x86_64.o):(tracy::tracy_malloc(unsigned long))>>> referenced 2 more times

mold: error: undefined symbol: tracy::GetTokenv
>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(OS_LinuxBSD::runv)>>> referenced 177 more times

mold: error: undefined symbol: tracy::rpfreePv
>>> referenced by os_linuxbsd.cpp
>>>               bin/obj/platform/linuxbsd/os_linuxbsd.linuxbsd.editor.x86_64.o:(tracy::tracy_free_fast(void*))
collect2: error: ld returned 1 exit status
scons: *** [bin/godot.linuxbsd.editor.x86_64] Error 1
scons: building terminated because of errors.

Using scons -c fixes the issue, but it ought not to be required.

Using a build with Tracy enabled is slightly slower to start/shut down, even if the Tracy profiler isn't running in the background:

❯ hyperfine -iw1 -m25 "bin/godot.linuxbsd.editor.x86_64 --path /tmp/4 -e --quit" "bin/godot.linuxbsd.editor.x86_64.tracy --path /tmp/4 -e --quit"
Benchmark 1: bin/godot.linuxbsd.editor.x86_64 --path /tmp/4 -e --quit
  Time (mean ± σ):      6.005 s ±  0.303 s    [User: 4.634 s, System: 0.607 s]
  Range (min … max):    5.206 s …  6.584 s    25 runs
 
Benchmark 2: bin/godot.linuxbsd.editor.x86_64.tracy --path /tmp/4 -e --quit
  Time (mean ± σ):      6.487 s ±  0.200 s    [User: 4.791 s, System: 0.772 s]
  Range (min … max):    5.548 s …  6.607 s    25 runs
 
Summary
  bin/godot.linuxbsd.editor.x86_64 --path /tmp/4 -e --quit ran
    1.08 ± 0.06 times faster than bin/godot.linuxbsd.editor.x86_64.tracy --path /tmp/4 -e --quit

This is a debug build, but still, that may be a good reason not to enable profiling in official builds.

[Ivorforce]

It works now, but I get a link error if I build without profiler=tracy after building with it once:

I noticed that too. I think it's because of caching / dependencies of generated files. Gotta figure that out before merging.

[kisg]

Overall, I think this PR is a great first step in properly instrumenting Godot to easily gather detailed profiling data.

With the changes I propose we were already able to find a possible performance bottleneck in an Android project we are working on.

[kisg]

My suggestion would be to change the function prototype as follows:

void godot_init_profiler(int argc, char **argv);

In the implementation of the function the selected profiling system should only be enabled, if a "--enable-profiler" or similar command line parameter is passed to Godot. This way (at least in the case of Perfetto) production builds of the engine can be delivered to end users, because when profiling is not enabled, the performance impact is negligible.

Note, that I don't propose to include Perfetto or Tracy in the official builds, only if someone is building a custom release build of the engine, they can easily include profiling functionality.

[Ivorforce]

Generally I agree. However, I think it's out of scope to do this now, so I'd rather keep it for a follow-up PR.

[Ivorforce]

Many thanks to @kisg!
I've pushed the suggested changes. Hopefully, perfetto works now (i'm still not able to test myself). The results are also a bit more detailed:

I've also fixed the profiler module not realizing when the profiler changes, to re-generate profiling.gen.h. This should fix the problem where profiling.gen.h has to be manually deleted to switch profiler configurations.

[ze2j]

Supporting Tracy would be a great addition to the engine. I integrated Tracy into my fork about a year ago, and it’s been incredibly helpful for optimizing my game — and even the engine itself.

However, since Tracy is statically linked to the executable, it can’t be used to profile GDExtension written in C++, even though they could benefit from it as well. In my custom fork, I built Tracy as a shared library and linked it both to the game executable and to the GDExtension shared library (as explained by Tracy's documentation). It works, but the setup is a bit more involved (LD_LIBRARY_PATH...) compared to this PR.

@Ivorforce Do you plan to support GDExtension as well? I’m concerned that a module-based implementation might make it difficult — or even impossible. I know modules can be built as shared libraries, but I’m not sure if linking a module to a GDExtension would be possible or even make sense.

[Ivorforce]

Supporting Tracy would be a great addition to the engine. I integrated Tracy into my fork about a year ago, and it’s been incredibly helpful for optimizing my game — and even the engine itself.

However, since Tracy is statically linked to the executable, it can’t be used to profile GDExtension written in C++, even though they could benefit from it as well. In my custom fork, I built Tracy as a shared library and linked it both to the game executable and to the GDExtension shared library (as explained by Tracy's documentation). It works, but the setup is a bit more involved (LD_LIBRARY_PATH...) compared to this PR.

@Ivorforce Do you plan to support GDExtension as well? I’m concerned that a module-based implementation might make it difficult — or even impossible. I know modules can be built as shared libraries, but I’m not sure if linking a module to a GDExtension would be possible or even make sense.

That's some excellent insight! I haven't actually planned this far :)

The main idea of this PR is to establish an interface between Godot and a profiler (i.e. the macros). It should be possible to 'hot swap' implementations to something more sophisticated afterwards.
The main reason I'm using a module now is just to have a nice, isolated way to enable / disable it and to encapsulate logic. I'm very open to moving it elsewhere though (now that you're saying it, it probably makes more sense as a SCSub?).

I think using a shared library for tracy (for GDExtension compatibility) would make for a great follow-up PR.

[Ivorforce]

I've moved the profiler to an SCSub inside main/profiling, as (loosely) suggested above. I think this is more appropriate than the previous solution (as a module), since the profiler integrates tightly into Godot, and isn't very isolatable - especially if we try to consolidate this new profiling code with what's floating around the codebase already.

[enetheru]

we often want to be able to add manual scope (ie BEGIN_ZONE / END_ZONE) manually for those cases.

BEGIN_ZONE / END_ZONE

I was looking for something like this, I haven't found it in tracy so far.

These can be found in the TracyC.h , I was using them in my fork.
There is also 'Fibers' for coroutine like things.

I do not see a way to annotate code that would not be scoped.

You can just add messages anywhere, the documentation details all the options for client markup under section 3

Is it technically possible to allow declaring Tracy sections from the scripting API, so you can profile GDScript/C# code (or at least the C++ side of the methods being called within that block)?

Trivially at least for gdscript, I have previously waded through all the macro's in the gdscript module and created scopes with names taken directly from the scripts themselves. It was not immediately straight forward given i didnt know the internals of gdscript when i started , and i'm not sure if I'm leaking strings.

so it may even be an option to profile all GDScript functions 'by default' (when using a tracy profile build). An interface could still be used to profile specific subsections of a function.

indeed it is possible.

The manual isn't really explicit on how you're supposed to start a profiling-enabled session, collecting data then viewing it.

#Ifdef TRACY_ENABLE then it's always collecting data unless ON_DEMAND is flagged, which makes it only profile once the monitor is attached missing early information. once the monitor is detached it stops profiling and cannot be restarted, there would need to be application logic to change that.

Should we enable Tracy support in official binaries?

I would recommend against it, due to it collecting data if enabled, you will slowly see your RAM fill up without a monitor attached to pull all the events or some network client to send it to.

my fork if anyone is interested in the automatic gdscript annotation, I also got threads, memory, and source capture for c++ or whatever working too. I was eventually going to get around to using fibers for gdscript coroutines, but havent had a need to work on it for a while. and i cant vouch for the quality of my code in this fork.

[enetheru]

Here's an example of using the TracyCZone macros to begin and end a zone, with a name change, in this case tzc_name was defined earlier by me pulling it out of the gdscript vm at runtime. which I have no idea if it will work in the future.

TracyCZone( tzc, "OPCODE_CALL[_ASYNC,_RETURN]");
TracyCZoneName(tzc, tzc_name, tz_name.size() );
TracyCZoneEnd(tzc);

[deralmas]

Hi, I gave this PR a spin to profile my Wayland embedder branch with tracy. It works great! Instrumented profiling is a really nice addition, thank you :D

One small thing though, I personally needed to apply this patch to make it build:

--- a/main/profiling/SCsub
+++ b/main/profiling/SCsub
@@ -6,6 +6,7 @@ import pathlib
 import methods
 
 Import("env")
+Import("env_modules")
 
 opts = Variables([], ARGUMENTS)
 opts.Add(

As it otherwise complained that it had no idea what env_modules was. No idea if this is the correct thing to do, just letting you know.