build: speedup

[sshane]

@deanlee interesting in checking out why some objects take so long to build?

time g++ -v -ftime-report -o opendbc/can/dbc.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/threepilot/opendbc_repo/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.pyenv/versions/3.11.4/include/python3.11 opendbc/can/dbc.cc

real    0m10.028s
user    0m7.034s
sys     0m2.953s

Time variable                                   usr           sys          wall               GGC
 phase parsing                      :   0.91 ( 30%)   0.95 ( 52%)   1.86 ( 38%)  158925 kB ( 38%)
 phase lang. deferred               :   0.47 ( 15%)   0.29 ( 16%)   0.76 ( 16%)   82164 kB ( 20%)
 phase opt and generate             :   1.61 ( 52%)   0.57 ( 31%)   2.18 ( 44%)  171455 kB ( 41%)
 |overload resolution               :   0.41 ( 13%)   0.25 ( 14%)   0.59 ( 12%)   57574 kB ( 14%)
 template instantiation             :   0.71 ( 23%)   0.50 ( 28%)   1.25 ( 26%)  129017 kB ( 31%)

[deanlee]

I’ll look into the build performance issue later.

[deanlee]

Due to the inefficiency of std::regex, it often takes over 90% of compile time (e.g., 10 seconds on my laptop). Many discussions online highlight that std::regex is slow both to compile and at runtime. Aside from switching to a more efficient regex library, there are few good solutions.

[sshane]

Why does removing -g speed it up significantly?

[sshane]

Also, this only takes 3s with all the regex:

batman@workstation-shane:~/openpilot/opendbc_repo$ time g++ dbc_test.cc -g -O2

real    0m3.445s
user    0m3.187s
sys     0m0.246s

It appears just adding an empty regex in the real dbc.cc causes a 2x slowdown, and subsequent patterns take almost no extra compile time.

[deanlee]

Why does removing -g speed it up significantly?

std::regex generates a large amount of complex template code, leading to template bloat, which causes the compiler to take more time to process. This is especially true when the -g flag is enabled to generate debug information, as the compiler needs to produce detailed debug data for all the expanded templates.

[deanlee]

RE2 looks like a good alternative to std::regex, it's has minimal headers and relatively small lib compare to other reg lib.
https://github.com/google/re2

[adeebshihadeh]

Can we precompile the regex stuff and check it in? I think another library is not worth it (unless super small), but it might be worth having to check in a new pre-compiled header when it changes.

[deanlee]

Clang compiles template-heavy code like std::regex more efficiently than g++, potentially improving DBC compile times by up to 30%. However, the core issue of std::regex's complex template instantiations remains. Precompiling dbc.cc into platform-specific libraries (e.g., /precompiled/Darwin, /precompiled/arm64) may help but adds complexity to the build process and feels somewhat unusual.
A more straightforward solution could be replacing std::regex with a faster alternative like Google's RE2, which is optimized for both performance and faster compilation.