Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get build times under 1s #1528

Open
adeebshihadeh opened this issue Nov 26, 2024 · 3 comments · May be fixed by #1529
Open

Get build times under 1s #1528

adeebshihadeh opened this issue Nov 26, 2024 · 3 comments · May be fixed by #1529
Labels
enhancement good first issue

Comments

@adeebshihadeh
Copy link
Contributor

adeebshihadeh commented Nov 26, 2024 •
edited
Loading

I have a modern Threadripper with tons of cores - why is building not instant? Is it the Cython? Is it all the C++ regex?

opendbc is a small project, so working in it should feel very snappy and instant. We're open to any changes necessary to get to <1s fresh builds.

It's possible there's one silly thing taking all that time or it's a few things we have to fix. Some notes in #1180.

(opendbc) batman@workstation-adeeb:~/src/opendbc$ echo "int main(void) {return 0;}" > test.c; time g++ test.c

real    0m0.041s
user    0m0.016s
sys     0m0.025s

The goal is to get scons -c && time scons -j40 to report <1s (on my workstation, though it shouldn't be too different on other machines). Here's where we are now:

(opendbc) batman@workstation-adeeb:~/src/opendbc$ scons -j40
Cannot read termcap database;
using dumb terminal settings.
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o opendbc/can/common.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/common.cc
g++ -o opendbc/can/dbc.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/dbc.cc
g++ -o opendbc/can/parser.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/parser.cc
g++ -o opendbc/can/packer.os -c -std=c++1z -DDBC_FILE_PATH='"/home/batman/src/opendbc/opendbc/dbc"' -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -fPIC -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 opendbc/can/packer.cc
cythonize opendbc/can/packer_pyx.pyx
cythonize opendbc/can/parser_pyx.pyx
python3 opendbc/dbc/generator/generator.py
Compiling /home/batman/src/opendbc/opendbc/can/packer_pyx.pyx because it changed.
[1/1] Cythonizing /home/batman/src/opendbc/opendbc/can/packer_pyx.pyx
Compiling /home/batman/src/opendbc/opendbc/can/parser_pyx.pyx because it changed.
[1/1] Cythonizing /home/batman/src/opendbc/opendbc/can/parser_pyx.pyx
g++ -o opendbc/can/packer_pyx.o -c -std=c++1z -g -fPIC -O2 -Wunused -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -Wno-#warnings -Wno-shadow -Wno-deprecated-declarations -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 -I.venv/lib/python3.11/site-packages/numpy/_core/include opendbc/can/packer_pyx.cpp
g++ -o opendbc/can/parser_pyx.o -c -std=c++1z -g -fPIC -O2 -Wunused -Wshadow -Wno-vla-cxx-extension -Wno-unknown-warning-option -Wno-#warnings -Wno-shadow -Wno-deprecated-declarations -I. -I/usr/lib/include -I/home/batman/.local/share/uv/python/cpython-3.11.4-linux-x86_64-gnu/include/python3.11 -I.venv/lib/python3.11/site-packages/numpy/_core/include opendbc/can/parser_pyx.cpp
g++ -o opendbc/can/libdbc.so -shared opendbc/can/dbc.os opendbc/can/parser.os opendbc/can/packer.os opendbc/can/common.os -Lopendbc/can
g++ -o opendbc/can/parser_pyx.so -pthread -shared -Wl,-rpath=/home/batman/src/opendbc/opendbc/can opendbc/can/parser_pyx.o -Lopendbc/can -Lopendbc/can -ldbc
g++ -o opendbc/can/packer_pyx.so -pthread -shared -Wl,-rpath=/home/batman/src/opendbc/opendbc/can opendbc/can/packer_pyx.o -Lopendbc/can -Lopendbc/can -ldbc
scons: done building targets.
@LouisVelez
Copy link
Contributor

It seems most of the delay is on the build. I would suggest looking into pre-compiling if possible. Hopefully this helps anyone else looking into the issue.

Output of scons -u -j$(nproc) --debug=time on an M2 mac

Total SConsign sync time: 0.002967 seconds
Total build time: 2.888204 seconds
Total SConscript file execution time: 0.216717 seconds
Total SCons execution time: 0.050112 seconds
Total command execution time: 2.621375 seconds

@BBBmau BBBmau linked a pull request Nov 26, 2024 that will close this issue
Draft
@andrewchambers
Copy link

andrewchambers commented Nov 30, 2024 •
edited
Loading

@adeebshihadeh it looks like you intended to post the output of time scons or maybe better yet scons --debug=time -j40 but forgot to include the time in your copy paste.

@qedjdev
Copy link

qedjdev commented Dec 5, 2024

It seems most of the delay is on the build. I would suggest looking into pre-compiling if possible

target: a clean build (scons -c && ...) being <1s
precompiled headers still need to be compiled on a clean build. this doesn't save time for this target but it should still (maybe?) be looked into.

for anyone willing to dig through profiling data...

scons -Q --profile build_profile.pstats

import pstats
p = pstats.Stats('build_profile.pstats')
p.sort_stats('cumulative').print_stats()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement good first issue
Projects
openpilot bounties
Status: Open
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

get build <1s with CacheDir BBBmau/opendbc
4 participants
@andrewchambers @LouisVelez @adeebshihadeh @qedjdev