Skip to content

[BOLT] Force frame pointers off for runtimes #148009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

peterwaller-arm
Copy link
Contributor

@peterwaller-arm peterwaller-arm commented Jul 10, 2025

Distributions are making the choice to turn frame pointers on by default. Nixpkgs recently turned them on, and the method they use to do so implies that everything is built with them on by default.

NixOS/nixpkgs#399014

Assuming that a well behaved distribution doing this puts -fno-omit-frame-pointer at the beginning of the compiler invocation, we can still re-enable omission by supplying -fomit-frame-pointer during compilation.

This fixes some segfaults from stack corruption in binaries rewritten by bolt with llvm-bolt -instrument.

See also: #147569
Fixes: #148595

@llvmbot
Copy link
Member

llvmbot commented Jul 10, 2025

@llvm/pr-subscribers-bolt

Author: Peter Waller (peterwaller-arm)

Changes

Distributions are making the choice to turn frame pointers on by default. Nixpkgs recently turned them on, and the method they use to do so implies that everything is built with them on by default.

NixOS/nixpkgs#399014

Assuming that a well behaved distribution doing this puts -fno-omit-frame-pointer at the beginning of the compiler invocation, we can still re-enable it by supplying -fomit-frame-pointer during compilation.

See also: #147569


Full diff: https://github.com/llvm/llvm-project/pull/148009.diff

1 Files Affected:

  • (modified) bolt/runtime/CMakeLists.txt (+4-1)
diff --git a/bolt/runtime/CMakeLists.txt b/bolt/runtime/CMakeLists.txt
index 87cc44812da11..4c496c2b800ef 100644
--- a/bolt/runtime/CMakeLists.txt
+++ b/bolt/runtime/CMakeLists.txt
@@ -35,7 +35,10 @@ set(BOLT_RT_FLAGS
   -fno-exceptions
   -fno-rtti
   -fno-stack-protector
-  -fPIC)
+  -fPIC
+  -mgeneral-regs-only
+  -fomit-frame-pointer # Runtime currently assumes omitted frame pointers. llvm/llvm-project#147569
+)
 if (CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
   set(BOLT_RT_FLAGS ${BOLT_RT_FLAGS} 
     -mno-sse 

@peterwaller-arm
Copy link
Contributor Author

extern "C" __attribute((naked)) void __bolt_instr_indirect_call()

I currently suspect the issue is arising from void __bolt_instr_indirect_call() or related functions, which are marked __attribute((naked)). Naively I would have expected this to omit the instructions for preserving the frame pointer, but that's not what I see:

0000000000000000 <__bolt_instr_indirect_call>:
       0:       a9bf7bfd        stp     x29, x30, [sp, #-16]!
       4:       910003fd        mov     x29, sp
       8:       a9bf07e0        stp     x0, x1, [sp, #-16]!
       c:       a9bf0fe2        stp     x2, x3, [sp, #-16]!
      10:       a9bf17e4        stp     x4, x5, [sp, #-16]!
      14:       a9bf1fe6        stp     x6, x7, [sp, #-16]!
      18:       a9bf27e8        stp     x8, x9, [sp, #-16]!

Mentioning it in case anyone has an idea what's happening there, it looks like it relates to -mno-omit-leaf-frame-pointer, which nixpkgs is also supplying, -fno-omit-frame-pointer does not appear to be enough to cause the problem by itself. However, -fomit-frame-pointer does negate the issue.

@nikic
Copy link
Contributor

nikic commented Jul 11, 2025

@peterwaller-arm It looks like GCC just doesn't support naked functions on AArch64 at all, see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77882.

@peterwaller-arm peterwaller-arm force-pushed the bolt-rt-force-frame-pointers-off branch from 27767c4 to 8d5cb4d Compare July 11, 2025 11:32
Copy link
Contributor

@nikic nikic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me, but a bolt maintainer should approve as well.

Copy link
Contributor

@maksfb maksfb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

Copy link
Member

@paschalis-mpeis paschalis-mpeis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Peter!

Distributions are making the choice to turn frame pointers on by
default. Nixpkgs recently turned them on, and the method they use to do
so implies that everything is built with them on by default.

NixOS/nixpkgs#399014

Assuming that a well behaved distribution doing this puts
'-fno-omit-frame-pointer' at the beginning of the compiler invocation,
we can still re-enable it by supplying -fomit-frame-pointer thereafter.

See also: llvm#147569
Fixes: llvm#148595
@peterwaller-arm peterwaller-arm force-pushed the bolt-rt-force-frame-pointers-off branch from 8d5cb4d to 01424e2 Compare July 14, 2025 09:51
@peterwaller-arm peterwaller-arm merged commit 81ad8e6 into llvm:main Jul 14, 2025
9 checks passed
@peterwaller-arm peterwaller-arm deleted the bolt-rt-force-frame-pointers-off branch July 14, 2025 10:18
@llvm-ci
Copy link
Collaborator

llvm-ci commented Jul 14, 2025

LLVM Buildbot has detected a new failure on builder premerge-monolithic-linux running on premerge-linux-1 while building bolt at step 7 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/153/builds/37967

Here is the relevant piece of the build log for the reference
Step 7 (test-build-unified-tree-check-all) failure: test (failure)
...
PASS: lld :: COFF/arm64-thunks.s (98411 of 101544)
PASS: cfi-standalone-lld-x86_64 :: cross-dso/icall/diag.cpp (98412 of 101544)
PASS: lit :: lit-opts.py (98413 of 101544)
PASS: lld :: COFF/arm-thumb-thunks.s (98414 of 101544)
PASS: lld :: COFF/arm64ec-exports.s (98415 of 101544)
PASS: lld :: COFF/arm64ec-cust-export-thunk.s (98416 of 101544)
PASS: lld :: COFF/arm64-dynamicbase.s (98417 of 101544)
PASS: lld :: COFF/align.s (98418 of 101544)
PASS: lld :: COFF/arm-thumb-thunks-pdb.s (98419 of 101544)
TIMEOUT: MLIR :: Examples/standalone/test.toy (98420 of 101544)
******************** TEST 'MLIR :: Examples/standalone/test.toy' FAILED ********************
Exit Code: 1
Timeout: Reached timeout of 60 seconds

Command Output (stdout):
--
# RUN: at line 1
"/etc/cmake/bin/cmake" "/build/buildbot/premerge-monolithic-linux/llvm-project/mlir/examples/standalone" -G "Ninja"  -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C_COMPILER=/usr/bin/clang  -DLLVM_ENABLE_LIBCXX=OFF -DMLIR_DIR=/build/buildbot/premerge-monolithic-linux/build/lib/cmake/mlir  -DLLVM_USE_LINKER=lld  -DPython3_EXECUTABLE="/usr/bin/python3.10"
# executed command: /etc/cmake/bin/cmake /build/buildbot/premerge-monolithic-linux/llvm-project/mlir/examples/standalone -G Ninja -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C_COMPILER=/usr/bin/clang -DLLVM_ENABLE_LIBCXX=OFF -DMLIR_DIR=/build/buildbot/premerge-monolithic-linux/build/lib/cmake/mlir -DLLVM_USE_LINKER=lld -DPython3_EXECUTABLE=/usr/bin/python3.10
# .---command stdout------------
# | -- The CXX compiler identification is Clang 16.0.6
# | -- The C compiler identification is Clang 16.0.6
# | -- Detecting CXX compiler ABI info
# | -- Detecting CXX compiler ABI info - done
# | -- Check for working CXX compiler: /usr/bin/clang++ - skipped
# | -- Detecting CXX compile features
# | -- Detecting CXX compile features - done
# | -- Detecting C compiler ABI info
# | -- Detecting C compiler ABI info - done
# | -- Check for working C compiler: /usr/bin/clang - skipped
# | -- Detecting C compile features
# | -- Detecting C compile features - done
# | -- Looking for histedit.h
# | -- Looking for histedit.h - found
# | -- Found LibEdit: /usr/include (found version "2.11") 
# | -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
# | -- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.13") 
# | -- Using MLIRConfig.cmake in: /build/buildbot/premerge-monolithic-linux/build/lib/cmake/mlir
# | -- Using LLVMConfig.cmake in: /build/buildbot/premerge-monolithic-linux/build/lib/cmake/llvm
# | -- Linker detection: unknown
# | -- Performing Test LLVM_LIBSTDCXX_MIN
# | -- Performing Test LLVM_LIBSTDCXX_MIN - Success
# | -- Performing Test LLVM_LIBSTDCXX_SOFT_ERROR
# | -- Performing Test LLVM_LIBSTDCXX_SOFT_ERROR - Success
# | -- Performing Test CXX_SUPPORTS_CUSTOM_LINKER
# | -- Performing Test CXX_SUPPORTS_CUSTOM_LINKER - Success
# | -- Performing Test C_SUPPORTS_FPIC
# | -- Performing Test C_SUPPORTS_FPIC - Success
# | -- Performing Test CXX_SUPPORTS_FPIC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BOLT] Frame pointers break instrumentation
6 participants