build issue with incomplete definition

[hzhuang1]

I tried to build libbpf-rs. But I got this error. I could reproduce it on both arm64 and x86. The test environment was ubuntu 20.04 and ubuntu 22.04. I tried both official rust version and ubuntu rust version.

How to fix this build issue?

:~/libbpf-rs$ cargo build
Compiling phf v0.11.2
Compiling capable v0.1.0 (/home/hzhuang1/libbpf-rs/examples/capable)
error: failed to run custom build command for capable v0.1.0 (/home/hzhuang1/libbpf-rs/examples/capable)

Caused by:
process didn't exit successfully: /home/hzhuang1/libbpf-rs/target/debug/build/capable-2a443e9fcbfb8a89/build-script-build (exit status: 101)
--- stderr
thread 'main' panicked at examples/capable/build.rs:14:10:
called Result::unwrap() on an Err value: Build(Failed to compile /tmp/.tmpvx8MpJ/capable.o from src/bpf/capable.bpf.c

Caused by:
0: Command clang -I/tmp/.tmpshftmq/bpf/src -fno-stack-protector -D__TARGET_ARCH_arm64 -g -O2 -target bpf -c src/bpf/capable.bpf.c -o /tmp/.tmpvx8MpJ/capable.o failed (exit status: 1)
1: src/bpf/capable.bpf.c:102:5: error: incomplete definition of type 'struct user_pt_regs'
int BPF_KPROBE(kprobe__cap_capable, const struct cred *cred,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:791:41: note: expanded from macro '___bpf_kprobe_args2'
#define ___bpf_kprobe_args2(x, args...) ___bpf_kprobe_args1(args), (void *)PT_REGS_PARM2(ctx)
^~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:790:72: note: expanded from macro '___bpf_kprobe_args1'
#define ___bpf_kprobe_args1(x) ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
^~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:486:44: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)
~~~~~~~~~~~~~~~~~^
src/bpf/capable.bpf.c:102:5: note: forward declaration of 'struct user_pt_regs'
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 5 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:790:72: note: expanded from macro '___bpf_kprobe_args1'
#define ___bpf_kprobe_args1(x) ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:486:27: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:228:42: note: expanded from macro '__PT_REGS_CAST'
#define __PT_REGS_CAST(x) ((const struct user_pt_regs *)(x))
^
src/bpf/capable.bpf.c:102:5: error: incomplete definition of type 'struct user_pt_regs'
int BPF_KPROBE(kprobe__cap_capable, const struct cred *cred,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:792:41: note: expanded from macro '___bpf_kprobe_args3'
#define ___bpf_kprobe_args3(x, args...) ___bpf_kprobe_args2(args), (void *)PT_REGS_PARM3(ctx)
^~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:791:76: note: expanded from macro '___bpf_kprobe_args2'
#define ___bpf_kprobe_args2(x, args...) ___bpf_kprobe_args1(args), (void *)PT_REGS_PARM2(ctx)
^~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:487:44: note: expanded from macro 'PT_REGS_PARM2'
#define PT_REGS_PARM2(x) (__PT_REGS_CAST(x)->__PT_PARM2_REG)
~~~~~~~~~~~~~~~~~^
src/bpf/capable.bpf.c:102:5: note: forward declaration of 'struct user_pt_regs'
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 5 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:790:72: note: expanded from macro '___bpf_kprobe_args1'
#define ___bpf_kprobe_args1(x) ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:486:27: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:228:42: note: expanded from macro '__PT_REGS_CAST'
#define __PT_REGS_CAST(x) ((const struct user_pt_regs *)(x))
^
src/bpf/capable.bpf.c:102:5: error: incomplete definition of type 'struct user_pt_regs'
int BPF_KPROBE(kprobe__cap_capable, const struct cred *cred,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:793:41: note: expanded from macro '___bpf_kprobe_args4'
#define ___bpf_kprobe_args4(x, args...) ___bpf_kprobe_args3(args), (void *)PT_REGS_PARM4(ctx)
^~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:792:76: note: expanded from macro '___bpf_kprobe_args3'
#define ___bpf_kprobe_args3(x, args...) ___bpf_kprobe_args2(args), (void *)PT_REGS_PARM3(ctx)
^~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:488:44: note: expanded from macro 'PT_REGS_PARM3'
#define PT_REGS_PARM3(x) (__PT_REGS_CAST(x)->__PT_PARM3_REG)
~~~~~~~~~~~~~~~~~^
src/bpf/capable.bpf.c:102:5: note: forward declaration of 'struct user_pt_regs'
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 5 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:790:72: note: expanded from macro '___bpf_kprobe_args1'
#define ___bpf_kprobe_args1(x) ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:486:27: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:228:42: note: expanded from macro '__PT_REGS_CAST'
#define __PT_REGS_CAST(x) ((const struct user_pt_regs *)(x))
^
src/bpf/capable.bpf.c:102:5: error: incomplete definition of type 'struct user_pt_regs'
int BPF_KPROBE(kprobe__cap_capable, const struct cred *cred,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
:26:1: note: expanded from here
___bpf_kprobe_args4
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:793:76: note: expanded from macro '___bpf_kprobe_args4'
#define ___bpf_kprobe_args4(x, args...) ___bpf_kprobe_args3(args), (void *)PT_REGS_PARM4(ctx)
^~~~~~~~~~~~~~~~~~
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:489:44: note: expanded from macro 'PT_REGS_PARM4'
#define PT_REGS_PARM4(x) (__PT_REGS_CAST(x)->__PT_PARM4_REG)
~~~~~~~~~~~~~~~~~^
src/bpf/capable.bpf.c:102:5: note: forward declaration of 'struct user_pt_regs'
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:818:20: note: expanded from macro 'BPF_KPROBE'
return ____##name(___bpf_kprobe_args(args));
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:798:41: note: expanded from macro '___bpf_kprobe_args'
#define ___bpf_kprobe_args(args...) ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_helpers.h:189:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
^
note: (skipping 5 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:790:72: note: expanded from macro '___bpf_kprobe_args1'
#define ___bpf_kprobe_args1(x) ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:486:27: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) (__PT_REGS_CAST(x)->__PT_PARM1_REG)
^
/tmp/.tmpshftmq/bpf/src/bpf/bpf_tracing.h:228:42: note: expanded from macro '__PT_REGS_CAST'
#define __PT_REGS_CAST(x) ((const struct user_pt_regs *)(x))
^
4 errors generated.
)
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

[anakryiko]

The problem is that currently all examples include x86-64's vmlinux.h. We need some equivalent to architecture-specific include that libbpf-bootstrap's C example are doing (https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/Makefile#L19)

@danielocfb any idea how it could be done with Rust setup?

[hzhuang1]

With latest code, this issue has been gone on x86.
On arm64, I still had this issue on building capable in example directory. So I used "bpftool" to generate a new vmlinux.h according to README. Then this issue has been resolved.

Is it possible to generate vmlinux.h automatically?

[danielocfb]

I would think that's something that #613 could help with? In general, it seems questionable at best to hard code a single header file in the repository that evidently should be different depending on for what architecture we build. So we should probably revisit this decision.

[hzhuang1]

#613 doesn't help on this issue. So I added my comments in pull request #613.