Building parallel find fails

[olafWW]

on a RHEL 9.4 system , cloned just from git ..
when running ./prepare.. it get stuck at

it clones LZ4 into pfind DIR .. (successfully)
[...]
Download LZ4 used for optimizing the network traffic
Cloning into 'lz4'...
remote: Enumerating objects: 17235, done.
remote: Counting objects: 100% (464/464), done.
remote: Compressing objects: 100% (249/249), done.
remote: Total 17235 (delta 292), reused 215 (delta 215), pack-reused 16771 (from 4)
Receiving objects: 100% (17235/17235), 7.07 MiB | 21.48 MiB/s, done.
Resolving deltas: 100% (11810/11810), done.
Updating files: 100% (211/211), done.
[..]
==> built with multithreading support
lz4 build completed
rm: cannot remove '*.a': No such file or directory
Building parallel find; <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< here it start building
Using LZ4 for optimization
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ~~~~~~~~^~~~~
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5,
inlined from ‘pfind_parse_args’ at src/pfind-options.c:122:3:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ^
/usr/bin/ld: ./lz4/lib/liblz4.a(lz4.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
collect2: error: ld returned 1 exit status

[JulianKunkel]

I tried to reproduce it.
Created a fresh installation via docker:
$ docker run -it rockylinux/rockylinux:9.4
$ yum install openmpi
$ export PATH=/usr/lib64/openmpi/bin/:$PATH
$ yum install git
$ git clone https://github.com/VI4IO/pfind.git
$ cd pfind
$ ./prepare.sh # was missing in this listing but ran, I missed to include it in this listing when recollecting my memory and history....
$ ./compile.sh

And it worked. Not sure what is broken.

[olafWW]

Thank you for taking the time ..

so I re-run [root@gpfs1 pfind]# ./compile.sh
rm: cannot remove '.a': No such file or directory
Building parallel find;
Using LZ4 for optimization
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ~~~~~~~~^~~~~
ar: ./lz4/lib/.o: No such file or directory

I would expect.. this is part of the downloaded lz4 install/compile before ??
in deeed, there is no ./lz4/lib/*.o file

root@gpfs-5 pfind]# ./compile.sh
rm: cannot remove '.a': No such file or directory
Building parallel find;
Using LZ4 for optimization
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ~~~~~~~~^~~~~
ar: ./lz4/lib/.o: No such file or directory <<<<<<<<<<<<<<<<<

which is really the case/reality ..

[root@gpfs-5 pfind]# ls -l ./lz4/lib/.o
ls: cannot access './lz4/lib/.o': No such file or directory

checking the compile.sh ...
^^^^^^^^^

if [[ -e ./lz4 ]] ; then
echo "Using LZ4 for optimization"
CFLAGS+=" $CFLAGS -DLZ4 -I./lz4/lib/"
LDFLAGS+=" ./lz4/lib/liblz4.a"
FILES+=./lz4/lib/*.o <<<<<<<<<<<<<<<<<<<<<<<<< this leads to the Error - I think
fi

So I wonder, why you did'nt get this error , cause the file's *.o are really not there ..

?? WORK AROUND - when I remove the DIR "lz4" compile works ...

[olafWW]

just to doc it... as a work around .. pfind w/o lz4 would work

I commented - in the compile.sh (under pfind)

Pfind can use lz4 to optimize the job stealing.
if you use ./prepare.sh it will try to download and compile lz4

then I rerun prepare under io500 . and it works..
[...]
OK: All required software packages are now prepared
ior mdtest md-workbench pfind
[root@gpfs-5 io500]#

So the open question remains.. how this lz4 compile flags gets corrected ..

[JulianKunkel]

I do not fully comprehend what you mean.

Do you mean the pfind version needs to be updated?
#86

How can we now resolve this?

[olafWW]

the difference from your little test (just pfind) to io500 install is... that when I re-run/as you/ the subset (just the pfind) .. it works for me too - but this procedure does not download the lz4 ...

so the compile.sh script is by-passing the section if [[ -e ./lz4 ]] ; then ...

when installing the io500 ... you gonna get this lz4 and with it.. this issue gets "active"

unfolding, the obvious difference comes, when using the prepare.sh from io500, it downloads the same LZ4.. but executes the prepare.sh inside pfind.. and this.. downloads lz4

so can your retry your little test and instead of directly executing compile.sh... execute prepare.sh and lets see, if you get the same issue as I do

[JulianKunkel]

I actually ran with ./prepare.sh but missed to include it in the description above as it was from memory.
I retried to be sure and can confirm it works. Here is the output of the last steps.

[root@132810c9784e /]# git clone https://github.com/VI4IO/pfind.git
Cloning into 'pfind'...
remote: Enumerating objects: 362, done.
remote: Counting objects: 100% (68/68), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 362 (delta 38), reused 44 (delta 22), pack-reused 294 (from 1)
Receiving objects: 100% (362/362), 83.28 KiB | 10.41 MiB/s, done.
Resolving deltas: 100% (225/225), done.
[root@132810c9784e /]# cd pfind/
[root@132810c9784e pfind]# ./prepare.sh 
Download LZ4 used for optimizing the network traffic
Cloning into 'lz4'...
remote: Enumerating objects: 17235, done.
remote: Counting objects: 100% (464/464), done.
remote: Compressing objects: 100% (249/249), done.
remote: Total 17235 (delta 292), reused 215 (delta 215), pack-reused 16771 (from 4)
Receiving objects: 100% (17235/17235), 7.07 MiB | 22.62 MiB/s, done.
Resolving deltas: 100% (11811/11811), done.
creating pkgconfig
CC cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/lz4.o
CC cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/lz4file.o
CC cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/lz4frame.o
CC cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/lz4hc.o
CC cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/xxhash.o
CC cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/lz4.o
CC cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/lz4file.o
CC cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/lz4frame.o
CC cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/lz4hc.o
CC cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/xxhash.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4file.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4frame.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/xxhash.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4hc.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/bench.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lorem.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4cli.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4io.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/timefn.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/threadpool.o
CC cachedObjs/d899e3357a27def6ef4a116bc839409e/util.o
AR cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/liblz4.a
LD cachedObjs/7ccdb4a5df110d76fee2fdf5b251c60f/liblz4.so.1.10.0
LD cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4
liblz4.so.1.10.0 created
==> built with multithreading support
lz4 build completed
[root@132810c9784e pfind]# ./compile.sh
rm: cannot remove '*.o': No such file or directory
rm: cannot remove '*.a': No such file or directory
Building parallel find;
Using LZ4 for optimization
In function ‘option_parse_token’,
    inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
  348 |           arg[-1] = '=';
      |           ~~~~~~~~^~~~~
[OK]

I also checked the last step to include the lz4 properly, which it does:

mpicc -g -O2 -Wextra -Wall -pipe -std=gnu99 -Wno-format-overflow -g -O2 -Wextra -Wall -pipe -std=gnu99 -Wno-format-overflow -DLZ4 -I./lz4/lib/ -o pfind option.o pfind-main.o pfind-options.o pfind.o -lm ./lz4/lib/liblz4.a

[olafWW]

Thank you .... hmmm weird... sending the output (last few lines) from the same steps...

while executing prepare.sh in pfind .. all looks good here too
[...]
liblz4.so.1.10.0 created
AR cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/liblz4.a
LD cachedObjs/d899e3357a27def6ef4a116bc839409e/lz4
==> built with multithreading support
lz4 build completed

now compiling...

[root@gpfs-51 pfind]# ./compile.sh
Building parallel find;
Using LZ4 for optimization
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ~~~~~~~~^~~~~
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5,
inlined from ‘pfind_parse_args’ at src/pfind-options.c:122:3:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ^
/usr/bin/ld: ./lz4/lib/liblz4.a(lz4.o): relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
collect2: error: ld returned 1 exit status

seems, that I have a linker/compile challenge here...

[root@fscc-sr650-51 pfind]# ls -l ./lz4/lib/liblz4.a
lrwxrwxrwx 1 root root 52 May 16 09:09 ./lz4/lib/liblz4.a -> cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/liblz4.a

[JulianKunkel]

The hashes are the same right?
AR cachedObjs/424cf2a9f5217f5f0a2fa0c274d82b6b/liblz4.a
Then the LZ4 version and compiler is identical (as it should be).

Maybe a difference with PFIND compilation?
I suppose you also used OpenMPI?
$ mpicc -v
shows
gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Can you run $ bash -x ./compile.sh
I wonder is the error happening here:
$ mpicc -g -O2 -Wextra -Wall -pipe -std=gnu99 -Wno-format-overflow -g -O2 -Wextra -Wall -pipe -std=gnu99 -Wno-format-overflow -DLZ4 -I./lz4/lib/ -o pfind option.o pfind-main.o pfind-options.o pfind.o -lm ./lz4/lib/liblz4.a
or here:
$ ar rcsT pfind.a option.o pfind-options.o pfind.o ./lz4/lib/liblz4.a

Let's see, when I run $ objdump -drz lz4/lib/liblz4.a
I also find sections like: R_X86_64_32S .rodata
Same as for you.
You also use the same git version, yes?

Here are my $ md5sum *.o lz4/lib/liblz4.a

2f057ac39a3bfcbe22202469a0814913  option.o
4456e8bca87cb48de7c99b7a1b494fe0  pfind-main.o
6671cf1b8359888d6bbb4f97c9cd265e  pfind-options.o
b22fc689851206d21d9e2fb3b2a5da66  pfind.o
78fea326300857af610ff6b3455fcd3f  lz4/lib/liblz4.a

[olafWW]

hmm.. not at the moment ..

[root@gpfs51 pfind]# mpicc -v
mpicc for MPICH version 4.1.1
Using built-in specs.
Reading specs from /usr/lib/rpm/redhat/redhat-hardened-cc1
Reading specs from /usr/lib/rpm/redhat/redhat-annobin-cc1
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-host-pie --enable-host-bind-now --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-plugin --enable-initfini-array --without-isl --enable-multilib --with-linker-hash-style=gnu --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_64=x86-64-v2 --with-arch_32=x86-64 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.1 20231218 (Red Hat 11.4.1-3) (GCC)

But that is a good idea.. let me try that

[olafWW]

That's it... good idea ;-)

==> built with multithreading support
lz4 build completed
[root@gpfs-1 pfind]# ./compile.sh
Building parallel find;
In function ‘option_parse_token’,
inlined from ‘poption_parse’ at src/option.c:386:5:
src/option.c:348:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
348 | arg[-1] = '=';
| ~~~~~~~~^~~~~
[OK]
[root@gpfs-1 pfind]#

I installed openmpi 5.0.7 and with the openmpi mpicc it works...

steps to solve,
installed openmpi -> /usr/local/bin ....
export PATH=/usr/local/bin/:$PATH
clean up (remove lz4 DIR)
pfind/prepare.sh # downloads/builds lz4
compile.sh .

many thx ... ok to close this issue ?