aarch64 compatibility

[chaplin89]

Hi,
I've tried to build rpitx on a Raspberry Pi 3 B+ running Kali linux but compilation fails badly while compiling csdr with this error:

gcc -std=gnu99 -O3 -ffast-math -fdump-tree-vect-details -dumpbase dumpvect -mfloat-abi=hard -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mvectorize-with-neon-quad -funsafe-math-optimizations -Wformat=0 -DNEON_OPTS fft_fftw.c libcsdr_wrapper.c  -g -lm -lrt -lfftw3f -DUSE_FFTW -DLIBCSDR_GPL -DUSE_IMA_ADPCM -Wno-unused-result -fpic -shared -Wl,-soname,libcsdr.so.0.15 -o libcsdr.so.0.15
gcc: error: unrecognized command line option ‘-mfloat-abi=hard’
gcc: error: unrecognized command line option ‘-mfpu=neon’
gcc: error: unrecognized command line option ‘-mvectorize-with-neon-quad’

So, the problem is that Kali Linux compiled against armv8/aarch64 architecture, but rpitx is tested only on armv7.

I've tried 2 things to solve the issue:

1. Modify the options in order to build an aarch64 version of csdr. This don't work because csdr contains some "neon" assembly code that is highly coupled with armv7-a.

2. Run

   apt install g++-arm-linux-gnueabihf gcc-arm-linux-gnueabihf
   dpkg --add-architecture armhf
   apt-get install -y libfftw3-dev:armhf
   

   And then manually patched cdsr/rpitx Makefile with:

   :%s/gcc/arm-linux-gnueabihf-gcc/g
   :%s/g++/arm-linux-gnueabihf-g++/g
   

This seems to solve the problem but unfortunately I'm getting "mmap error" in the constructor of generalgpio.

After digging a little bit I've found the culprit to be the output of /proc/cpuinfo.

This is the output on Raspbian:

processor	: 0
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: BCM2835
Revision	: a020d3
Serial		: 0000000039dff3bd

And this is the output on Kali:

root@kali:~# cat /proc/cpuinfo 
processor	: 0
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
BogoMIPS	: 38.40
Features	: fp asimd evtstrm crc32 cpuid
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

As you can see, the lines about hardware and revision at the end is totally missing in kali.
For the moment, I've hardcoded the revision in the code and I managed to get rpitx run correctly, but it would be great to fix this.

Do you have any hints about how to either make the process of retreiving the revision of the raspberry more reliable or fix this in kali?

Thanks

[F5OEO]

Thanks in investigating, including rpitx on Kali could effectively be a good point.
The issues you had seems more relative to csdr (external project from HA7ILM).

For revision detection, think that there is some more easy way I doing now. Have to check it. Hope to get back on project soon.

[chaplin89]

I saw that you're patching the makefile of csdr so maybe those issues could be easily solved in the same way.

From what I've read, the output of /proc/cpuinfo is related to the output of CPUID instruction (and I've 0 clues about why kali is showing a different thing) so maybe it could be solved with some assembly/instinsic instruction but if there exists a simpler solution would be nice!

[MeFisto94]

I have also stumbled across the very same issue and I guess it's not really kali but the fact that it uses aarch64. Actually have a look at F5OEO/librpitx#11

I wonder how you solved all these problems and why rpitx is running correctly for you, I gave up at the mmap error after having fixed everything else so far, but first things first:
https://github.com/MeFisto94/csdr/tree/improve_arm_detection fixes your concerns about detecting the revision (it checks for the device name and if it returns raspberry pi, and if it's 64bit (aarch64) it omits all the abi=hard, because that seems to be implied by the architecture).

The next thing were a few fixes to librpitx and commenting out dvbt etc, but the terminating issue for now is the implementation of "hello-fft" from broadcom, which is what librpitx uses especially in mailbox.c. Why?

1. Apart from it being very bad and unportable code (unsigned instead of fixed size ints, sizeof *p where it's clear that p is unsigned *, so that sizeof(unsigned) would've been sufficient), there is an issue with mmap
2. In Order to talk to the GPU, /dev/mem (the physical memory) is memory mapped using mmap. This seems unable to do on aarch64 due to security reasons (dd is also unable of accesing and I tried many boot/config.txt adjustments). Just google for mmap EPERM /dev/mem there. Maybe I just failed at the correct params but I suspect that the kernel is just compiled to not work with it anymore.
3. Due to 1, even if we fix 2, the whole code would have to be inspected for traps which could be triggered because 64bit wasn't in mind at the time of making. Maybe starting with a 64bit implementation of the GPU Powered FFT.

So I just installed a fresh 32bit kali and rpitx works (I just need to figure out doing FSK in an easy way, but that's my task for the weekend)

Oh: And maybe all I've said is untrue and maybe there is an aarch64 implementation of hello_fft already. Or cross compiling for 32bit (or whatever @chaplin89 did ) might be a viable solution (for that matter maybe even prebuilt files would be a solution, it's actually much faster on a pi anyway)

[chaplin89]

You're right, I've cross-compiled rpitx for 32bit.

At first, I've tried to compile everything for aarch64 but I've seen that without changing the options, csdr try to compile non-portable armv7 assembly code so I've changed to approach without investigating too much. I guess the reason you were able to compile csdr on aarch64 was because you're compiling without -DNEON_OPTS and other non-portable options.

As for the mmap error, I think I was definitely able to mmap the addresses so I'm not really sure about the kernel thing.
I also had issues with mmap, but they were related to the detection of the revision of my raspberry-pi and I've solved it burning the revision number in the code. I'm not saying it is a good way to solve the problem, I was only trying to prove that rpitx can run also on an aarch64 system.

If you see the ctor of generalgpio, it accept an address that is computed by adding a base offset to an offset that is dependent on the specific revision of the raspberry. If the detection fails, the latter offset become zero and you get the error.

As I've said, this is related to the output of /proc/cpuinfo, that it seems to be different between armv8/aarch64 and armv7 (the incriminated code is this: https://github.com/F5OEO/librpitx/blob/5475c41ccf202b544cac6cf0c670ae40210a9f4b/src/raspberry_pi_revision.c#L371). If you fix this, you also fix the mmap error.

[MeFisto94]

I guess the reason you were able to compile csdr on aarch64 was because you're compiling without -DNEON_OPTS and other non-portable options.
Yes, because NEON isn't on aarch64/it's automatically implied (see ha7ilm/csdr#43 for reference, https://stackoverflow.com/a/29891469)

As for the mmap error, I think I was definitely able to mmap the addresses so I'm not really sure about the kernel thing.
My problem was here: https://github.com/F5OEO/librpitx/blob/5475c41ccf202b544cac6cf0c670ae40210a9f4b/src/mailbox.c#L66
It always returned -EPERM (and the libexplain said to that: Process does not have permissions to use the mmap syscall). Btw this code shouldn't return mem but errno, would be more helpful, because mem is always MAP_FAILED there...

I don't know if you were able to work around this because of 32bit cross compilation, I just tried tune -f 492000000 (another fun fact: I was only able to read the output untruncated (after I changed the format string) when launching with gdb).
Does tune work for you? it tried to mmap /dev/mem at 0x2000 ? and the kernel was like "no mmaping of /dev/mem"

Oh, after reading the gpio.cpp code, it could very well be that mmap refuses to work because that region is wrong for the raspberry pi 3 and thus is blocked.

The problem is, rpi_revision looks very detailed to me, my solution was:

cat /proc/device-tree/model 
Raspberry Pi 3 Model B Plus Rev 1.3

device-tree might expose more information, but I don't know if it will be as detailed as what is required/if one needs (unreliable) string parsing.

[chaplin89]

Oh, after reading the gpio.cpp code, it could very well be that mmap refuses to work because that region is wrong for the raspberry pi 3 and thus is blocked.

This.

At the moment I don't see anything to backup the hypothesis that I was able to mmap only because I'm building against armv7, because if I don't hard-code the revision inside getRaspberryPiRevision() I get your same error.

I had the issue in the same code you linked, but this is the consequence of the wrong detection of the revision that is done into getRaspberryPiRevision():

• Failing to detect the revision make this function return RPI_PERIPHERAL_BASE_UNKNOWN==0.
• If that function return 0, this sum will become GENERAL_BASE == 0x200000
• If that is GENERAL_BASE, subsequent mmaps fails.

This does not surprise me too much because that address is in the first few MB of RAM and by the way it's a totally different address from what it should be if this worked well, as GetPeripheralBase() is supposed to return only those addresses when it work well:

#define RPI_BROADCOM_2835_PERIPHERAL_BASE 0x20000000
#define RPI_BROADCOM_2836_PERIPHERAL_BASE 0x3F000000
#define RPI_BROADCOM_2837_PERIPHERAL_BASE 0x3F000000

cat /proc/device-tree/model

I really don't know if this information is enough to infer all the information librpitx needs in order to run correctly and if it has the same format between all the raspberrys, all the distros etc.
As I've suggested, maybe hacking a little bit with the CPUID instruction could be a viable (and hopefully more reliable) solution.

[MeFisto94]

At the moment I don't see anything to backup the hypothesis that I was able to mmap only because I'm building against armv7, because if I don't hard-code the revision inside getRaspberryPiRevision() I get your same error.

Yeah, your findings look like that is the issue. I would've imagined that 64bit is more strict than 32bit in regards of physical memory protection and that for 32bit emulation it maybe runs in compability mode. But I guess that's not the case.

I really don't know if this information is enough to infer all the information librpitx needs in order to run correctly and if it has the same format between all the raspberrys, all the distros etc.
As I've suggested, maybe hacking a little bit with the CPUID instruction could be a viable (and hopefully more reliable) solution.

I think /proc/device-tree/* is better because even if cpuid would work (and I guess cat /proc/cpuinfo is populated by cpuid already?), you can't rely on it, as not necessairly every BCM2837 device is a rpi. But maybe the correct solution is ahybrid of this, the best would actually be to test the various outputs on a few popular distributions (though I guess mainly raspbian and kali will suffice or at least be better than the present state).

[chaplin89]

as not necessairly every BCM2837 device is a rpi.

Well, that's some kind of extreme situation, don't you think?

It's not clear to me why anyone would need to put a software that is called "raspberrypitx" on a device that is not a raspberry pi and by the way I guess that whoever does such a thing is savvy enough to know what he's doing.

Anyway, those addresses are taken from datasheet. As you can read here at page 90, for that particular chipset at bus address 0x7e200000 (that is, physical address GetPeripheralBase() + GENERAL_BASE, as you can read at page 6) there is always GPIO. This "peripheral" may or may not be connected to some real, physical pins depending on the specific device, but that's another story.

The gotcha here is that you're not doing anything wrong accessing that specific region expecting to find DMA for GPIO, or, in other words: you can run that code on every device, on every system, etc. Provided that the chipset is BCM2837, you're pointing to the DMA for GPIO pins every time, so it may even have some sense to try to mantain some kind of agnosticism to the specific device this code is running on.

[F5OEO]

Thanks for your hard work @MeFisto94 . Fixed in current commit