Memory access issues (alignment exception, corrupted memory, uninitialized memory access)

[bmcbarron]

Howdy,
I'm enjoying this project, and working on adding a firebase plugin (current progress).

It's working well, except the flutter process regularly dies with a memory issue. Most often, it's "Unhandled fault: alignment exception", but I've also seen "munmap_chunk(): invalid pointer", and some others more rarely. It seems possible that their all related.

I tried some manual investigation, and couldn't track down the issue. So, I downloaded and built the latest valgrind, and ran it on flutter (valgrind --tool=memcheck --track-origins=yes flutter-pi). It found a lot of "Conditional jump or move depends on uninitialized value(s)" in a variety of libraries: libflutter_engine.so.debug, libxkbcommon.so, libudev, etc. I don't know if these are spurious or not.

Inevitably, the process dies with "Your program just tried to execute an instruction that Valgrind did not recognize". It seems like memory corruption could cause this too. Regardless, there wasn't enough information for me to diagnose the source of the issue. I don't have debugging symbols for the libraries above, so it's a needle in a haystack.

Or, maybe the memory issue is a red herring (I don't think so, though). At this point, I'm starting down the path of building the Flutter engine myself, so I can re-run valgrind with better debugging symbols. In the meantime, if you have any ideas, I'd appreciate it.

Cheers, Brian

[ardera]

I'm enjoying this project, and working on adding a firebase plugin (current progress).

Nice! What do you think of upstreaming it (when it's ready)?
I have a big refactor planned for platform messages and the general structure of flutter-pi. Upstreaming it would allow me to apply these refactors to your plugin as well.

Inevitably, the process dies with "Your program just tried to execute an instruction that Valgrind did not recognize".

I get this too when debugging with valgrind, it seems valgrind doesn't support arm that well. The instruction being encountered is most likely a real ARM instruction, it's just that valgrind doesn't know it. If it wouldn't be a real instruction, the program would terminate with an Illegal Instruction error when running without valgrind. Raspbian provides some overrides for common C stdlib functions like memcpy, memmove etc with some speed optimizations, but these also seem to contain instructions not recognized by valgrind. There's a workaround though.

I haven't tested yet whether this workaround will make valgrind work with flutter-pi. I'm working on the android auto plugin now & I use openssl there and valgrind fails inside openssl too with some unrecognized instruction error.

Maybe a better method is using address sanitizer or ubsan.

It found a lot of "Conditional jump or move depends on uninitialized value(s)" in a variety of libraries: libflutter_engine.so.debug, libxkbcommon.so, libudev, etc. I don't know if these are spurious or not.

I get that too. I think it's probably fine

It's working well, except the flutter process regularly dies with a memory issue. Most often, it's "Unhandled fault: alignment exception", but I've also seen "munmap_chunk(): invalid pointer", and some others more rarely. It seems possible that their all related.

May I ask what board & kernel you're using?

I think when I tested using ubsan, it too printed out some alignment errors. Most probably, that comes from the platform message encoding / decoding using the standard message codec (it does a lot of raw pointer arithmetic). However, it all worked just fine on my machine so I assumed it would run fine everywhere.

The munmap_chunk() is probably some free failing. There should be more info about that when running with address sanitizer. However, I've tested a lot using asan and never got such an error.

[bmcbarron]

Board: Pi 4 Model B - 2GB RAM
Kernel: 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

I do plan to contribute the firebase plugin once it's ready. Right now, I'm focusing on getting this thing working for an event on Friday. 😅

I'm running this on the debug engine in JIT mode, because it's faster for me to accept new Flutter builds from a friend who don't have a setup to build AOT on their machine.

I can believe those valgrind errors are spurious. I'll try asan and continue with my build of the engine. I'll also try running a different Flutter app to see if the problems are specific to my plugin. I saw that in your AOT discussion, you were getting alignment exceptions too. Did you ever trace anything down on that?

[bmcbarron]

Ok, I tracked down the memory alignment issue by using clang. It didn't give me all the information, but I was able to narrow to inside FlutterEngineSendPlatformMessage, whenever a FlutterPlatformMessageCreateResponseHandle is passed in. I think the response handle is created correctly, so the bug may be in the engine, not sure. I'm still building the engine in debug mode (you weren't kidding that's a pain). Once I do, I should be able to root cause it. For now, I'm just not using response_handle with my sent messages.

[ardera]

Oh yeah, there's one multithreading bug with platch_send. If you invoke it on a thread that is not the flutter-pi main / platform task thread, registering a response callback won't work. The flutter engine function that creates the response handle (FlutterPlatformMessageCreateResponseHandle) needs to be called on the main / platform task thread (the thread where any platform message handlers are called). However, platch_send will try to create a response handle on the thread it's being called on. So if it's not called on the main thread, that won't work. And if it later tries to release the response handle, that fails too (and probably prints the munmap_chunk error) because the response handle is not valid.

This is one of the things I'm fixing in the big refactor I was talking about.

On the other hand, it seems like that SnapshotListener could be perfectly implemented using event channels. On event channels, you can send 2 types of events to flutter: value events and error events. These events will be made available inside flutter as a stream.

Use platch_send_success_event_std to send a value into an event channel, and use platch_send_error_event_std to send an error into an event channel. (there's also platch_send_success_event_json and platch_send_error_event_json, but I suggest using the std variant since that one has less overhead.)

One more thing: Flutter will call listen on the channel to indicate that you should start sending events and cancel to indicate you should stop. You need to listen to these method calls (so register a receiver using platch_set_receiver) and just send a null response back. If flutter receives a not implemented response, an error will be thrown inside flutter.
I think (99% sure) you also don't need to respect these listen and cancel calls at all, so you can send events before flutter called listen and after flutter called cancel if you want too.

The omxplayer_video_player plugin uses an event channel, so you can see a reference implementation there. The listen and cancel method call listener is on_receive_evch and there's also an invocation of platch_send_success_event_std here.

[bmcbarron]

Hi Hannes, The team that I was developing the Firebase plugins for has since disbanded. It works, although not all of the firebase methods are implemented. I think you've done some refactorings, so it may take a bit of work to merge upstream. I was going to do that soon, if you thought it was a good idea. Let me know. Cheers, Brian

…

On Tue, Nov 17, 2020 at 5:10 PM Hannes Winkler ***@***.***> wrote: Oh yeah, there's one multithreading bug with platch_send. If you invoke it on a thread that is not the flutter-pi main / platform task thread, registering a response callback won't work. The flutter engine function that creates the response handle ( FlutterPlatformMessageCreateResponseHandle) needs to be called on the main / platform task thread (the thread where any platform message handlers are called). However, platch_send will try to create a response handle on the thread it's being called on. So if it's not called on the main thread, that won't work. And if it later tries to release the response handle, that fails too (and probably prints the munmap_chunk error) because the response handle is not valid. This is one of the things I'm fixing in the big refactor I was talking about. On the other hand, it seems like that SnapshotListener could be perfectly implemented using event channels <https://api.flutter.dev/flutter/services/EventChannel-class.html>. On event channels, you can send 2 types of events to flutter: value events and error events. These events will be made available inside flutter as a stream. Use platch_send_success_event_std <https://github.com/bmcbarron/flutter-pi/blob/7bd863422b25448e8e327ee352780a5d61a981e8/src/platformchannel.c#L1307> to send a value into an event channel, and use platch_send_error_event_std <https://github.com/bmcbarron/flutter-pi/blob/7bd863422b25448e8e327ee352780a5d61a981e8/src/platformchannel.c#L1319> to send an error into an event channel. (there's also platch_send_success_event_json and platch_send_error_event_json, but I suggest using the std variant since that one has less overhead.) One more thing: Flutter will call listen on the channel to indicate that you should start sending events and cancel to indicate you should stop. You need to listen to these method calls (so register a receiver using platch_set_receiver) and just send a null response back. If flutter receives a not implemented response, an error will be thrown inside flutter. I think (99% sure) you also don't need to respect these listen and cancel calls at all, so you can send events before flutter called listen and after flutter called cancel if you want too. The omxplayer_video_player plugin uses an event channel, so you can see a reference implementation there. The listen and cancel method call listener is on_receive_evch <https://github.com/bmcbarron/flutter-pi/blob/7bd863422b25448e8e327ee352780a5d61a981e8/src/plugins/omxplayer_video_player.c#L942> and there's also an invocation of platch_send_success_event_std here <https://github.com/bmcbarron/flutter-pi/blob/7bd863422b25448e8e327ee352780a5d61a981e8/src/plugins/omxplayer_video_player.c#L678> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#122 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMQC4FZU27NROCMQS242JDSQLYERANCNFSM4TYEGSVQ> .

[ardera]

@bmcbarron Forgot about this, my bad. Do you still have the code lying around?

[bmcbarron]

Yes, although it's dated now. The fork https://github.com/bmcbarron/flutter-pi has all of the code I completed. The methods that are implemented do work (our team used it for a short while). However, I have not resynced with master since you changed how the build works.

[kaiavi]

Hey, @ardera & @bmcbarron What are the possibilities of using Firebase Cloud Messaging in an app running using flutter-pi? Is that feasible? I badly wanted a solution that would let me push data from the cloud to the Pi device? I found that FCM is a solution for the same in Android and iOS. Do you think it would work on Pi? If yes, how do you think I should proceed?