New Plugin: Crowdsource_pi #1199
Comments
|
Heyas, just commenting here so you guys have a link to my github, and so that I get notifications when you comment here :) This is my first plugin, so the build infrastructure / scripts are quite new to me. The plugin builds for linux, windows and macos right now. However, for some reason it does not load into opencpn on macos (haven't tested windows yet, my dev env is debian linux). If someone could help me test and figure out the build problems that'd be awesome. All my built artefacts are currently here: https://cloudsmith.io/~kahu/repos/crowdsource_pi-beta/packages/?page=1&sort=-name |
|
Also, please see jongough/testplugin_pi#420 regarding macos build issue |
|
I have just tried importing the plugin into macOS ventura 13 and got the following in the opencpn.log I looked for the missing library and found it here: So it looks like there may be either something wrong with your build process or where you have setup to find the wxWidgets libraries. I am not a mac person so really don't know how it works. nohal (Pavel) helps fix this stuff up for me. |
|
I'm not a mac person either, but already had a chat with @nohal who said that during the build, it always references wxwidgets stuff in /usr/local, and then https://github.com/KAHU-radar/testplugin_pi/blob/master/cmake/fix-macos-libs.sh is supposed to fix these paths in the generated file, replacing the absolute paths with relative ones. Which it does. But /after/ the tar.gz package is built. So the issue is a pure cmake issue. And I'm not good enough at cmake to figure out why the order of these build steps is wrong, so that's what I'm asking you for help with :) He does something very similar here: "I myself do it using this script: https://github.com/nohal/dashboardsk_pi/blob/main/cmake/fix-macos-libs.sh (Invoked from https://github.com/nohal/dashboardsk_pi/blob/main/cmake/PluginInstall.cmake#L64)" |
|
I am not macos or really a programer, but take a But first realize that he is using the Shipdriver Template which does things very differently. Then check out https://github.dev/nohal/dashboardsk_pi/blob/main/cmake/PluginInstall.cmake at fixmacos-libs.sh |
|
@redhog Egil I have created crowdsource-alpha, crowsource-beta and crowsource-prod and uploaded a placeholder txt file to each of those repositories. I believe you have an account and are using your own organization repositories right now, here: I would like to send you an invitation from Cloudsmith OpenCPN organization to join the Plugin Team. |
|
I have checked in the tarball and extracted the dylib files for testplugin and crowdsource, then looked in the binary for
So it would appear that the FE2 build process is working OK, but something has changed in the crowdsource implementation of the FE2 that is not working. Try copying all the FE2 build files over the top of your current ones and then build for macOS and see if that helps. |
|
I think it would be productive to compare your CCI logs against the Testplugin CCI logs, specifically for MacOS. |
|
Up to the completion of the build script the systems seem the same. Not sure what the jobs are after the upload to cloudsmith, but they should not effect anything. I can only assume that wrong dylib is being put into the tarball, i.e. the before the fix up script version. |
|
Where exactly is the tar file made (in what scripts, run from where)? |
|
@rgleason https://github.com/KAHU-radar/crowdsource_pi/blob/master/cmake/PluginInstall.cmake#L203 seems very similar to https://github.com/nohal/dashboardsk_pi/blob/main/cmake/PluginInstall.cmake#L59C1-L66C8 , byut the file is organized differently (but both are written by @nohal, althoug the dashboard one has later updates by Alec Leamas). |
|
Elig, which template are you using? I think it is Testplugin. DashboardSK is using the Shipdriver template. Is that why you are looking at the changes that were made to DashboardSK plugininstall.cmake which seems to have code that is providing the correct path. Perhaps this code should be in testplugin files too, however Jon may have made other accommodations to properly build MacOS. Or is this problem triggered by trying to build a certain library? I think we need Jon look into it and to respond here. |
1 similar comment
|
Elig, which template are you using? I think it is Testplugin. DashboardSK is using the Shipdriver template. Is that why you are looking at the changes that were made to DashboardSK plugininstall.cmake which seems to have code that is providing the correct path. Perhaps this code should be in testplugin files too, however Jon may have made other accommodations to properly build MacOS. Or is this problem triggered by trying to build a certain library? I think we need Jon look into it and to respond here. |
|
Elig, now I remember what needs to be done so I can send you a Cloudsmith invite. You need to allow your user name on cloudsmith to be public, then the search will find you. |
|
Yes, my plugin is based on testplugin_pi. |
|
I've made my account on cloudsmith public now. My account is here: https://cloudsmith.io/users/egil-moller/ |
|
The main reason I'm looking at the dashboard one, is that Pavel told me about his solution in a convo on Zulip. He didn't mention his plugin is based on shipdriver though... This script https://github.com/KAHU-radar/crowdsource_pi/blob/master/cmake/fix-macos-libs.sh is identital between shipdriver and testplugin (and my plugin). The only problem is about /when/ this is called - before or after the tgz package is built. |
|
Elig, since it does not work now, why not compare Pavel's to your current plugininstall.cmake, make note of the differences and determine what needs to be changed. Then try that and see if it compiles for MacOS. If it does and if it works, then perhaps the same change should be done to Jon's Testplugin version too. (I wonder why other plugins built with MacOS are working?) Also have you checked that all your testplugin template files are current? I would do that first. Then try the change, you can always revert it. |
I think I managed to send the invite. Let me know if there are any problems. |
|
Invite worked fine |
|
Where is the latest CCI build log? |
|
This is from rerunning it with ssh, I logged in and ran otool -L generating the output seen in jongough/testplugin_pi#420 |
|
Ok, doing it by hand: I logged in to the circleci build machine and reran what the make target "package" does by hand: So cpack surprisingly runs fix-macos-libs.sh (I thought this would only run during the install make target!), but on the wrong files? Not sure how, as CPackConfig.cmake does not reference it. |
|
Further, the library file installed when cpack invokes the install target (from the package target), has the wrong paths! |
|
I assume the culprtit is in the |
|
OK, the interesting stuff is: Sure looks like the paths are being adjusted before the tarball is built. |
|
So removing
|
|
As far as I can see at the moment the '*dylib' file is created correctly and the change in libraries is done correctly. The listing at the end shows the contents of the modified file. I am not sure what is happening then as the cmake packaging process should have taken that whole directory system and zipped it up. The upload code is just putting the xml file inside the tar file at the root level for Plugin manager to process. I think there needs to be more debugging information around what is happening with the dylib and tar files. |
|
I have cloned the repository and tried doing the circleci build, but it appears the 'libs' directory is not building correctly for the universal macos build. It appears to be getting mixed up between arm64 and x86 builds: I may not have configured it all properly, but.... |
|
That's weird, that works fine for me... Note that the dylib in the _CPack_Packages has broken paths. I assume the install target modifies files in another directory, and ignores the cpack stuff, at least for the fixmacos script... But I know nothing of cpack, so hard to figure out why... |
|
The next step is to clone the opencpn/plugins directory locally, then use the scripts in the root of the project to download, and check the xml files generated in the build. There is a README file giving you basic instructions on what to do. See how you go and if you have issues let us know. |
|
Redhog, Awesome. Congratulations. Downloaded it and tried to create an account, but it is not setup yet I guess 401. |
|
Oh, oupsi. The "sign up with github" works, but I guess the email one is broken? This is using django-allauth so I guess I have to poke at it a bit more... Will ping you @rgleason as soon as I've debugged it! |
|
Thanks. |
|
This is a bit embarrasing: By default, email verification is on, but I hadn't set up an smtp account :P But, fixing that, it turned out that the allauth JS/react front end didn't have any code to handle email verification flows, so had to code that up too... The whole auth flow is still a bit rough around the edges - it needs styling, and a bunch of automatic redirects / next-page stuff isn't there, but now at least it works. Give the thing a spin! Test hint: You can test the crowdsource plugin without sailing around with your radar on with the help of the VDR plugin if you have a vdr.txt with $RATTM messages (as well as GPS positions and heading for own boat) in it. |
|
Very impressed with your quick skills setting up for CI builds with testplugin. I don't have one of those SRATTM messages with GPS position and headings, but maybe Hakan does as he is active with Radar_pi. Downloaded and installed. Signed up and the confirm email was recieved, and I acknowledged it and signed in using my email and PW. Then I got this message |
|
I got back in and made an API key and pasted it into the Preferences api slot. |
|
Cool! It doesn't have a UI except the preferences window but runs in the background. Any $RATTM messages get cached in a sqlite database, and whenever you have internet they get uploaded. You can see number of unsent tracks and track points in the preferences dialog. It uses an exponential backoff for reconnection attempts. So no file copying :) The idea is that you might have internet either nearly all the time via say iridium or something, or only when close enough to shore via 3G, or maybe even only over wifi when you're at the dock. The plugin will cope :) Maybe this needs a bit of an explanation / help text in the pref. dialog? |
|
Btw, you might find this script helpful: It generates synthetic (not very realistic) $RATTM messages: https://github.com/KAHU-radar/makerattm/blob/master/makerattm.py You still need heading and position messages to mix them with though. This reminds me: It would be very useful to have a test-rig for opencpn that generates synthetic NMEA fo simulated but realistic voyages and/or a library of recorded real voyages. And, for the radar plugin specifically: A simulator radar that generates realistic images (based on a map and vessel positions). |
|
Btw: What is the practical difference between alpha, beta and prod? Which of them actually show up in the plugin list in opencpn (or do they all, but prod has priority)? |
|
We use them alpha for trial and testing, beta is used for co mmit tagged pushes that are not from master. Prod is used for commits that are tagged and pushed. These CS all have different retention policies. Prod is longer and larger. We like to see metadata from versions in Prod being pushed to the master catalog. I am currently moving the old metedata versions from the master publish plugins catalog to the beta plugins catalog. Do not know If i will continue that. |
|
But all of them go out to the GUI? Or can you somewhere in OpenCPN say if you want to install e.g. beta plugins and not just prod? |
|
Maybe this will help Push to Cloudsmith [plugin]-beta repository via Github & CircleCI Push to Cloudsmith [plugin]-prod repository via Github & CircleCI |
|
Added docs (Hope I did all the linking/naming right), and made a PR against the plugins repo |
|
Excellent job. |
|
So, I saw my PR was merged. But no plugin shows up in the plugin manager in OpenCPN. I also don't see my docs show up in https://opencpn-manuals.github.io/main/opencpn-plugins/index.html. So, what did I do wrong? |
|
You did nothing wrong Elig, we need to get the necessary hooks established so that your manual is brought into the Plugin Manual. Your plugin manual has to be added to a plugin list that is executed. Mike Rossiter can do that. Will you please contact him?, perhaps through github [Rasbats] If you have trouble, let me know. |
13:38:00.470 MESSAGE plugin_loader.cpp:640 PluginLoader searching for PlugIns in location C:\Users\fcgle\AppData\Local\opencpn\plugins |
|
I did test the windows build using "Import plugin..." and that worked... But when a plugin isn't yet installed, it surely can't find the dll locally yet? That needs download first, right? When you just update the plugin catalog, you just get a bunch of xml-files, right? |
|
I pinged Mike Rossiter's github on https://opencpn.zulipchat.com . What's their github username btw? |
|
The import of the tar file uses the xml embedded in the tar file for compatibility testing. The plugin code is placed, I think, in the plugin cache, but it is not loaded. You then just need to select the plugin in plugin manager and try activating it. This will then try to make the plugin work and will highlight any issues. The plugin manager catalog, when you download it, just downloads the catalog xml files that are in the prod/beta/alpha repository, but does not download the tar file until you install it. I think the latter process automatically activates the plugin as well, so it appears as one process. |
|
@jongough Yep, so then the question is still, why isn't it showing up in the catalog / the list when updating the catalog? |
|
Aha! It is. In the xml. But: I'm on debian bookworm, and that debian version isn't in the circleci list, so no plugin is compiled for my platform! I'll fix and make a new version. |
|
If you are using FE2 then all the supported platforms/versions will be generated. If you have generated into cloudsmith and download the tar file from there, then you need to download into a local clone of plugins the XML using the supplied download scripts. It is advised to check the XML using the validate script before adding/staging, committing, pushing and creating a pull request on opencpn/plugins. It will then get merged, normally within a hour or so and will become available through the 'catalog update' process. |
|
Elig,
That's what I told you earlier. It is in the catalog, but it doesn't appear on my Windows machine either and there is absolutely no message that it is incompatible or fails, AND I can see that the plugin manager is loading the po files. I have never seen this happen. BTW Jon, Elig's crowdsource is using TP Frontend2 |
|
I have just started a windows 10 machine, updated OCPN to 5.10.2, updated the plugin catalog for master, looked for and found As for bookworm the current FE2 builds for" So it should be there, but I don't have a test machine for that, so cannot verify. |
|
@jongough If you get that far, the only step left is to either load some $RATTM messages using the VDR plugin (but then please tell me so I can mark the tracks you upload as tests, should add a GUI for that), or by adding the radar plugin and actually go for a sailing trip (no idea where you are and if the season is sailable, here in norway all the boats are up on land right now) :) |
|
### Crowdsource is not listed by Plugin Manager Have you looked carefully at the metadata output? I downloaded one of the metadata file for this tarball and looked at it. When I put that url into my browser, the file is found and downloaded to my downloads directory. So why are the plugins dlls not being found by Plugin Manager? I see what happened now. Crowdsource ----> Kahu Radar HubThis must be changed. It is very confusing to the user. What is this plugin Crowdsource or Kahu Radar Hub? |
|
I only installed the plugin to demonstrate that it was there. I don't have radar where I am (not on the boat). But got most of the plugin working. However, as Rick says above, the plugin is not called I'm in Australia, so it is supposed to be good weather, but we seem to be having quite a few storms and LOTS of rain. So, not as good as it should be! |
Contribute AIS and ARPA targets from your vessel to crowdsourcing for marine safety!
This plugin lets you upload AIS and radar ARPA targets (or any NMEA) to an internet server. Upload can be continuous, over intermittent internet, or scheduled, and tracks can be downsampled to fit your bandwidth.
Using Apache Avro. Designed to be used with radar_pi.
by "redhog"
https://github.com/KAHU-radar/crowdsource_pi
I am going to set Egil "redhog" as an Cloudsmith OpenCPN.org Plugin Teammember and make the 3 repositories that he can deploy too.
However Egil may need some help getting the Android versions to work.
The text was updated successfully, but these errors were encountered: