Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CKAN Harvest troubleshooting: ERDDAP WAF harvests failures/outdated datasets #238

Closed
mwengren opened this issue Mar 7, 2023 · 11 comments
Closed

CKAN Harvest troubleshooting: ERDDAP WAF harvests failures/outdated datasets #238

mwengren opened this issue Mar 7, 2023 · 11 comments
Labels
high priority
Milestone
Release 1.7: CKAN...

Comments

@mwengren
Copy link
Member

mwengren commented Mar 7, 2023 •
edited
Loading

Note:

This issue is more about restoring overall harvesting functionality than troubleshooting individual harvest sources. We'll work off of a separate, newly created issue: ioos/catalog#87 to restore all of the RA/data provider harvest sources once CKAN harvesting is functioning more consistently/reliably.

Example harvests:

Example list of some harvest sources that have had issues in the past for reference

GCOOS Oceanographic: https://data.ioos.us/harvest/gcoos-erddap-oceanographic/job/last - no datasets

GCOOS Biological: https://data.ioos.us/harvest/gcoos-erddap-biological/job/last - no datasets

GCOOS Historical: https://data.ioos.us/harvest/gcoos-waf-historical/job/last - no datasets

CARICOOS: https://data.ioos.us/harvest/caricoos-erddap-server-iso-waf/job/last - no datasets
@mwengren
Copy link
Member Author

mwengren commented Jun 5, 2023 •
edited
Loading

The GCOOS Oceanographic WAF (https://data.ioos.us/harvest/gcoos-erddap-oceanographic/job/last) was deleted from CKAN at some point, need to restore it.

URLs:

https://erddap.gcoos.org/erddap/
https://erddap.gcoos.org/erddap/metadata/iso19115/xml/

@mwengren
Copy link
Member Author

Looks like ERDDAP WAF harvests are working better as of June/July, closing this issue.

@mwengren
Copy link
Member Author

mwengren commented Jan 8, 2024 •
edited
Loading

Reopening and renaming this issue as there is still an issue where some harvest jobs get stuck in a frozen state and the cleanup script does not properly identify them and restart the jobs. This resulted in significantly outdated datasets or WAFs where no datasets were present due to a failed harvest.

During today's meeting @benjwadams updated the script to use the created field date in the CKAN database rather than the gather_started date, and this appeared to help. Several harvests that had been frozen since ~ early November were cleared out and restarted (AOOS ERDDAP, SECOORA ERDDAP).

Let's keep this issue open until the CKAN harvesting is functioning properly and consistently for most harvests (CS-W excepted).

@mwengren mwengren reopened this Jan 8, 2024
@mwengren mwengren changed the title Harvest troubleshooting: Empty ERDDAP WAFs CKAN Harvest troubleshooting: ERDDAP WAF harvests failures/outdated datasets Jan 8, 2024
@mwengren mwengren mentioned this issue Feb 5, 2024
Closed
@benjwadams
Copy link
Contributor

GCOOS WAF Historical WAF - https://data.ioos.us/harvest/gcoos-waf-historical/job Looks OK
CARICOOS - Possibly moved to http://dm3.caricoos.org:8002/erddap/index.html
GCOOS Oceanographic WAF - https://data.ioos.us/harvest/gcoos-oceanographic-waf/job -- OK, different link
GCOOS Biological WAF - OK

These all look OK, with CARICOOS having possibly changed ERDDAP location.
@mwengren, please provide feedback.

@mwengren mwengren mentioned this issue Mar 4, 2024
Closed
@mwengren
Copy link
Member Author

mwengren commented Mar 4, 2024

Spot check today of harvest sources shows many jobs not finished properly (no finish date in job metadata) since ~ Feb 20 - cleared by cleanup script instead?

Harvester configuration (Docker containers):

  • Fetch harvester (Docker container) had issues/crashed ~ 10 days ago: set to automatically restart but didn’t happen properly. @benjwadams to look into adding alerting scripts to monitor Docker container availability/uptime.
  • Gather harvester: reads WAF URLs from CKAN harvest sources and sends job to Fetch harvester. Also a separate Docker container but not the problem source here.

@benjwadams
Copy link
Contributor 


ckan=> with t as (select max(gather_started) start_gather, max(finished) finish_time, source_id from harvest_job group by source_id) SELECT t.start_gather, t.finish_time, harvest_source.title, frequency FROM
harvest_source JOIN t ON t.source_id = harvest_source.id ORDER BY t.finish_time;
        start_gather        |        finish_time         |             title              | frequency
----------------------------+----------------------------+--------------------------------+-----------
 2024-03-04 20:02:03.340298 | 2024-03-04 20:02:23.262072 | GCOOS ERDDAP Biological WAF    | MANUAL
 2024-03-18 19:45:35.552933 | 2024-03-18 19:45:38.088894 | Unidata WAF                    | DAILY
 2024-03-18 19:45:38.177757 | 2024-03-18 19:45:39.099912 | US Navy WAF                    | DAILY
 2024-03-18 19:45:41.042588 | 2024-03-18 19:45:42.04804  | SCCOOS WAF                     | DAILY
 2024-03-18 19:45:45.955383 | 2024-03-18 19:45:46.86199  | NANOOS WAF                     | DAILY
 2024-03-18 19:45:46.875716 | 2024-03-18 19:45:47.045233 | MARACOOS WAF                   | DAILY
 2024-03-18 19:46:27.009209 | 2024-03-18 19:46:27.17146  | COMT WAF                       | DAILY
 2024-03-18 19:46:27.186076 | 2024-03-18 19:46:27.891141 | CDIP WAF                       | DAILY
 2024-03-18 19:45:39.15572  | 2024-03-18 19:46:50.60057  | SECOORA WAF                    | DAILY
 2024-03-18 19:45:42.063436 | 2024-03-18 19:49:07.514375 | PacIOOS WAF                    | DAILY
 2024-03-18 20:00:31.763946 | 2024-03-18 20:00:32.020717 | AOOS WAF                       | DAILY
 2024-03-18 20:00:32.040762 | 2024-03-18 20:00:32.317932 | HF Radar NDBC DAC WAF          | DAILY
 2024-03-18 20:00:32.34567  | 2024-03-18 20:00:32.839261 | HF Radar SCRIPPS DAC WAF       | DAILY
 2024-03-18 20:24:39.450978 | 2024-03-18 20:24:48.056865 | CARICOOS ERDDAP SERVER ISO WAF | DAILY
 2024-03-18 20:26:53.86549  | 2024-03-18 20:26:58.293914 | MARACOOS Stevens WAF           | DAILY
 2024-03-18 20:27:44.012121 | 2024-03-18 20:27:44.742146 | NERACOOS UMAINE REALTIME WAF   | DAILY
 2024-03-18 20:27:44.767549 | 2024-03-18 20:27:48.752397 | NDBC WAF                       | DAILY
 2024-03-18 20:27:48.774543 | 2024-03-18 20:27:51.546565 | CO-OPS WAF                     | DAILY
 2024-03-18 20:27:56.544681 | 2024-03-18 20:27:58.922549 | NANOOS 52NSOS WAF              | DAILY
 2024-03-18 20:27:59.737989 | 2024-03-18 20:27:59.936469 | NERACOOS BIO WAF               | DAILY
 2024-03-18 20:27:59.951109 | 2024-03-18 20:27:59.99071  | IOOS RA Portal WAF             | DAILY
 2024-03-18 20:28:00.174308 | 2024-03-18 20:28:03.795819 | NERACOOS UMAINE WAF            | DAILY
 2024-03-18 19:45:47.057392 | 2024-03-18 20:29:21.302243 | Glider DAC WAF                 | DAILY
 2024-03-18 19:46:20.87883  | 2024-03-18 20:32:01.389469 | GCOOS WAF                      | DAILY
 2024-03-18 19:46:26.561527 | 2024-03-18 20:32:24.244869 | CeNCOOS WAF                    | DAILY
 2024-03-18 20:00:30.84191  | 2024-03-18 20:33:02.297173 | CARICOOS WAF                   | DAILY
 2024-03-18 20:15:38.831879 | 2024-03-19 06:14:49.159909 | GCOOS WAF Historical WAF       | DAILY
 2024-03-18 20:24:38.348843 | 2024-03-19 06:16:46.888727 | SCCOOS ERDDAP WAF              | DAILY
 2024-03-18 20:24:48.06904  | 2024-03-19 06:16:52.360527 | GLOS TDS WAF                   | DAILY
 2024-03-18 20:24:49.969294 | 2024-03-19 06:24:19.505666 | MARACOOS ERDDAP WAF            | DAILY
 2024-03-18 20:24:59.630462 | 2024-03-19 06:24:48.310598 | NANOOS ERDDAP WAF              | DAILY
 2024-03-18 20:25:00.423622 | 2024-03-19 08:05:25.845988 | AOOS ERDDAP WAF                | DAILY
 2024-03-18 20:26:01.288978 | 2024-03-19 09:23:52.162843 | SECOORA ERDDAP WAF             | DAILY
 2024-03-18 20:26:58.310068 | 2024-03-19 10:00:50.104343 | CeNCOOS ERDDAP WAF             | DAILY
 2024-03-18 20:27:51.573423 | 2024-03-19 10:11:37.085535 | NERACOOS ERDDAP WAF            | DAILY
 2024-03-18 20:27:56.03104  | 2024-03-19 10:11:41.529252 | NERACOOS NECOFS WAF            | DAILY
 2024-03-18 20:27:56.218396 | 2024-03-19 10:12:05.338978 | CO-OPS ERDDAP WAF              | DAILY
 2024-03-18 20:27:58.950147 | 2024-03-19 10:13:10.703116 | NANOOS TDS WAF                 | DAILY
 2024-03-19 06:00:28.709693 | 2024-03-19 10:20:28.178464 | GCOOS Oceanographic WAF        | WEEKLY
 2024-03-19 16:16:41.489843 | 2024-03-19 16:17:04.056604 | USGS CS-W WAF                  | DAILY
                            |                            | GLOS CS-W WAF                  | DAILY

Further investigation shows that after the fixes for automatic job starts for stuck harvests were removed, non-daily jobs appear to be running on the appropriate schedule. Here is "GCOOS Oceanographic WAF", which is WEEKLY schedule as reported in the above query:

ckan=> select gather_started, finished from harvest_job JOIN harvest_source ON harvest_job.source_id = harvest_source.id WHERE harvest_source.title = 'GCOOS Oceanographic WAF' ORDER BY gather_started DESC LIMIT 10;
       gather_started       |          finished          
----------------------------+----------------------------
 2024-03-19 06:00:28.709693 | 2024-03-19 10:20:28.178464
 2024-03-12 06:00:23.097766 | 2024-03-12 12:26:20.600568
 2024-03-04 18:01:08.246057 | 
 2024-03-04 16:19:27.016422 | 2024-03-04 17:47:26.659073
 2024-02-28 01:01:13.77649  | 
 2024-02-25 00:01:13.575157 | 
 2024-02-21 23:01:05.557467 | 
 2024-02-20 19:01:16.111546 | 2024-02-21 19:12:56.837212
 2024-02-20 01:01:09.780335 | 2024-02-20 18:16:19.516556
 2024-02-19 11:01:17.130323 | 2024-02-20 00:41:55.482358
(10 rows)

@benjwadams
Copy link
Contributor

Related to #247

@mwengren mwengren mentioned this issue Apr 8, 2024
Closed
@mwengren mwengren mentioned this issue Jun 3, 2024
Open
@mwengren
Copy link
Member Author

mwengren commented Jun 3, 2024

During today's Catalog meeting, we changed the GCOOS ERDDAP Biological WAF harvest to manual to test whether CKAN honors these settings properly or not. Follow up to confirm in the next few days.

@mwengren
Copy link
Member Author

mwengren commented Jul 8, 2024

GCOOS ERDDAP Biological WAF has not run a harvest since June 4, so it looks like the harvest settings are being honored.

ERDDAP harvests in general appear to be working more reliably, so closing this issue again as resolved.

@mwengren mwengren closed this as completed Jul 8, 2024
@mwengren
Copy link
Member Author

mwengren commented Nov 4, 2024 •
edited
Loading

This may be a one-off, but the AOOS ERDDAP WAF harvest job appears to be 'stuck'. It hasn't run since July 30:

https://data.ioos.us/harvest/aoos-erddap/job

Also:

Reopening this issue to address this and in case there are other problems with harvest jobs not being run on an ongoing (weekly) basis.

@mwengren mwengren reopened this Nov 4, 2024
@github-project-automation github-project-automation bot moved this from Done to In progress in IOOS Catalog Nov 4, 2024
@mwengren
Copy link
Member Author

With changes made to harvest sources to weekly frequency per ioos/catalog#91, the harvests appear to be running more reliably now.

Closing this issue again.

@github-project-automation github-project-automation bot moved this from In progress to Done in IOOS Catalog Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority
Projects
IOOS Catalog
Status: Done
Milestone
Release 1.7: CKAN 2.9 Migration/Catal...
Development

No branches or pull requests
2 participants
@mwengren @benjwadams