Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawler exit stuck #769

Open
benoit74 opened this issue Feb 12, 2025 · 5 comments
Open

Crawler exit stuck #769

benoit74 opened this issue Feb 12, 2025 · 5 comments

Comments

@benoit74
Copy link
Contributor

Crawler version : 1.5.1 (will update "soon")

We have a situation where the crawler gets interrupted due to the time limit but never exits. I sent a SIGTERM to the crawler and this is the result (we have multiple occurences of the crawler getting stuck, and for two of them I sent the SIGTERM and got same result as shown below. What is weird is that we've probably been blacklisted because all pages before the time limit seems to be ending with Direct fetch of page URL timed out error.

{"timestamp":"2025-02-11T12:58:32.272Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907"}}
{"timestamp":"2025-02-11T12:58:32.274Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T12:58:32.271Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:BookSources\\/0953926907\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:00:02.305Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","workerid":0}}
{"timestamp":"2025-02-11T13:00:02.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland"}}
{"timestamp":"2025-02-11T13:00:02.326Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:00:02.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Talk:Ireland\",\"added\":\"2025-02-11T11:04:42.907Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:01:32.357Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:01:32.380Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland"}}
{"timestamp":"2025-02-11T13:01:32.382Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":1157,"total":6498,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-11T13:01:32.379Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.cruiserswiki.org\\/wiki\\/Special:WhatLinksHere\\/Ireland\",\"added\":\"2025-02-11T11:04:42.934Z\",\"depth\":2}"]}}
{"timestamp":"2025-02-11T13:03:02.413Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","workerid":0}}
{"timestamp":"2025-02-11T13:03:02.418Z","logLevel":"info","context":"general","message":"Time threshold reached 7279.758 > 7200, stopping","details":{}}
{"timestamp":"2025-02-11T13:03:02.430Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-11T13:03:02.431Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:33:39.133Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:33:39.134Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Limerick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.141Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Fahan","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Galway","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Sligo","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Westport,_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Ballycastle,_N_Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Bangor","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Belfast_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Carrickfergus","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.142Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Donegal","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Foyle","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Larne","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Portaferry","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Phennick","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Strangford_Lough","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Template:Port","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/File:Transport_map.jpg","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:BookSources/0953926907","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Talk:Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.143Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.cruiserswiki.org/wiki/Special:WhatLinksHere/Ireland","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.146Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.cruiserswiki.org/wiki/Fahan","workerid":0}}
{"timestamp":"2025-02-12T20:33:39.150Z","logLevel":"warn","context":"general","message":"Link Extraction failed in frame","details":{"page":"https://www.cruiserswiki.org/wiki/Kilrush","workerid":0,"type":"exception","message":"Protocol error (Runtime.callFunctionOn): Target closed","stack":"TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at Connection.dispose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:177:22)\n    at CdpBrowser.disconnect (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:226:26)\n    at CdpBrowser.close (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Browser.js:222:20)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Browser.close (file:///app/dist/util/browser.js:241:13)\n    at async Crawler.serializeAndExit (file:///app/dist/crawler.js:923:13)\n    at async handleTerminate (file:///app/dist/main.js:30:13)"}}
{"timestamp":"2025-02-12T20:33:39.260Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

{"timestamp":"2025-02-12T07:35:08.324Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm"}}
{"timestamp":"2025-02-12T07:35:08.325Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:35:08.323Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Strasbourg.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:36:38.353Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","workerid":0}}
{"timestamp":"2025-02-12T07:36:38.384Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm"}}
{"timestamp":"2025-02-12T07:36:38.387Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:36:38.383Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Tallinn.htm\",\"added\":\"2025-02-12T05:41:02.015Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:38:08.417Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","workerid":0}}
{"timestamp":"2025-02-12T07:38:08.444Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm"}}
{"timestamp":"2025-02-12T07:38:08.446Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:38:08.443Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/other-cities\\/trains-from-Toulouse.htm\",\"added\":\"2025-02-12T05:41:02.017Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:39:38.452Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","workerid":0}}
{"timestamp":"2025-02-12T07:39:38.480Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm"}}
{"timestamp":"2025-02-12T07:39:38.481Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":151,"total":1188,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":1,\"started\":\"2025-02-12T07:39:38.479Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.seat61.com\\/international-trains\\/trains-from-Venice.htm\",\"added\":\"2025-02-12T05:41:02.018Z\",\"depth\":1}"]}}
{"timestamp":"2025-02-12T07:41:08.509Z","logLevel":"error","context":"fetch","message":"Direct fetch of page URL timed out","details":{"seconds":90,"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","workerid":0}}
{"timestamp":"2025-02-12T07:41:08.517Z","logLevel":"info","context":"general","message":"Time threshold reached 7213.174 > 7200, stopping","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T07:41:08.536Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T20:30:50.242Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:30:50.243Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:30:50.263Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-tickets-by-phone.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-train-booking-horizons.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/european-travel-from-uk-regions.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-reservations.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/interrail-and-eurail-passes-on-eurostar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/ferry-to-spain.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Algarve.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Madrid.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.264Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Amsterdam.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Athens.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Barcelona.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Switzerland.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Belgrade.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Berlin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bordeaux.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Bruges.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bratislava.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.265Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Brussels.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Bucharest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Budapest.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Nice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cesky-Krumlov.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Cinque-Terre.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Cologne.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Copenhagen.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dublin.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Dubrovnik.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.266Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Florence.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Frankfurt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Gibraltar.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Hallstatt.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Hamburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Helsinki.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Innsbruck.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.267Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Istanbul.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kyiv.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Krakow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Lisbon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Ljubljana.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.268Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Luxembourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Kiev.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Lyon.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Marseille.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Milan.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.269Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Montpellier.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Moscow.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Munich.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Naples.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Oslo.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.270Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Paris.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Passau.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Pisa.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Porto.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Prague.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Riga.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Rome.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-St-Petersburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Salzburg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-San-Sebastian.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.271Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Sofia.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Stockholm.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Strasbourg.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.272Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Tallinn.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/other-cities/trains-from-Toulouse.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.273Z","logLevel":"warn","context":"recorder","message":"Error getting cookies","details":{"page":"https://www.seat61.com/international-trains/trains-from-Venice.htm","e":{"name":"TargetCloseError","cause":{"name":"ProtocolError"}}}}
{"timestamp":"2025-02-12T20:30:50.283Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":0,"frameUrls":[],"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.320Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":0,"page":"https://www.seat61.com/european-train-booking-horizons.htm","workerid":0}}
{"timestamp":"2025-02-12T20:30:50.566Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}

I have another stuck task which is a bit different:

{"timestamp":"2025-02-12T08:56:22.845Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/goToSite/2800","workerid":0}}
{"timestamp":"2025-02-12T08:56:23.573Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://bananaguide.com/article/67290/you-love-jack-review"}}
{"timestamp":"2025-02-12T08:56:23.575Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":917,"total":4682,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2025-02-12T08:56:22.877Z\",\"extraHops\":0,\"url\":\"https:\\/\\/bananaguide.com\\/article\\/67290\\/you-love-jack-review\",\"added\":\"2025-02-12T07:48:33.215Z\",\"depth\":3}"]}}
{"timestamp":"2025-02-12T08:56:24.963Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:25.299Z","logLevel":"warn","context":"recorder","message":"Skipping URL from unknown frame","details":{"url":"https://bananaguide.com/article/67290/you-love-jack-review","frameId":"77D42A85B5CCDF255253DE5BCC2CE7FA"}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://bananaguide.com/article/67290/you-love-jack-review"],"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.136Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.658Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://bananaguide.com/article/67290/you-love-jack-review","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:27.659Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.660Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T08:56:28.664Z","logLevel":"info","context":"general","message":"Size threshold reached 4307928621 >= 4294967296, stopping","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"general","message":"Crawler interrupted, gracefully finishing current pages","details":{}}
{"timestamp":"2025-02-12T08:56:28.692Z","logLevel":"info","context":"worker","message":"Worker done, all tasks complete","details":{"workerid":0}}
{"timestamp":"2025-02-12T08:57:24.804Z","logLevel":"warn","context":"recorder","message":"Failed to load response body","details":{"url":"https://www.lucasentertainment.com/tour/?nats=MTAwMDA0MC4yNy4yLjU1LjAuMC4wLjAuMA","networkId":"00904D649DE4D581E0F7CD7BF89BAAA8","type":"exception","message":"Protocol error (Fetch.getResponseBody): Target closed","stack":"TargetCloseError: Protocol error (Fetch.getResponseBody): Target closed\n    at CallbackRegistry.clear (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/CallbackRegistry.js:77:36)\n    at CdpCDPSession._onClosed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/CDPSession.js:106:25)\n    at #onClose (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Connection.js:171:21)\n    at WebSocket.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/node/NodeWebSocketTransport.js:43:30)\n    at callListener (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)\n    at WebSocket.onClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:220:9)\n    at WebSocket.emit (node:events:518:28)\n    at WebSocket.emitClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:272:10)\n    at Socket.socketOnClose (/app/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1341:15)\n    at Socket.emit (node:events:518:28)","page":"https://bananaguide.com/article/67290/you-love-jack-review","workerid":0}}
{"timestamp":"2025-02-12T20:34:03.730Z","logLevel":"info","context":"general","message":"SIGTERM received...","details":{}}
{"timestamp":"2025-02-12T20:34:03.732Z","logLevel":"info","context":"general","message":"SIGNAL: stopping crawl now...","details":{}}
{"timestamp":"2025-02-12T20:34:03.735Z","logLevel":"info","context":"general","message":"Exiting, Crawl status: interrupted","details":{}}
@benoit74 benoit74 changed the title Crawler exit stuck while waiting for cookies? Crawler exit stuck Feb 12, 2025
@benoit74 benoit74 mentioned this issue Feb 12, 2025
Open
@benoit74
Copy link
Contributor Author

Issue still present on 1.5.4 ; I strongly suspect this is somehow linked to new retry logic and the usage of sizeLimit / timeLimit. How can I help to further diagnose the problem?

@ikreymer
Copy link
Member

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

@ikreymer
Copy link
Member

Have not seen that before - and I don't think its related to retries, since all that happens after is:

    logger.info(`Exiting, Crawl status: ${status}`);

    await this.closeLog();

    if (this.crawlState && status) {
      await this.crawlState.setStatus(status);
    }
    process.exit(exitCode);

We could add a timeout to closeLog() and setStatus(), have not seen any issues stalling there before..

@benoit74
Copy link
Contributor Author

Hm, just to confirm, the crawler prints Exiting, Crawl status: interrupted on its own but does not exit from there?

No, sorry, look at timestamps, it got stuck at Failed to load response body. Then I sent a SIGTERM and it printed the last two lines. I kept them as reference in the logs to indicate that process was really still alive, waiting for something to happen

@benoit74
Copy link
Contributor Author

And first task got stuck at Worker done, all tasks complete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Webrecorder Projects
Status: Triage
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ikreymer @benoit74