Repository navigation
[NUTCH-1564] AdaptiveFetchSchedule sync_delta forces refetch of unmodified pages - #880
Conversation
In setFetchSchedule, make sure 'refTime' is not in the past. Add unit test to reproduce the situation described in Jira. Unrelated fix in FetcherThread
Convert the fraction of the delta to a ratio of max interval, to avoid next fetchTime in the past. Add unit tests for different scenarios.
|
Nice work @igiguere |
sebastian-nagel
left a comment
sebastian-nagel
left a comment
There was a problem hiding this comment.
Hi @igiguere, thanks for this fix of a long outstanding bug! Highly appreciated!
There are only few minor points which should be improved, see the inline comments.
One additional point: there are "deactivated" unit tests for NUTCH-1564 implemented as part of NUTCH-1502. Maybe you can move the unit tests from TODOTestCrawlDbStates.java into TestCrawlDbStates.java and then delete the class TODOTestCrawlDbStates.java. You can then verify whether the "activated" unit tests pass per ant test-core -Dtestcase=TestCrawlDbStates. Thanks!
Add TestCrawlDbStatesExtended (was TODOTestCrawlDbStates)
Done. I renamed TODOTestCrawlDbStates as TestCrawlDbStatesExtended to avoid a clash with the existing TestCrawlDbStates. |
sebastian-nagel
left a comment
sebastian-nagel
left a comment
There was a problem hiding this comment.
Thanks, @igiguere!
Will merge in a few days, but waiting for a potential second review.
3 participants
Ticket
https://issues.apache.org/jira/browse/NUTCH-1564
Description
For a full description of the issue, please refer to the ASF Jira ticket.
Solution
If the
offsetcalculated from thedelta(difference between last fetch time and last modification time) andsync_delta_rateis larger than themax_interval, then, theoffsetis re-calculated proportionaly to themax_interval.This ensures that when the
interval(most likely themax_interval) is added to therefTime, the resulting newfetchTimeis not is the past, triggering an immediate re-fetch.Note that I also played with some "brute force" ideas:
offset>max_interval, then setrefTimeto currentfetchTimeoffset>max_interval, then re-setoffsettooffset-max_interval(i.e.: 9-7=2), then, calculaterefTimeas before from that. (equivalent tofetchTime- 2, in the example)The suggested approach allows a smooth-ish selection of the next fetch time, relative to the gap between fetch time and last modification time.
Unrelated change in FetcherThread required on my side because my IDE runs on Java 21. Nutch was built separately on Java 17 too.
Tests
ant clean runtime test