detect logout URLs automatically

[kmike]

We should prevent logging out caused by following logout links.

[lopuhin]

There is some logout detection here

undercrawler/undercrawler/middleware/autologin.py

Line 133 in 9ecc4a2

any(self.auth_cookies.get(name) and m.value == ''

but it is not quite accurate: for example it does not work for the phpbb3, where on logout one of the cookies is set to 1, not to an empty string (in fact it works there). If we detect logout, we try to log in again, and will avoid this link in the future.

There are two more problems with current logout mechanics:

• the logout can happen due to cookie expiration, not due to following a logout link, but we will still mark this link as a logout link and will avoid revisiting it. This can be solved by requiring at least two logout to mark link as leading to logout.
• the logout link could be unique each time (for example ?action=logout&sid=<session id here>) - in this case we would be logging out periodically, which will slow the crawl (we will be also retrying all requests that were in-fly or scheduled in twisted when the logout happened). I don't known how to solve this apart from using some ML.

Did you have something more specific in mind here @kmike ?

[kmike]

Nothing more specific; I was thinking about howto detect logout links before following them. Maybe even just 'logout' in URL or link text covers 99% cases.

[lopuhin]

Hm, just though that it would be even better if this support was off when the site is skipped, and autologin is effectively disabled, although AUTOLOGIN_ENABLED is True.

[lopuhin]

Hm, just though that it would be even better if this support was off when the site is skipped, and autologin is effectively disabled, although AUTOLOGIN_ENABLED is True.

Done in 794526f