You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A full test run is now about 20-30 minutes which is a problem for testing changes. I tried to investigate it and so far I've found that a single spider run in tests takes 5-7 seconds here, if ScrapyZyteAPIDownloadHandler is enabled. I will continue the investigation later, and if we cannot fix this we may want to enable parallel test runs.
The text was updated successfully, but these errors were encountered:
The reason for "a single spider run in tests takes 5-7 seconds" is that the first scrapy.core.engine.ExecutionEngine._next_request() doesn't do anything and the next one is only called after 5s (self.slot.heartbeat.start(5)). It doesn't do anything because by the time it's first executed, the engine isn't yet marked as running, because scrapy.core.engine.ExecutionEngine.start() waits until all engine_started handlers finish (and both scrapy-zyte-api download handlers have non-trivial async engine_started handlers, though that's likely not important), and my logging shows that the first _next_request() call is between they finish and self.running = True is executed. Which may be just an unfortunate order of coroutines executed, but I don't know if Scrapy is really expected to wait 5s before downloading the first request in certain cases so maybe there is a different underlying cause.
A full test run is now about 20-30 minutes which is a problem for testing changes. I tried to investigate it and so far I've found that a single spider run in tests takes 5-7 seconds here, if
ScrapyZyteAPIDownloadHandler
is enabled. I will continue the investigation later, and if we cannot fix this we may want to enable parallel test runs.The text was updated successfully, but these errors were encountered: