Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests take too long #235

Open
wRAR opened this issue Dec 6, 2024 · 1 comment
Open

Tests take too long #235

wRAR opened this issue Dec 6, 2024 · 1 comment

Comments

@wRAR
Copy link
Contributor

wRAR commented Dec 6, 2024

A full test run is now about 20-30 minutes which is a problem for testing changes. I tried to investigate it and so far I've found that a single spider run in tests takes 5-7 seconds here, if ScrapyZyteAPIDownloadHandler is enabled. I will continue the investigation later, and if we cannot fix this we may want to enable parallel test runs.

@wRAR
Copy link
Contributor Author

wRAR commented Jan 8, 2025

The reason for "a single spider run in tests takes 5-7 seconds" is that the first scrapy.core.engine.ExecutionEngine._next_request() doesn't do anything and the next one is only called after 5s (self.slot.heartbeat.start(5)). It doesn't do anything because by the time it's first executed, the engine isn't yet marked as running, because scrapy.core.engine.ExecutionEngine.start() waits until all engine_started handlers finish (and both scrapy-zyte-api download handlers have non-trivial async engine_started handlers, though that's likely not important), and my logging shows that the first _next_request() call is between they finish and self.running = True is executed. Which may be just an unfortunate order of coroutines executed, but I don't know if Scrapy is really expected to wait 5s before downloading the first request in certain cases so maybe there is a different underlying cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant