Use session for HTTP requests in AMUniversal and USA Today downloaders#320
Open
clodpated wants to merge 1 commit intothisisparker:mainfrom
Open
Use session for HTTP requests in AMUniversal and USA Today downloaders#320clodpated wants to merge 1 commit intothisisparker:mainfrom
clodpated wants to merge 1 commit intothisisparker:mainfrom
Conversation
…oaders AMUniversalDownloader.fetch_data() and USATodayDownloader.fetch_data() and find_by_date() use bare requests.get()/requests.head() instead of self.session, bypassing the session's configured User-Agent header. The AMUniversal API (gamedata.services.amuniversal.com) returns 403 to the default python-requests User-Agent. Using self.session sends the xword-dl User-Agent configured by BaseDownloader, which resolves the issue. This also brings these downloaders in line with the pattern used by other downloaders (AmuseLabs, CrosswordCompiler, etc.) that already use self.session for their HTTP requests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AMUniversalDownloader.fetch_data()uses barerequests.get()instead ofself.session.get(), bypassing the session's configured User-Agent headergamedata.services.amuniversal.com) returns 403 to the defaultpython-requestsUser-Agent, making the Universal downloader failUSATodayDownloader.fetch_data()andUSATodayDownloader.find_by_date()(requests.get()andrequests.head())Switching to
self.sessionsends thexword-dl/VERSIONUser-Agent configured byBaseDownloader, which resolves the 403. This also brings these two downloaders in line with the pattern used by other downloaders (AmuseLabs, CrosswordCompiler, etc.) that already useself.session.Changes
Three one-line changes in
amuniversaldownloader.py:AMUniversalDownloaderfetch_datarequests.get(solver_url)self.session.get(solver_url)USATodayDownloaderfind_by_daterequests.head(url)self.session.head(url)USATodayDownloaderfetch_datarequests.get(solver_url)self.session.get(solver_url)Test plan
xword-dl unisuccessfully downloads Universal puzzle (previously returned 403)xword-dl usastill works correctlyrequestsimport is still needed (requests.HTTPErrorreference on line 138)