Fix DataForSEO backend (never returned data) and urllib3 incompatibility - #10
Open
dspv wants to merge 1 commit into
Open
Fix DataForSEO backend (never returned data) and urllib3 incompatibility#10dspv wants to merge 1 commit into
dspv wants to merge 1 commit into
Conversation
Two issues made a fresh install unusable on both backends.
1. urllib3 2.x breaks pytrends
pytrends 4.9.2 (its latest release) calls Retry(method_whitelist=...), which
urllib3 removed in 2.0. A fresh install resolves urllib3 2.x, so every request
fails with "Retry.__init__() got an unexpected keyword argument
'method_whitelist'". Pinned urllib3<2.
2. run_dataforseo never printed a row
The parser didn't match the API's actual response shape:
- it read item["keyword"], but the graph item carries `keywords` (a list);
- it read item["data"]["values"], but `data` is a list of time points, each
with a `values` array parallel to `keywords` — so .get() raised
AttributeError on a list and was swallowed by the except, printing
"Falling back to Google Trends..." while no fallback exists;
- --geo, --group and --timeframe were ignored entirely: every query ran
worldwide, web, past 12 months regardless of flags.
Now it resolves each --geo to a location_code via the /locations endpoint
(cached per process), maps --group and --timeframe to DataForSEO's dialect,
queries per region, and reads `averages` when present. Unsupported timeframes
and countries absent from Trends (e.g. RU) warn instead of silently changing
what was asked for, and a region that falls back to worldwide is labelled
"RU -> WW" rather than "RU".
Verified against the live API: US and GB return different numbers (43/8 vs
35/11 for "ai agents"/"vibe coding"), matching the pytrends backend's own
output for US (42.51/8.26) — the two backends now agree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! Thanks for the tool — the 429 problem it solves is real, and the DataForSEO backend idea is exactly right. I hit two issues trying to use it for market research, so here are fixes.
1.
urllib32.x breaks the free backendpytrends 4.9.2 (its latest release) calls
Retry(method_whitelist=...), which urllib3 removed in 2.0. A freshpip install trends-checkerresolves urllib3 2.x, so every request fails:Pinned
urllib3<2. Since 4.9.2 is pytrends' latest, this can't be fixed by bumping it.2. The DataForSEO backend never returned a row
The parser didn't match the API's real response shape:
item["keyword"], but the graph item carrieskeywords(a list);item["data"]["values"], butdatais a list of time points, each holding avaluesarray parallel tokeywords..get()on that list raisedAttributeError, which the bareexceptswallowed — printing "Falling back to Google Trends..." even though no fallback path exists. The command then exits 0 with an empty table, which is easy to mistake for "no data" in a cron job;--geo,--groupand--timeframewere never sent: every query ran worldwide / web / past-12-months regardless of flags.This PR resolves each
--geoto alocation_codevia the/locationsendpoint (fetched once per process), maps--groupand--timeframeto DataForSEO's dialect, queries per region, and prefers the API's precomputedaverages.Two small judgment calls, happy to change either:
--timeframeand countries missing from Trends' location list (e.g. RU, which is inDEFAULT_GEOS) previously would have quietly returned a different period/region than requested.RU -> WW, notRU— a[RU]header over worldwide numbers seemed worth avoiding.Verification
Against the live API, geo filtering now demonstrably applies:
And the two backends now agree — pytrends returns 42.51 / 8.26 for US, DataForSEO 43 / 8. Also checked:
WW,--group youtube,--timeframe "today 5-y", an unmapped timeframe, and a clean venv install (urllib3 resolves to 1.26.20, free backend works).