Skip to content

Fix DataForSEO backend (never returned data) and urllib3 incompatibility - #10

Open
dspv wants to merge 1 commit into
akvise:masterfrom
dspv:fix/dataforseo-backend-and-urllib3
Open

Fix DataForSEO backend (never returned data) and urllib3 incompatibility#10
dspv wants to merge 1 commit into
akvise:masterfrom
dspv:fix/dataforseo-backend-and-urllib3

Conversation

@dspv

@dspv dspv commented Jul 15, 2026

Copy link
Copy Markdown

Hi! Thanks for the tool — the 429 problem it solves is real, and the DataForSEO backend idea is exactly right. I hit two issues trying to use it for market research, so here are fixes.

1. urllib3 2.x breaks the free backend

pytrends 4.9.2 (its latest release) calls Retry(method_whitelist=...), which urllib3 removed in 2.0. A fresh pip install trends-checker resolves urllib3 2.x, so every request fails:

[error] US: Retry.__init__() got an unexpected keyword argument 'method_whitelist'
No summary data produced.

Pinned urllib3<2. Since 4.9.2 is pytrends' latest, this can't be fixed by bumping it.

2. The DataForSEO backend never returned a row

$ trends-checker --keywords "vibe coding,ai agents" --dataforseo-key user:pass
DataForSEO request failed: 'list' object has no attribute 'get'
Falling back to Google Trends...

📊 DataForSEO Trends (no rate limits)
───────────────────────────────────────────────────────
                                     <- nothing, and exit 0

The parser didn't match the API's real response shape:

  • it read item["keyword"], but the graph item carries keywords (a list);
  • it read item["data"]["values"], but data is a list of time points, each holding a values array parallel to keywords. .get() on that list raised AttributeError, which the bare except swallowed — printing "Falling back to Google Trends..." even though no fallback path exists. The command then exits 0 with an empty table, which is easy to mistake for "no data" in a cron job;
  • --geo, --group and --timeframe were never sent: every query ran worldwide / web / past-12-months regardless of flags.

This PR resolves each --geo to a location_code via the /locations endpoint (fetched once per process), maps --group and --timeframe to DataForSEO's dialect, queries per region, and prefers the API's precomputed averages.

Two small judgment calls, happy to change either:

  • Unsupported input warns instead of silently substituting. An unmapped --timeframe and countries missing from Trends' location list (e.g. RU, which is in DEFAULT_GEOS) previously would have quietly returned a different period/region than requested.
  • A region that falls back to worldwide is labelled RU -> WW, not RU — a [RU] header over worldwide numbers seemed worth avoiding.

Verification

Against the live API, geo filtering now demonstrably applies:

$ trends-checker --keywords "vibe coding,ai agents" --geo US,GB --dataforseo-key ...
[US]
  ai agents      [████████░░░░░░░░░░░░] 43/100
  vibe coding    [█░░░░░░░░░░░░░░░░░░░] 8/100
[GB]
  ai agents      [███████░░░░░░░░░░░░░] 35/100
  vibe coding    [██░░░░░░░░░░░░░░░░░░] 11/100

And the two backends now agree — pytrends returns 42.51 / 8.26 for US, DataForSEO 43 / 8. Also checked: WW, --group youtube, --timeframe "today 5-y", an unmapped timeframe, and a clean venv install (urllib3 resolves to 1.26.20, free backend works).

Two issues made a fresh install unusable on both backends.

1. urllib3 2.x breaks pytrends
   pytrends 4.9.2 (its latest release) calls Retry(method_whitelist=...), which
   urllib3 removed in 2.0. A fresh install resolves urllib3 2.x, so every request
   fails with "Retry.__init__() got an unexpected keyword argument
   'method_whitelist'". Pinned urllib3<2.

2. run_dataforseo never printed a row
   The parser didn't match the API's actual response shape:
   - it read item["keyword"], but the graph item carries `keywords` (a list);
   - it read item["data"]["values"], but `data` is a list of time points, each
     with a `values` array parallel to `keywords` — so .get() raised
     AttributeError on a list and was swallowed by the except, printing
     "Falling back to Google Trends..." while no fallback exists;
   - --geo, --group and --timeframe were ignored entirely: every query ran
     worldwide, web, past 12 months regardless of flags.

   Now it resolves each --geo to a location_code via the /locations endpoint
   (cached per process), maps --group and --timeframe to DataForSEO's dialect,
   queries per region, and reads `averages` when present. Unsupported timeframes
   and countries absent from Trends (e.g. RU) warn instead of silently changing
   what was asked for, and a region that falls back to worldwide is labelled
   "RU -> WW" rather than "RU".

Verified against the live API: US and GB return different numbers (43/8 vs
35/11 for "ai agents"/"vibe coding"), matching the pytrends backend's own
output for US (42.51/8.26) — the two backends now agree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant