fix(duckduckgo): rate limit errors, and add new search engines option - #3829
Open
nikkoyuthinkcol wants to merge 7 commits into
Open
fix(duckduckgo): rate limit errors, and add new search engines option#3829nikkoyuthinkcol wants to merge 7 commits into
nikkoyuthinkcol wants to merge 7 commits into
Conversation
…e limits Workflows that loop over several search nodes fire dozens of scrapes from one host within a minute. ddgs then gets non-200 or captcha pages from every engine and surfaces it as "No results found.", which aborts the run. - Wrap DDGS calls in search_with_retry: 3 attempts with a fresh DDGS() each time (new fingerprint, reshuffled engines), exponential backoff 2s/4s plus jitter, and a process-wide 2s throttle so parallel nodes queue instead of bursting. - Raise the ddgs timeout from 5s to 10s so slower engines are not dropped before they answer, leaving only the ones most likely to be blocked. - Add an optional `backend` parameter to text and image search so users can pin engines that still answer from their host. - Re-raise with a message that says the host was likely blocked and how to work around it instead of the misleading upstream text. - Document rate limiting and mitigations in the README. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5g7Ac5MwrDtHJFAGcnyhi
Mairuis
reviewed
Sep 7, 2026
Contributor
There was a problem hiding this comment.
The reviewed diff adds a shared retry/throttling helper for text and image searches without new regression tests. I have not reproduced a defect in the helper.
Suggested coverage: transient failure then success, attempt exhaustion, backend/proxy forwarding, and search-slot spacing across concurrent calls, using a mocked DDGS client and clock. These are testing suggestions rather than confirmed bugs or a separate repository policy.
For the empty-result message, keep the cause qualified: the helper cannot itself distinguish a genuinely empty result from an upstream block.
Add regression tests for the shared ddgs retry helper, mocking the DDGS client, the clock and the jitter source so no test touches the network: transient failure then success, attempt exhaustion with exact backoff delays, both final-error message branches, proxy/backend forwarding, and search-slot spacing for parallel and looped callers. Also qualify the message raised when every attempt returns "No results found.". ddgs collapses an upstream block and a genuinely empty result set into the same exception and discards the status code, so the helper cannot distinguish them; the message now says a block is the likely cause rather than asserting it, and keeps the workarounds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GMMc1xxBx6EwADxtHQfhyA
Contributor
Author
|
Hi @Mairuis , I've created some tests, mind to review again? Thanks. |
Main independently bumped duckduckgo to 0.0.12 for an icon/dependency refresh, colliding with this branch's own bump. Bump once more so the rate-limit fix stays a distinct, installable version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HTVkHb8tWqz3qoaP6aVqV7
…x/duckduckgo-rate-limit
crazywoola
approved these changes
Sep 11, 2026
crazywoola
had a problem deploying
to
tools/duckduckgo
September 11, 2026 14:29 — with 
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #3803. Search now works, but if you search too many times, it will have rate limit error. The error message is not obvious though, with
error type: DDGSException, error details: No results found.The solution is the do a failover, and exponential retry if that happens, this will be handy if users are using duckduckgo in parallel nodes.
Updated README.md as well.
Release Notes
Search no longer fails outright when your server is rate-limited by the upstream engines.
duckduckgo,brave,yahooif some engines block your host. Leave empty to rotate throughall of them.
Change Type
Screenshots / Videos
Version
versioninmanifest.yaml(not the one undermeta) — 0.0.11 → 0.0.12dify_plugin>=0.9.0Testing