Multi-provider search with Exa → Tavily → Serper → DuckDuckGo fallback chain #219
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Exa.ai provides AI-native neural search which offers significant advantages for research agents: - Semantic understanding: Finds relevant results based on meaning, not just keyword matching - Query optimization: Built-in autoprompt improves query quality - Direct content retrieval: Can fetch full page text in a single call - Better for complex queries: Neural embeddings excel at nuanced research questions This change simplifies the codebase by removing the dual search provider system and standardizing on Exa.ai.
- Add category parameter to filter results (research paper, news, github, etc.) - Add AI-generated highlights for better content extraction - Include author information in search results - Document all available Exa categories in docstrings
Implements multi-provider search with automatic fallback: - Exa.ai as primary (best semantic search quality) - Serper.dev as first fallback (Google results) - DuckDuckGo as second fallback (free, always available) Fallback triggers: missing API key, rate limit, or API errors. Each provider has 3 retry attempts before failing over.
chindris-mihai-alexandru
changed the title
- Reorder providers by quality: Exa -> Tavily -> Serper -> DuckDuckGo - Tavily is purpose-built for RAG/LLMs with 1,000 free requests/month - Simplify implementation with cleaner function structure - Remove unnecessary class-based approach for providers - Update .env.example with TAVILY_API_KEY documentation
chindris-mihai-alexandru
changed the title
- Use Exa type: auto instead of neural for better results - Switch Tavily to Bearer token auth (per API docs) - Add Exa 402 (payment required) error handling - Add Tavily 432/433 quota error handling - Switch Serper from http.client to requests for consistency - Add sanitize_query() for input validation - Add format_results() for consistent output - Handle DuckDuckGoSearchException - Add ConnectionError, JSONDecodeError handling - Better error messages with provider names
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a multi-provider search system with automatic fallback for DeepResearch. It provides a 4-provider fallback chain optimized for quality and reliability.
Fallback Chain
The system automatically falls back to the next provider when:
Why This Approach?
Provider Comparison
Changes
inference/tool_search.py: Complete rewrite with 4-provider fallbacksearch_exa(),search_tavily(),search_serper(),search_duckduckgo()sanitize_query()for input validationformat_results()for consistent outputmulti_provider_search()orchestrates the fallback chain.env.example: Updated with all provider API keysConfiguration
Error Handling
Testing
Tested all four paths:
type: "auto"search