-
Notifications
You must be signed in to change notification settings - Fork 20.4k
feat: add Desearch integration #31276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
b59ca53
feat: desearch
changelia acaee64
docs: update README.md with website link and API key instructions
changelia 3e8aca9
docs: add Desearch integration
changelia eeca57a
docs: update Desearch integration description with link
changelia c206744
docs: clean up Desearch documentation and update notebook metadata
changelia 2567e27
docs: enhance Desearch notebook structure with section updates and im…
changelia 00550c0
docs: refine Desearch notebook with updated section titles and additi…
changelia 311fc79
docs: update Desearch notebook to use markdown for API key and add ma…
changelia 6d7704b
docs: improve formatting and structure in Desearch notebook for bette…
changelia 36e0ffc
docs: enhance Desearch notebook with additional spacing for improved …
changelia 28fed8d
docs: reorder import statements in Desearch notebook for consistency …
changelia 087aff6
Merge branch 'master' into desearch/integration
changelia f764f6e
Merge branch 'master' into desearch/integration
changelia File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| # Desearch | ||
|
|
||
| [Desearch](https://desearch.ai/) is a static-site generator which provides out-of-the-box documentation features. | ||
|
|
||
|
|
||
| ## Installation and Setup | ||
|
|
||
|
|
||
| ```bash | ||
| pip install -U langchain-desearch | ||
| ``` | ||
|
|
||
| ## Document Loader | ||
|
|
||
| See a [usage example](/docs/integrations/tools/desearch). | ||
|
|
||
| ```python | ||
| from langchain_desearch.tools import DesearchTool, BasicWebSearchTool, BasicTwitterSearchTool | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,319 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Overview" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "AI-Powered Search Engine and API for Advanced Data Discovery [Desearch](https://desearch.ai).\n", | ||
| "\n", | ||
| "Welcome to the comprehensive guide on integrating Desearch with LangChain to enable advanced Retrieval-Augmented Generation (RAG) and agent-style workflows. Desearch is a cutting-edge, privacy-first search API that excels in delivering relevant, real-time web and Twitter content, specifically optimized for use in Large Language Model (LLM) applications.\n", | ||
| "\n", | ||
| "First, get an Desearch API key and add it as an environment variable. Get $5 free credit (plus more by completing certain actions like making your first search) by [signing up here](https://console.desearch.ai)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Setup\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| ".env DESEARCH_API_KEY=your_api_key_here" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from dotenv import load_dotenv\n", | ||
| "\n", | ||
| "load_dotenv()\n", | ||
| "# Set your API key as an environment variable" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "And install the integration package" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install -U langchain-desearch" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Instantiation" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 🛠️ Tools Overview\n", | ||
| "\n", | ||
| "Desearch provides a suite of modular tools compatible with LangChain, each designed for specific use cases:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 🔎 Search Tools\n", | ||
| "\n", | ||
| "* **DesearchTool**: A comprehensive tool for searching across web, AI, and Twitter posts.\n", | ||
| "* **BasicWebSearchTool**: A lightweight tool focused solely on web searches.\n", | ||
| "* **BasicTwitterSearchTool**: A specialized tool for searching tweets with advanced filtering options." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Invocation" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 🧪 Quick Examples" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Basic Usage" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Here's a simple example to demonstrate how to use the DesearchTool for web searches" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from langchain_desearch.tools import DesearchTool\n", | ||
| "\n", | ||
| "# Initialize the tool\n", | ||
| "tool = DesearchTool()\n", | ||
| "\n", | ||
| "# Run a search query\n", | ||
| "response = tool._run(\n", | ||
| " prompt=\"Bittensor network activity\",\n", | ||
| " tool=[\"web\"],\n", | ||
| " model=\"NOVA\",\n", | ||
| " date_filter=\"PAST_24_HOURS\",\n", | ||
| ")\n", | ||
| "\n", | ||
| "# Print the response\n", | ||
| "print(response)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "#### Search Twitter\n", | ||
| "\n", | ||
| "To search Twitter for specific topics, use the BasicTwitterSearchTool" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from langchain_desearch.tools import BasicTwitterSearchTool\n", | ||
| "\n", | ||
| "# Initialize the Twitter search tool\n", | ||
| "tool = BasicTwitterSearchTool()\n", | ||
| "\n", | ||
| "# Execute a search query\n", | ||
| "response = tool._run(query=\"AI governance\", sort=\"Top\", count=5)\n", | ||
| "\n", | ||
| "# Display the results\n", | ||
| "print(response)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Use within an agent" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 🤖 Building a RAG Chain\n", | ||
| "\n", | ||
| "Creating a RAG chain involves integrating Desearch with LangChain's prompt and LLM capabilities" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from langchain_core.output_parsers import StrOutputParser\n", | ||
| "from langchain_core.prompts import ChatPromptTemplate, PromptTemplate\n", | ||
| "from langchain_core.runnables import RunnableParallel, RunnablePassthrough\n", | ||
| "from langchain_deepseek import ChatDeepSeek\n", | ||
| "from langchain_desearch.tools import DesearchTool\n", | ||
| "\n", | ||
| "# Initialize the Desearch tool\n", | ||
| "tool = DesearchTool()\n", | ||
| "\n", | ||
| "\n", | ||
| "# Define a function to fetch context\n", | ||
| "def fetch_context(query):\n", | ||
| " return tool._run(prompt=query, tool=\"desearch_web\", model=\"NOVA\")\n", | ||
| "\n", | ||
| "\n", | ||
| "# Create a prompt template\n", | ||
| "prompt = ChatPromptTemplate.from_messages(\n", | ||
| " [\n", | ||
| " (\"system\", \"You are a research assistant.\"),\n", | ||
| " (\n", | ||
| " \"human\",\n", | ||
| " \"Answer this using the provided context:\\n\\n<context>{context}</context>\",\n", | ||
| " ),\n", | ||
| " ]\n", | ||
| ")\n", | ||
| "\n", | ||
| "# Set up the LLM\n", | ||
| "llm = ChatDeepSeek(model=\"deepseek-chat\", temperature=0.7)\n", | ||
| "parser = StrOutputParser()\n", | ||
| "\n", | ||
| "# Build the RAG chain\n", | ||
| "chain = (\n", | ||
| " RunnableParallel(\n", | ||
| " {\n", | ||
| " \"query\": RunnablePassthrough(),\n", | ||
| " \"context\": lambda q: fetch_context(q[\"query\"]),\n", | ||
| " }\n", | ||
| " )\n", | ||
| " | prompt\n", | ||
| " | llm\n", | ||
| " | parser\n", | ||
| ")\n", | ||
| "\n", | ||
| "# Invoke the chain with a query\n", | ||
| "result = chain.invoke(\"What is the latest update on the EU AI Act?\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 🧠 Using Desearch in a LangChain Agent\n", | ||
| "\n", | ||
| "Integrate Desearch into a LangChain agent to automate complex workflows:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from langchain_deepseek import ChatDeepSeek\n", | ||
| "from langchain_desearch.agent import create_search_agent\n", | ||
| "\n", | ||
| "# Initialize the LLM\n", | ||
| "llm = ChatDeepSeek(model=\"deepseek-chat\", temperature=0.7)\n", | ||
| "\n", | ||
| "# Create a search agent\n", | ||
| "agent = create_search_agent(llm=llm)\n", | ||
| "\n", | ||
| "# Invoke the agent with an input message\n", | ||
| "response = agent.invoke({\"input_message\": \"Summarize current AI regulation trends\"})\n", | ||
| "print(response[\"output\"])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## API reference\n", | ||
| "\n", | ||
| "For detailed API reference, please refer to the official documentation or the source code comments for each tool and method." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## ✅ Testing\n", | ||
| "\n", | ||
| "### Unit Tests (Mocked)\n", | ||
| "\n", | ||
| "To ensure your integration works as expected, run unit tests:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pytest tests/integration_tests/test_tools.py" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pytest tests/unit_tests/test_imports.py" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "This structure adds the required headers and organizes the content under them. You can adjust the content under each header as needed to better fit the specific details of your project." | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "language_info": { | ||
| "name": "python" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please only include documentation about the tool. You can use
tool.invoke()to show how to invoke the tool.Remove the documentation about the agent, including the create_agent code from the package as it's using deprecated agent functionality and it's not something we'd like to recommend to users.