🤖 Agentic RAG Pipeline

A powerful Retrieval-Augmented Generation (RAG) system built with LangGraph, FastAPI, and Qdrant Cloud. This project provides an intelligent document processing and question-answering system that can ingest PDFs and other documents, then answer questions using advanced AI techniques.

✨ Features

• 🚀 Fast Document Ingestion: Upload PDFs and other documents via REST API
• 🧠 Intelligent RAG Pipeline: LangGraph-powered multi-step retrieval and generation
• 🔍 Advanced Vector Search: Qdrant Cloud integration for efficient similarity search
• 🌐 RESTful API: Clean FastAPI endpoints for easy integration
• 🔧 Configurable: Flexible chunking, embedding, and generation parameters
• ☁️ Cloud-Ready: Built for production with Qdrant Cloud and OpenAI

🏗️ Architecture

Overall System Architecture

graph TB
    A[📄 Documents] --> B[🔧 Document Processor]
    B --> C[📊 Vector Store<br/>Qdrant Cloud]
    C --> D[🔍 LangGraph RAG Pipeline]
    D --> E[🤖 OpenAI LLM]
    E --> F[💬 Answer]
    
    G[🌐 FastAPI] --> H[📤 /ingest]
    G --> I[❓ /query]
    H --> B
    I --> D

Loading

Advanced Query Processing Pipeline

Our RAG system uses an innovative Query Decomposition + Parallel Retrieval architecture that significantly improves retrieval accuracy:

Key Features:

• Query Decomposition: Breaks complex queries into focused, searchable components
• Parallel Retrieval: Simultaneous searches across multiple query aspects
• Semantic Fusion: Intelligent combination of results from different searches
• Component-Level Rewriting: Targeted improvement of failed query components
• Relevance Judgment: LLM-powered evaluation of retrieved context quality

Example Flow:

1. Input: "eye surgery in Dubai"
2. Decomposition: ["What is covered for eye surgery procedures?", "What coverage is available for medical treatment in Dubai or UAE?", "What are the international medical coverage terms?"]
3. Parallel Search: All three queries search simultaneously
4. Fusion: Results are intelligently combined and deduplicated
5. Answer: Comprehensive response with page references

🚀 Quick Start

Prerequisites

• Python 3.9+
• OpenAI API key
• Qdrant Cloud account (free tier available)

1. Installation

# Clone the repository
git clone https://github.com/yourusername/agentic-rag-pipeline.git
cd agentic-rag-pipeline/app

# Install dependencies
pip install -e .

# Copy environment template
cp env.example .env

2. Configuration

Edit your .env file:

# OpenAI Configuration (Required)
OPENAI_API_KEY=your_openai_api_key_here

# Qdrant Cloud Configuration (Required)
QDRANT_URL=https://your-cluster-url.qdrant.tech
QDRANT_API_KEY=your_qdrant_api_key_here
QDRANT_COLLECTION=documents

# Optional: API Configuration
API_HOST=0.0.0.0
API_PORT=8000

3. Run the Application

python run.py

The API will be available at http://localhost:8000

📚 API Documentation

Document Ingestion

Upload PDF Files

curl -X POST "http://localhost:8000/ingest" \
     -H "Content-Type: multipart/form-data" \
     -F "files=@document1.pdf" \
     -F "files=@document2.pdf"

Response

{
  "message": "Successfully processed 2 files",
  "processed_files": [
    {
      "filename": "document1.pdf",
      "chunks": 15,
      "status": "success"
    }
  ]
}

Query Documents

Ask Questions

curl -X POST "http://localhost:8000/query" \
     -H "Content-Type: application/json" \
     -d '{"query": "What is machine learning?"}'

Response

{
  "answer": "Machine learning is a subset of artificial intelligence...",
  "sources": [
    {
      "content": "Relevant document excerpt...",
      "metadata": {"source": "document1.pdf", "page": 1}
    }
  ]
}

Health Check

curl -X GET "http://localhost:8000/"

Project Structure

src/agent/
├── simple_api.py          # FastAPI application
├── graph.py               # LangGraph RAG pipeline
├── simple_processor.py    # Document processing
└── simple_vector_store.py # Qdrant integration

Key Components

1. Document Processor (simple_processor.py): Handles PDF parsing and text extraction
2. Vector Store (simple_vector_store.py): Manages Qdrant Cloud operations
3. RAG Pipeline (graph.py): LangGraph workflow for retrieval and generation
4. API Layer (simple_api.py): FastAPI endpoints and request handling

Adding New Features

1. New Document Types: Extend SimpleDocumentProcessor with additional loaders
2. Custom Chunking: Modify chunking logic in the processor
3. Enhanced RAG: Add new nodes to the LangGraph pipeline
4. API Endpoints: Add new routes in simple_api.py

🔧 Configuration Options

Document Processing

• Chunk Size: Text chunk size (default: 1000 characters)
• Chunk Overlap: Overlap between chunks (default: 200 characters)
• Supported Formats: PDF, TXT, MD, HTML, DOCX

Vector Store

• Collection Name: Qdrant collection identifier
• Embedding Model: OpenAI text-embedding-3-small (default)
• Similarity Threshold: Minimum similarity score for retrieval

RAG Pipeline

• Max Tokens: Maximum response length
• Temperature: Response creativity (0.0-1.0)
• Retrieval Count: Number of relevant chunks to retrieve

OPENAI_API_KEY=your_production_key
QDRANT_URL=your_production_cluster
QDRANT_API_KEY=your_production_key
QDRANT_COLLECTION=production_docs

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• LangChain for the RAG framework
• LangGraph for the agentic pipeline
• Qdrant for vector storage
• FastAPI for the API framework
• OpenAI for embeddings and language models