-
Notifications
You must be signed in to change notification settings - Fork 13
feat: KGRag - Knowledge Graph-Enhanced RAG with Mellea #39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
ydzhu98
wants to merge
36
commits into
generative-computing:main
Choose a base branch
from
ydzhu98:yzhu/missing_components
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
e11a209
add kgrag layer 4
7dfd58d
fix some formatting issue
ec5e2d6
Layer 3 of knowledge graph
bd107db
layer 2 of the project
bebc66b
layer 1 of kgrag
4e30946
Fill in the most important functions of KGRag
36457f3
Fill in the core missing functions
411ba59
Finished the less important functions
1ebf39c
Adding testing files
44825be
Fix issues found in the testing
42964ec
Implement Phase 2: 8 runnable scripts for KG-RAG pipeline
6e119ba
Extract common functions from the scripts
35919aa
Add test function for the utility files
aa4fa5d
Add the final run.sh script so that one can call it directly.
7be0ce1
Add two read me files for the library and the example
4e994c9
fix the issue so that run.sh is runable. However further update are e…
3cf11b4
Rewrite preprocessing and embedding scripts for mellea-style pipeline
b130db0
Enhance run_kg_update.py to match mellea's architecture and functiona…
eb017a3
Add comprehensive KG-RAG pipeline architecture documentation
60073db
Add run_kg_update.py as Step 3 to run.sh pipeline
d8a603d
Enable run_kg_update.py to load LLM/Neo4j config from .env file
e40da15
Fix run_kg_update.py to properly use RITS model from environment
272de0d
Update KG-RAG example README with three-stage pipeline documentation
2b36a1e
Remove large data files from git tracking (keep locally)
fca3a15
Add dataset directory documentation for large data files
523b244
update the run.sh for step 0
6e08755
Fix the run_kg_update and run_qa
6c70efe
fix the run_eval.py and readme
fe0f99e
remove some intermediate files
e025a6e
remove the data from git
2ad1b85
Refactor the scripts to better utilize the library
8d75eab
Address some of the issue from comments
427f644
Additional changes found during review
f3af8ba
Address additional issues of the comments from the PR
7a15435
Fix the embeder after refactoring
6125e56
Update embedder
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. | ||
|
|
||
| ## Project Overview | ||
|
|
||
| **mellea-contribs** is an incubation repository for contributions to the Mellea ecosystem. It provides a library for incubating generative programming tools and utilities that integrate with the Mellea LLM agent framework. | ||
|
|
||
| - **Tech Stack**: Python 3.10+, PDM build system, Mellea framework with LiteLLM | ||
| - **Key Dependencies**: rapidfuzz (fuzzy matching), eyecite (legal citations), playwright (web scraping), markdown | ||
| - **License**: Apache License 2.0 | ||
|
|
||
| ## Common Development Commands | ||
|
|
||
| ### Setup and Installation | ||
|
|
||
| ```bash | ||
| # Install with development dependencies | ||
| pdm install --group dev | ||
|
|
||
| # Or using uv (faster): | ||
| uv pip install -e . --group dev | ||
| ``` | ||
|
|
||
| ### Code Quality | ||
|
|
||
| ```bash | ||
| # Format code (ruff is used for both formatting and linting) | ||
| ruff format . | ||
| ruff check . --fix | ||
|
|
||
| # Type checking | ||
| mypy mellea_contribs/ | ||
|
|
||
| # Run all linters individually | ||
| isort mellea_contribs/ test/ | ||
| pylint mellea_contribs/ test/ | ||
| ruff check mellea_contribs/ test/ | ||
|
|
||
| # Run pre-commit hooks | ||
| pre-commit run --all-files | ||
| ``` | ||
|
|
||
| ### Testing | ||
|
|
||
| ```bash | ||
| # Run all tests | ||
| pytest | ||
|
|
||
| # Run a specific test file | ||
| pytest test/test_citation_exists.py | ||
|
|
||
| # Run a specific test function | ||
| pytest test/test_citation_exists.py::test_function_name | ||
|
|
||
| # Run tests with verbose output | ||
| pytest -v | ||
|
|
||
| # Run only tests marked as qualitative (LLM-dependent) | ||
| pytest -m qualitative | ||
|
|
||
| # Run tests excluding qualitative tests (useful in CI) | ||
| pytest -m "not qualitative" | ||
|
|
||
| # Run tests with asyncio verbose output (asyncio_mode is auto-configured) | ||
| pytest -v --asyncio-mode=auto | ||
| ``` | ||
|
|
||
| ### Documentation | ||
|
|
||
| ```bash | ||
| # Build Sphinx documentation | ||
| sphinx-build -b html docs/ docs/_build/ | ||
| ``` | ||
|
|
||
| ## Architecture Overview | ||
|
|
||
| ### Module Organization | ||
|
|
||
| **`mellea_contribs/tools/`** - Reusable LLM-based selection and ranking algorithms: | ||
| - `top_k.py`: Generic Top-K selection engine using LLM judgment. Selects top K from N items with rejection sampling and caching. | ||
| - `double_round_robin.py`: Pairwise comparison scoring engine. Performs all-pair comparisons and returns items ranked by accumulated scores. | ||
|
|
||
| **`mellea_contribs/reqlib/`** - Domain-specific validators extending Mellea's validation framework: | ||
| - `citation_exists.py`: Validates legal case citations via case.law metadata + fuzzy matching | ||
| - `is_appellate_case.py`: Classifies cases as appellate by court abbreviation patterns | ||
| - `import_repair.py`: Fixes Python import errors in LLM-generated code via AST analysis | ||
| - `import_resolution.py`: Parses and resolves module not found / import errors with confidence-scored suggestions | ||
| - `grounding_context_formatter.py`: Structures multi-field context for LLM prompts (auto-skips empty fields) | ||
| - `common_aliases.py`: Module name mappings and relocations for import resolution | ||
| - `statute_data.py`: Legal statute data handling | ||
|
|
||
| **`mellea_contribs/kg/`** - Knowledge Graph database abstraction: | ||
| - `base.py`: Core data structures (GraphNode, GraphEdge, GraphPath) | ||
| - `graph_dbs/base.py`: GraphBackend abstract interface | ||
| - `graph_dbs/neo4j.py`: Production Neo4j implementation (requires [kg] dependencies) | ||
| - `graph_dbs/mock.py`: In-memory mock backend for testing without infrastructure | ||
| - `components/`: Query, result, traversal components (minimal Layer 4 implementations) | ||
|
|
||
| **Installation**: `pip install mellea-contribs[kg]` for Neo4j support | ||
|
|
||
| ### Design Patterns | ||
|
|
||
| **Mellea Integration Pattern**: All validators are Requirement classes with a `validation_fn(output) → ValidationResult`. This enables iterative LLM refinement via the Instruct-Validate-Repair loop. | ||
|
|
||
| **Caching Strategy**: Tools use decorator-based caching keyed on item hash + context + prompts to avoid redundant LLM calls. | ||
|
|
||
| **Model Interaction**: Uses `mellea.instruct()` with grounding context, system prompts for output formatting (JSON arrays, single tokens), and rejection sampling (loop_budget=2) for reliability. | ||
|
|
||
| **Data Validation Layers**: | ||
| - Legal citations: Direct lookup → fuzzy match → LLM judgment | ||
| - Python imports: Static AST analysis + dynamic error parsing | ||
| - Court classification: Pattern matching + database lookup | ||
|
|
||
| ### Test Infrastructure | ||
|
|
||
| - Large test databases: `test/data/citation_exists_database.json` (~2.8MB) | ||
| - Qualitative test markers for LLM-dependent tests (auto-xfail in CI when `MELLEA_SKIP_QUALITATIVE` env var is set) | ||
| - Neo4j integration tests: Marked with `@pytest.mark.neo4j`, skipped unless NEO4J_URI is set | ||
| - Separate database for CI efficiency | ||
| - Shared fixtures via `test/conftest.py` | ||
| - Async-first testing: `asyncio_mode = "auto"` (no explicit marking needed) | ||
|
|
||
| ## Code Quality Standards | ||
|
|
||
| - **Docstrings**: Google-style convention (enforced via ruff rule D) | ||
| - **Complexity**: Maximum cyclomatic complexity of 20 (ruff C901) | ||
| - **Type Hints**: Full type annotation with mypy checking enabled | ||
| - **Imports**: isort enforced for consistent ordering with `combine-as-imports` | ||
| - **Pre-commit hooks**: Automatic formatting and validation on commit | ||
|
|
||
| ## Release and Versioning | ||
|
|
||
| - **Semantic Versioning**: Angular commit parser (feat → minor, fix → patch) | ||
| - **Automated Releases**: python-semantic-release on main branch | ||
| - **CI/CD**: GitHub Actions workflows (ci.yml, cd.yml, pypi.yml) | ||
| - **PyPI Publishing**: Automated on semantic version tags |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Graph Database Configuration (Neo4j default; adjust for other graph DBs) | ||
| NEO4J_URI=bolt://localhost:7687 | ||
| NEO4J_USER=neo4j | ||
| NEO4J_PASSWORD=password | ||
|
|
||
| # Data Directory | ||
| KG_BASE_DIRECTORY=./dataset | ||
| DATA_PATH=./data | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Primary LLM — any OpenAI-compatible endpoint | ||
| # --------------------------------------------------------------------------- | ||
| # Option A: OpenAI | ||
| # API_KEY=sk-... | ||
| # MODEL_NAME=gpt-4o-mini | ||
|
|
||
| # Option B: Local Ollama | ||
| # API_BASE=http://localhost:11434/v1 | ||
| # API_KEY=ollama | ||
| # MODEL_NAME=llama3.2 | ||
|
|
||
| # Option C: vLLM / self-hosted | ||
| # API_BASE=http://localhost:8000/v1 | ||
| # API_KEY=dummy | ||
| # MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct | ||
|
|
||
| # Option D: Azure OpenAI | ||
| # API_BASE=https://<your-resource>.openai.azure.com/openai/deployments/<deployment>/ | ||
| # API_KEY=<azure-key> | ||
| # MODEL_NAME=gpt-4o-mini | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Optional: Separate evaluation model (defaults to primary LLM if unset) | ||
| # --------------------------------------------------------------------------- | ||
| # EVAL_API_BASE=... | ||
| # EVAL_API_KEY=... | ||
| # EVAL_MODEL_NAME=... | ||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Optional: Embedding model for vector entity alignment | ||
| # --------------------------------------------------------------------------- | ||
| # EMB_API_BASE=http://localhost:11434/v1 | ||
| # EMB_API_KEY=ollama | ||
| # EMB_MODEL_NAME=nomic-embed-text | ||
| # VECTOR_DIMENSIONS=768 | ||
|
|
||
| # Request Configuration | ||
| MAX_RETRIES=3 | ||
| TIME_OUT=1800 | ||
|
|
||
| # OpenTelemetry — disable if you don't have a collector running | ||
| OTEL_SDK_DISABLED=true |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This .env_template should be cleaned. Do not try to make it RITS specific. Make it LLM server ergodic. Especially, make it work for openai, ollama, etc.