feat: RDF4J Client (#3306)

* feat: add RDF4JClient, RepositoryManager, and initial Repository implementation

Also set up and add tests

* test: add test for repo listing format error and repo not healthy error

* chore: add testcontainers comment

* docs: add google style docstring for mkdocs

* test: add rdf4j client test

* chore: add todo to only run rdf4j tests on python 3.9 or greater

* chore: add todo to only run rdf4j tests on python 3.9 or greater

* refactor: organise rdf4j package into rdflib.contrib

* feat: add http_client property

* feat: bootstrap a minimal graphdb client

* chore: remove unused rdf4j testcontainer

* chore: add testcontainer pytest marker, refactor rdf4j test structure, add overwrite and get methods on Repository class and add tests for them

* test: rearrange unit tests

* feat: Repository get method

* feat: Repository delete method

* feat: Repository size method

* test: add e2e tests for the new Repository methods

* feat: add Repository upload and graphs method

* test: fix up and improve existing tests

* test: add tests for Repository.graphs

* chore: add docstring to overwrite and upload methods

* feat: defer repository manager creation until accessed, and also amend some docstring content

* feat: add RDF4J NamespaceManager

* chore: fix mypy issues

* test: add e2e tests for RDF4J Repository NamespaceManager

* chore: doc improvements

* feat: the repository's namespace prefixes are now bound to the return object of Repository.get() method

* feat: add Repository GraphStoreManager

* test: add e2e test for Repository GraphStoreManager

Also fixes a httpx limitation with key-only query params.

* chore: remove redundant re-raise of httpx exceptions

* docs: add docstring to protocol method

* style: formatting

* feat: add Repository.query method

* feat: add Repository.update method

* feat: add Repository Transaction with ping and commit

* refactor: prep Repository.size method to be used by transactions as well

* test: refactor e2e test file names to avoid clash

* test: fix test

* Revert "refactor: prep Repository.size method to be used by transactions as well"

This reverts commit 323f303.

* test: fix test

* feat: Add Transaction.size method

* feat: Add Transaction rollback, add, and query methods

* feat: Add Transaction delete, and also Transaction upload test

* feat: Add Transaction update

* feat: Add Transaction get

* test: update transaction tests for upload and delete

* fix: improve error handling

* feat: dynamically import httpx by checking its existence with find_spec. Update tests to skip if httpx is not available

* test: put httpx import behind has_httpx condition

* test: fix test annotations

* fix: add conditional import for GraphDBClient

* test: ignore rdf4j and graphdb client.py for docstring tests

* feat: RDF4J store (#3316)

* fix: handle graph_name when it's a str

* feat: wip RDF4JStore

Implements:

- init/open
- close
- add
- addN
- contexts
- add_graph
- remove_graph
- __len__

* feat: RDF4J Store now supports handling namespaces and prefixes

* feat: RDF4J Store triples and quads querying

* feat: ensure no bnodes are used to cross document/query boundaries

* chore: formatting

* test: improve e2e test speed by reusing the same container and cleaning up the repo between each tests

* feat: add RDF4JStore remove

* feat: add RDF4JStore triples_choices tests

* feat: add RDF4JStore SPARQL query and update tests

* chore: fix mypy issues

* test: error handling on client fixture

* test: mark testcontainer tests and put test imports behind the has_httpx flag

* build: remove upper python bound, bump testcontainers, and revert back to stable v7 poetry.lock

* test: put testcontainer tests behind a flag for unsupported python versions

* test: install rdf4j extras for python 3.9 and above

* ci: skip testcontainer tests on non-linux runners

* chore: rename RDF4J's NamespaceManager class

---------

Co-authored-by: Nicholas Car <[email protected]>

Author: edmondchuc

.github/workflows/validate.yaml

@@ -12,7 +12,6 @@ env:
   POETRY_CACHE_DIR: ${{ github.workspace }}/.var/cache/pypoetry
   PIP_CACHE_DIR: ${{ github.workspace }}/.var/cache/pip
 
-
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
@@ -52,7 +51,7 @@ jobs:
             PREPARATION: "sudo apt-get install -y firejail"
             extensive-tests: true
             TOX_TEST_HARNESS: "firejail --net=none --"
-            TOX_PYTEST_EXTRA_ARGS: "-m 'not webtest'"
+            TOX_PYTEST_EXTRA_ARGS: "-m 'not (testcontainer or webtest)'"
     steps:
       - uses: actions/checkout@v4
       - name: Cache XDG_CACHE_HOME
@@ -84,6 +83,13 @@ jobs:
         shell: bash
         run: |
           ${{ matrix.PREPARATION }}
+      - name: Set testcontainer exclusion for non-Linux
+        if: ${{ matrix.os != 'ubuntu-latest' }}
+        shell: bash
+        run: |
+          if [ -z "${{ matrix.TOX_PYTEST_EXTRA_ARGS }}" ]; then
+            echo "TOX_PYTEST_EXTRA_ARGS=-m 'not testcontainer'" >> $GITHUB_ENV
+          fi
       - name: Run validation
         shell: bash
         run: |
@@ -97,7 +103,7 @@ jobs:
             gha:validate
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          TOX_PYTEST_EXTRA_ARGS: ${{ matrix.TOX_PYTEST_EXTRA_ARGS }}
+          TOX_PYTEST_EXTRA_ARGS: ${{ matrix.TOX_PYTEST_EXTRA_ARGS || env.TOX_PYTEST_EXTRA_ARGS }}
           TOX_TEST_HARNESS: ${{ matrix.TOX_TEST_HARNESS }}
           TOX_EXTRA_COMMAND: ${{ matrix.TOX_EXTRA_COMMAND }}
       - uses: actions/upload-artifact@v4

Taskfile.yml

@@ -378,3 +378,11 @@ tasks:
             sys.stderr.write(f"removing {path}\n")
             shutil.rmtree(path, ignore_errors=True)
         ' {{.RIMRAF_TARGET}}
+
+  test:rdf4j:
+    desc: Run fast tests against rdflib.contrib.rdf4j package
+    cmd: '{{.TEST_HARNESS}}{{.RUN_PREFIX}} pytest -m "not (testcontainer or webtest)" test/test_rdf4j'
+
+  test:rdf4j:all:
+    desc: Run all tests against rdflib.contrib.rdf4j package
+    cmd: '{{.TEST_HARNESS}}{{.RUN_PREFIX}} pytest test/test_rdf4j'

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
     - Container: apidocs/rdflib.container.md
     - Collection: apidocs/rdflib.collection.md
     - Paths: apidocs/rdflib.paths.md
+    - RDF4J: apidocs/rdflib.contrib.rdf4j.md
     - Util: apidocs/rdflib.util.md
     - Plugins:
       - Parsers: apidocs/rdflib.plugins.parsers.md

poetry.lock

@@ -49,6 +49,7 @@ networkx = {version = ">=2,<4", optional = true}
 html5rdf = {version = ">=1.2,<2", optional = true}
 lxml = {version = ">=4.3,<6.0", optional = true}
 orjson = {version = ">=3.9.14,<4", optional = true}
+httpx = {version = "^0.28.1", optional = true}
 
 [tool.poetry.group.dev.dependencies]
 black = "24.8.0"
@@ -63,6 +64,7 @@ coverage = {version = "^7.0.1", extras = ["toml"]}
 types-setuptools = ">=68.0.0.3,<72.0.0.0"
 setuptools = ">=68,<72"
 wheel = ">=0.42,<0.46"
+testcontainers = {version = "^4.13.2", python = ">=3.9.2"}
 
 [tool.poetry.group.docs.dependencies]
 typing-extensions = "^4.11.0"
@@ -85,6 +87,7 @@ html = ["html5rdf"]
 # lxml support is optional, it is used only for parsing XML-formatted SPARQL results
 lxml = ["lxml"]
 orjson = ["orjson"]
+rdf4j = ["httpx"]
 
 [build-system]
 requires = ["poetry-core>=1.4.0"]
@@ -207,6 +210,8 @@ addopts = [
     "--ignore=admin",
     "--ignore=devtools",
     "--ignore=rdflib/extras/external_graph_libs.py",
+    "--ignore=rdflib/contrib/graphdb/client.py",
+    "--ignore=rdflib/contrib/rdf4j/client.py",
     "--ignore-glob=docs/*.py",
     "--ignore-glob=site/*",
     "--strict-markers",
@@ -218,6 +223,7 @@ filterwarnings = [
     "ignore:Code. _pytestfixturefunction is not defined in namespace .*:UserWarning",
 ]
 markers = [
+    "testcontainer: mark a test that uses testcontainer",
     "webtest: mark a test as using the internet",
 ]
 # log_cli = true

pyproject.toml

@@ -0,0 +1,6 @@
+from rdflib.contrib.rdf4j import has_httpx
+
+if has_httpx:
+    from .client import GraphDBClient
+
+    __all__ = ["GraphDBClient"]

rdflib/contrib/__init__.py

@@ -0,0 +1,61 @@
+import httpx
+
+import rdflib.contrib.rdf4j
+from rdflib.contrib.rdf4j import RDF4JClient
+from rdflib.contrib.rdf4j.exceptions import (
+    RepositoryNotFoundError,
+    RepositoryNotHealthyError,
+)
+
+
+class Repository(rdflib.contrib.rdf4j.client.Repository):
+    """GraphDB Repository"""
+
+    def health(self, timeout: int = 5) -> bool:
+        """Repository health check.
+
+        Parameters:
+            timeout: A timeout parameter in seconds. If provided, the endpoint attempts
+                to retrieve the repository within this timeout. If not, the passive
+                check is performed.
+
+        Returns:
+            bool: True if the repository is healthy, otherwise an error is raised.
+
+        Raises:
+            RepositoryNotFoundError: If the repository is not found.
+            RepositoryNotHealthyError: If the repository is not healthy.
+            httpx.RequestError: On network/connection issues.
+            httpx.HTTPStatusError: Unhandled status code error.
+        """
+        try:
+            params = {"passive": str(timeout)}
+            response = self.http_client.get(
+                f"/repositories/{self.identifier}/health", params=params
+            )
+            response.raise_for_status()
+            return True
+        except httpx.HTTPStatusError as err:
+            if err.response.status_code == 404:
+                raise RepositoryNotFoundError(
+                    f"Repository {self._identifier} not found."
+                )
+            raise RepositoryNotHealthyError(
+                f"Repository {self._identifier} is not healthy. {err.response.status_code} - {err.response.text}"
+            )
+        except httpx.RequestError:
+            raise
+
+
+class RepositoryManager(rdflib.contrib.rdf4j.client.RepositoryManager):
+    """GraphDB Repository Manager"""
+
+    def get(self, repository_id: str) -> Repository:
+        _repo = super().get(repository_id)
+        return Repository(_repo.identifier, _repo.http_client)
+
+
+class GraphDBClient(RDF4JClient):
+    """GraphDB Client"""
+
+    # TODO: GraphDB specific API methods.

rdflib/contrib/graphdb/__init__.py

@@ -0,0 +1,10 @@
+from importlib.util import find_spec
+
+has_httpx = find_spec("httpx") is not None
+
+if has_httpx:
+    from .client import RDF4JClient
+
+    __all__ = ["RDF4JClient", "has_httpx"]
+else:
+    __all__ = ["has_httpx"]