Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 109 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@ below are generated from those pages, so this README and the pages never drift.
<!-- GENERATED:CONNECTORS -->
<!-- Generated from docs/connectors/*.md by scripts/sync_readme.py — do not edit by hand. -->

### Memory & RAG

<details>
<summary><strong>Supermemory</strong> — documents → server-side embedding, idempotent on the canonical id</summary>

Expand Down Expand Up @@ -215,35 +217,87 @@ Decision: [`rac/decisions/`](rac/decisions) — ADR-004 (Mem0 backend, resync id
</details>

<details>
<summary><strong>Zep</strong> — documents → a Zep knowledge graph; idempotent by graph resync</summary>
<summary><strong>Qdrant</strong> — documents → external embedding → a Qdrant collection; idempotent on the canonical id</summary>

A one-way, outbound push of the `rac export --documents` stream into
[Zep Cloud](https://getzep.com). Same stream and flags as the other documents
backends, a different subcommand:
[Qdrant](https://qdrant.tech), the open-source vector database.

Qdrant stores vectors but does **not** produce them (unlike Supermemory/Mem0/Zep,
which embed server-side). So this connector embeds each record's text through a
**configured external embedding service** — any OpenAI-compatible `/embeddings`
endpoint, with a [LiteLLM](https://litellm.ai) gateway the reference deployment —
then upserts the vector. The model and credentials live in that endpoint, never
in RAC (the engine stays AI-optional, rac-core ADR-002/ADR-066); see
[ADR-009](rac/decisions/adr-009-vector-store-connectors-embed-externally.md).

```bash
pip install 'rac-connectors[zep]'
export ZEP_API_KEY=z_...
pip install 'rac-connectors[qdrant]'
export QDRANT_URL=http://localhost:6333 # and QDRANT_API_KEY if your server needs auth
export RAC_EMBED_BASE_URL=https://your-litellm/v1 # OpenAI-compatible /embeddings endpoint
export RAC_EMBED_MODEL=text-embedding-3-small # whatever your gateway routes
export RAC_EMBED_API_KEY=sk-... # if the endpoint requires auth

rac export rac/ --documents | rac-connect qdrant # embed + upsert every record
rac export rac/ --documents | rac-connect qdrant --dry-run # preview, no embed, no API call
rac-connect qdrant --input corpus.jsonl # read a file, not stdin
```

Each record maps to one Qdrant point:

rac export rac/ --documents | rac-connect zep # upsert every record
rac export rac/ --documents | rac-connect zep --dry-run # preview, no API call
rac-connect zep --input corpus.jsonl # read a file, not stdin
```
record → upsert(point_id=uuid5(canonical id),
vector=embed(text),
payload={rac_id, type, status, title, text, …metadata})
```

- **A corpus maps to a Zep graph.** A `source` becomes a Zep `graph_id`; each
record is added as a `type="text"` episode carrying the canonical `rac_id`,
`type`, `status`, and `title` in metadata.
- **Idempotent by graph resync.** Zep has no per-record upsert key, so each push
deletes and recreates the corpus graph, then re-adds — re-running never
duplicates.
- **No embeddings here.** Zep derives its knowledge graph and embeds; the
connector only ships text + metadata. Zep's copy is an associative index, not a
citation — authoritative text is always re-fetched from Lore.
- **Auth via `ZEP_API_KEY`** — never hard-coded.
| Flag | Meaning |
|---|---|
| `--dry-run` | Print what would be sent; embed nothing and call no API. |
| `--input`, `-i` | Read JSONL from a file (default: stdin; `-` also means stdin). |
| `--strict` | Fail on a malformed line instead of skipping it. |
| `--verbose`, `-v` | Print per-record actions on a live push too. |

Decision: [`rac/decisions/`](rac/decisions) — ADR-005 (Zep backend, graph-resync idempotency).
- **Idempotent on the canonical `id`.** The point id is `uuid5(id)`, so a re-push
upserts in place rather than duplicating.
- **One collection per corpus `source`** (falling back to `lore`); the collection
is created on first use with the embedder's vector dimension and cosine distance.
- **Embeddings live in the external endpoint**, not here. **Pin the embedding
model** — the vectors, and the collection's dimension, are tied to it; changing
the model means re-embedding the corpus.
- **Auth via `QDRANT_URL` / `QDRANT_API_KEY`** and the `RAC_EMBED_*` variables —
never hard-coded.

**Full page:** [`docs/connectors/zep.md`](docs/connectors/zep.md)
### Live smoke test

The connector is wired and unit-tested against fakes, but the live path (a real
Qdrant plus a real embeddings endpoint) is unproven until someone runs it — this
page is `drafted (live run pending)`. To validate end to end:

1. **Start Qdrant:** `docker run -p 6333:6333 qdrant/qdrant`.
2. **Pick an embeddings endpoint** — a LiteLLM (or any OpenAI-compatible)
`/embeddings` gateway; note the model and its vector dimension.
3. **Configure the environment:**

```bash
export QDRANT_URL=http://localhost:6333 # + QDRANT_API_KEY if needed
export RAC_EMBED_BASE_URL=https://your-litellm/v1
export RAC_EMBED_MODEL=text-embedding-3-small
export RAC_EMBED_API_KEY=sk-... # if the endpoint requires it
```

4. **Dry-run first** (no embed, no calls) — confirms records and collections:
`rac export rac/ --documents | rac-connect qdrant --dry-run`.
5. **Live push:** `rac export rac/ --documents | rac-connect qdrant`.
6. **Verify in Qdrant:** the collection (named after the corpus `source`,
default `lore`) exists with the model's vector size; the point count equals the
artifact count; a point's payload carries `rac_id`, `type`, `status`, `title`,
and `text`.
7. **Re-run the push** and confirm the point count is unchanged — the upsert is
idempotent on `uuid5(rac_id)`.

Then flip this page's `status` to `shipped`.

**Full page:** [`docs/connectors/qdrant.md`](docs/connectors/qdrant.md)

</details>

Expand Down Expand Up @@ -281,6 +335,41 @@ Decision: [`rac/decisions/`](rac/decisions) — ADR-006 (Letta backend, archive-

</details>

### Knowledge graph

<details>
<summary><strong>Zep</strong> — documents → a Zep knowledge graph; idempotent by graph resync</summary>

A one-way, outbound push of the `rac export --documents` stream into
[Zep Cloud](https://getzep.com). Same stream and flags as the other documents
backends, a different subcommand:

```bash
pip install 'rac-connectors[zep]'
export ZEP_API_KEY=z_...

rac export rac/ --documents | rac-connect zep # upsert every record
rac export rac/ --documents | rac-connect zep --dry-run # preview, no API call
rac-connect zep --input corpus.jsonl # read a file, not stdin
```

- **A corpus maps to a Zep graph.** A `source` becomes a Zep `graph_id`; each
record is added as a `type="text"` episode carrying the canonical `rac_id`,
`type`, `status`, and `title` in metadata.
- **Idempotent by graph resync.** Zep has no per-record upsert key, so each push
deletes and recreates the corpus graph, then re-adds — re-running never
duplicates.
- **No embeddings here.** Zep derives its knowledge graph and embeds; the
connector only ships text + metadata. Zep's copy is an associative index, not a
citation — authoritative text is always re-fetched from Lore.
- **Auth via `ZEP_API_KEY`** — never hard-coded.

Decision: [`rac/decisions/`](rac/decisions) — ADR-005 (Zep backend, graph-resync idempotency).

**Full page:** [`docs/connectors/zep.md`](docs/connectors/zep.md)

</details>

<details>
<summary><strong>Cognee</strong> — documents → a Cognee knowledge graph; content-hash idempotent</summary>

Expand Down
1 change: 1 addition & 0 deletions docs/connectors/cognee.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Cognee
tagline: documents → a Cognee knowledge graph; content-hash idempotent
category: Knowledge graph
extra: cognee
order: 50
status: drafted (live run pending)
Expand Down
1 change: 1 addition & 0 deletions docs/connectors/letta.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Letta
tagline: documents → Letta archives (cloud or self-hosted); idempotent by archive resync
category: Memory & RAG
extra: letta
order: 40
status: drafted (live run pending)
Expand Down
1 change: 1 addition & 0 deletions docs/connectors/mem0.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Mem0
tagline: documents → server-side embedding; idempotent by container resync
category: Memory & RAG
extra: mem0
order: 20
status: drafted (live run pending)
Expand Down
1 change: 1 addition & 0 deletions docs/connectors/neo4j.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Neo4j
tagline: graph → typed nodes & edges via Cypher MERGE; idempotent on the canonical id
category: Knowledge graph
extra: neo4j
order: 60
status: drafted (live run pending)
Expand Down
87 changes: 87 additions & 0 deletions docs/connectors/qdrant.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
<!-- rac-connector
name: Qdrant
tagline: documents → external embedding → a Qdrant collection; idempotent on the canonical id
category: Memory & RAG
extra: qdrant
order: 25
status: drafted (live run pending)
-->
# Qdrant

A one-way, outbound push of the `rac export --documents` stream into
[Qdrant](https://qdrant.tech), the open-source vector database.

Qdrant stores vectors but does **not** produce them (unlike Supermemory/Mem0/Zep,
which embed server-side). So this connector embeds each record's text through a
**configured external embedding service** — any OpenAI-compatible `/embeddings`
endpoint, with a [LiteLLM](https://litellm.ai) gateway the reference deployment —
then upserts the vector. The model and credentials live in that endpoint, never
in RAC (the engine stays AI-optional, rac-core ADR-002/ADR-066); see
[ADR-009](../../rac/decisions/adr-009-vector-store-connectors-embed-externally.md).

```bash
pip install 'rac-connectors[qdrant]'
export QDRANT_URL=http://localhost:6333 # and QDRANT_API_KEY if your server needs auth
export RAC_EMBED_BASE_URL=https://your-litellm/v1 # OpenAI-compatible /embeddings endpoint
export RAC_EMBED_MODEL=text-embedding-3-small # whatever your gateway routes
export RAC_EMBED_API_KEY=sk-... # if the endpoint requires auth

rac export rac/ --documents | rac-connect qdrant # embed + upsert every record
rac export rac/ --documents | rac-connect qdrant --dry-run # preview, no embed, no API call
rac-connect qdrant --input corpus.jsonl # read a file, not stdin
```

Each record maps to one Qdrant point:

```
record → upsert(point_id=uuid5(canonical id),
vector=embed(text),
payload={rac_id, type, status, title, text, …metadata})
```

| Flag | Meaning |
|---|---|
| `--dry-run` | Print what would be sent; embed nothing and call no API. |
| `--input`, `-i` | Read JSONL from a file (default: stdin; `-` also means stdin). |
| `--strict` | Fail on a malformed line instead of skipping it. |
| `--verbose`, `-v` | Print per-record actions on a live push too. |

- **Idempotent on the canonical `id`.** The point id is `uuid5(id)`, so a re-push
upserts in place rather than duplicating.
- **One collection per corpus `source`** (falling back to `lore`); the collection
is created on first use with the embedder's vector dimension and cosine distance.
- **Embeddings live in the external endpoint**, not here. **Pin the embedding
model** — the vectors, and the collection's dimension, are tied to it; changing
the model means re-embedding the corpus.
- **Auth via `QDRANT_URL` / `QDRANT_API_KEY`** and the `RAC_EMBED_*` variables —
never hard-coded.

### Live smoke test

The connector is wired and unit-tested against fakes, but the live path (a real
Qdrant plus a real embeddings endpoint) is unproven until someone runs it — this
page is `drafted (live run pending)`. To validate end to end:

1. **Start Qdrant:** `docker run -p 6333:6333 qdrant/qdrant`.
2. **Pick an embeddings endpoint** — a LiteLLM (or any OpenAI-compatible)
`/embeddings` gateway; note the model and its vector dimension.
3. **Configure the environment:**

```bash
export QDRANT_URL=http://localhost:6333 # + QDRANT_API_KEY if needed
export RAC_EMBED_BASE_URL=https://your-litellm/v1
export RAC_EMBED_MODEL=text-embedding-3-small
export RAC_EMBED_API_KEY=sk-... # if the endpoint requires it
```

4. **Dry-run first** (no embed, no calls) — confirms records and collections:
`rac export rac/ --documents | rac-connect qdrant --dry-run`.
5. **Live push:** `rac export rac/ --documents | rac-connect qdrant`.
6. **Verify in Qdrant:** the collection (named after the corpus `source`,
default `lore`) exists with the model's vector size; the point count equals the
artifact count; a point's payload carries `rac_id`, `type`, `status`, `title`,
and `text`.
7. **Re-run the push** and confirm the point count is unchanged — the upsert is
idempotent on `uuid5(rac_id)`.

Then flip this page's `status` to `shipped`.
1 change: 1 addition & 0 deletions docs/connectors/supermemory.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Supermemory
tagline: documents → server-side embedding, idempotent on the canonical id
category: Memory & RAG
extra: supermemory
order: 10
status: shipped
Expand Down
1 change: 1 addition & 0 deletions docs/connectors/zep.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!-- rac-connector
name: Zep
tagline: documents → a Zep knowledge graph; idempotent by graph resync
category: Knowledge graph
extra: zep
order: 30
status: drafted (live run pending)
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,10 @@ letta = ["letta-client>=1,<2"]
# The live Cognee push needs the cognee pipeline; the core install and the
# fake-driven test-suite stay dependency-free. Pinned to the verified v1 major.
cognee = ["cognee>=1,<2"]
# The live Qdrant push needs the official client; the core install and the
# fake-driven test-suite stay dependency-free. Embeddings come from an external
# endpoint over stdlib HTTP (no SDK), so only qdrant-client is required here.
qdrant = ["qdrant-client>=1.7,<2"]
dev = ["pytest>=7.0", "ruff", "mypy"]

[project.scripts]
Expand Down Expand Up @@ -91,5 +95,6 @@ module = [
"zep_cloud.*",
"letta_client.*",
"cognee.*",
"qdrant_client.*",
]
ignore_missing_imports = true
Loading