feat: ingest synced skill documents into knowledge graph

[senamakel]

Skill Submission

Skill name: notion, gmail
Type: [x] Coded (skill.ts)

Description

Wire Notion and Gmail skill sync flows into the memory ingestion pipeline so that synced documents are automatically processed into the knowledge graph via memory.insert().

Problem: Skills sync documents into their local SQLite DB but the content never reaches the Rust ingestion pipeline (GLiNER entity/relation extraction → knowledge graph). The Notion skill had a backend.submitData() call that was commented out, and Gmail had no submission flow at all — despite both having all the DB tracking infrastructure (backend_submitted flag, getUnsubmitted*, markSubmitted helpers) already in place.

Solution: Replace the old backend.submitData() batching approach with per-document memory.insert() calls, which route through the existing Rust pipeline: put_doc() → IngestionQueue → GLiNER extraction → graph storage.

Notion (src/core/notion/sync.ts)

• Replaced submitNewData() (commented-out backend.submitData + chunking/batching) with ingestNewDocuments()
• Calls memory.insert() per unsubmitted page and database row
• Uses documentId: "notion-page-{id}" / "notion-row-{id}" for dedup/upsert
• Sets sourceType: 'doc' with page metadata (url, timestamps, parent info)
• Reuses existing backend_submitted tracking

Gmail (src/core/gmail/sync.ts)

• Added ingestNewEmails() function
• Marks sensitive emails as submitted first (never ingested)
• Calls memory.insert() per unsubmitted email with sourceType: 'email'
• Uses documentId: "gmail-email-{id}" for dedup
• Wired into both performInitialSync() and onSync() (incremental) flows

Checklist

• No hardcoded API keys, tokens, or secrets
• No eval(), Function(), or dynamic code execution
• yarn typecheck passes
• yarn build passes

Testing

1. Verified yarn typecheck passes with no errors
2. Verified yarn build completes successfully (all 3 skills bundle)
3. Reviewed that memory.insert() bridge is fire-and-forget async on the Rust side — no blocking risk during sync

Category

• DeFi
• Trading
• Research
• Community
• NFT
• Security
• Other: Core skill infrastructure — ingestion pipeline integration

🤖 Generated with Claude Code

Summary by CodeRabbit

• Refactor
  • Gmail sync now ingests newly synced emails into the knowledge graph automatically.
  • Notion ingestion simplified to directly insert unsubmitted pages/rows into the knowledge graph.
• Chores
  • Updated subproject reference to a new commit.
• Tests
  • Adjusted server-ping tests to use a local server URL.
• Bug Fixes
  • Ensured ping-now awaits completion of the ping routine.
• Documentation
  • Updated developer-facing live-test usage paths and messages.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: be233667-4343-407a-a9f6-96f5108c010f

📥 Commits

Reviewing files that changed from the base of the PR and between 31c0314 and 62f6ab2.

📒 Files selected for processing (5)

• openhuman
• src/core/gmail/live-test.ts
• src/core/notion/live-test.ts
• src/core/server-ping/__tests__/test-server-ping.ts
• src/core/server-ping/tools/ping-now.ts

✅ Files skipped from review due to trivial changes (2)

• openhuman
• src/core/notion/live-test.ts

---

📝 Walkthrough

Walkthrough

Added direct ingestion flows to Gmail and Notion syncs that import newly synced items into the knowledge graph; also updated live-test import paths/usages, adjusted server-ping tests to use localhost, awaited ping-now’s doPing, and advanced the openhuman submodule pointer.

Changes

Cohort / File(s)	Summary
Gmail Sync Ingestion src/core/gmail/sync.ts	Added ingestNewEmails() called after syncs. Fetches up to INGEST_QUERY_LIMIT unsubmitted emails, marks sensitive emails as submitted, derives content from `body_text
Notion Sync Ingestion src/core/notion/sync.ts	Removed Phase 5 chunking/submission pipeline; added ingestNewDocuments() to synchronously fetch up to INGEST_QUERY_LIMIT unsubmitted pages/rows and insert non-empty content_text/properties_text into memory per-item, tracking and marking submitted IDs; per-item errors do not abort the loop.
Live Test Import/Usage Updates src/core/gmail/live-test.ts, src/core/notion/live-test.ts	Adjusted header usage messages and changed relative import path for dev/test-harness from ../../../../dev/test-harness to ../../../dev/test-harness; simplified JWT missing-env message in Gmail live-test.
Server-ping Tests src/core/server-ping/__tests__/test-server-ping.ts	Updated test inputs to use http://localhost:7799 for serverUrl in setup and tools tests (replacing external URLs).
Ping-now Tool src/core/server-ping/tools/ping-now.ts	Changed pingNowTool.execute() to await globalThis.doPing() (treat as async Promise<void>) before continuing to fetch skill state and logs.
Submodule Pointer openhuman	Updated openhuman subproject pointer from one commit to another (commit hash advanced).

Sequence Diagram(s)

sequenceDiagram
    participant Sync as Sync Module
    participant DB as Database
    participant Memory as Knowledge Graph

    Sync->>DB: getUnsubmittedItems(INGEST_QUERY_LIMIT)
    DB-->>Sync: return items
    Sync->>DB: markSensitiveAsSubmitted(ids) (Gmail)
    DB-->>Sync: ack

    loop per item
        Sync->>Sync: extract content (body_text || snippet / content_text)
        alt content exists
            Sync->>Memory: memory.insert(title, content, metadata, timestamps)
            Memory-->>Sync: inserted
        else no content
            Sync-->>Sync: mark as submitted (skip insert)
        end
        Sync->>Sync: collect processed ID
    end

    Sync->>DB: markItemsSubmitted(all_processed_ids)
    DB-->>Sync: ack

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I hop and nibble lines of code,

Into the graph new data flowed.
Gmail, Notion, now intertwined,
Small hops make a knowledge mind.
Bravo—more carrots for the node! 🥕

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: ingest synced skill documents into knowledge graph' directly and accurately summarizes the main change: wiring Gmail and Notion sync flows into the memory ingestion pipeline to process synced documents into the knowledge graph.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}