fix(memory): gmail canonicalisation, signature unification, adaptive page retry

[yh928]

Summary

Three fixes to the Composio/Gmail sync and chunk-embedding layers, all surfaced by the same live investigation into why gmail-ingested memories were unretrievable.

• feat(sync/gmail) — Gmail sync stored to_string_pretty(&item.raw) as document content: a MIME tree with base64 part bodies, so the mail's words never reached the store and recall could not match them. Each message now routes through the existing email::canonicalise (headers as a small block, body through email_clean::clean_body), reusing parse_message_date and falling back to the epoch when undated so a message doesn't re-chunk on every sync. Adds GmailSyncPipeline::with_executor so the host can wrap the executor with its response reshape.
• fix(chunks) — the tree keyed per-model sidecars by {model}@{dims} while the namespace store used provider=…;model=…;dims=…. Per-signature reads are exact string matches, so one vector space was partitioned in two; a format change (not a model change) stranded live tree chunks. Writes now emit the canonical spelling (delegated to the same tinyagents formatter the namespace store uses) and reads match any spelling via signature_variants — nothing on disk is rewritten, legacy rows simply become readable again and count as coverage.
• fix(sync) — a page the provider refuses purely for size (Gmail 413 Upstream_PayloadTooLarge) is the one provider failure a smaller request can fix; the orchestrator now halves the source's page-size argument and retries down to one item, sticky for the run. Sources opt in via page_size_arg_key (max_results for Gmail); others are unchanged.

Tests

cargo test --features sync — 1298 unit + integration tests pass. New coverage: gmail canonicalisation (5), signature parsing/equivalence/variants + legacy-spelling read-back (18 in the chunks suite), too-large-page retry (4).

Notes

Consumed by an OpenHuman core PR (durable re-embed sweep + connector-aware recall) that depends on the new EmbeddingConfig.provider field and GmailSyncPipeline::with_executor.

Summary by CodeRabbit

• New Features

  • Added provider-aware embedding configuration and compatibility with legacy and canonical signature formats.
  • Gmail sync now stores cleaner canonical Markdown and supports provider-specific pagination.
  • Oversized sync requests automatically retry with smaller page sizes.

• Bug Fixes

  • Preserved access to existing chunk and summary embeddings after signature updates.
  • Improved Gmail content, recipient parsing, and timestamp handling.
  • Increased database resilience during transient cold-start I/O failures.
  • Allowed max_items for Composio and RSS sources while rejecting it for unsupported source types.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds canonical and legacy embedding-signature matching, provider-aware page-size retries, canonical Gmail Markdown rendering, transient database bootstrap retries, and Composio source validation.

Changes

Embedding signature compatibility

Layer / File(s)	Summary
Signature contract and configuration src/memory/config.rs, src/memory/chunks/mod.rs, src/memory/chunks/signature.rs, src/memory/chunks/signature_tests.rs	Embedding signatures support canonical and legacy spellings, equivalence checks, variant expansion, SQL IN clauses, and provider defaults.
Variant-aware embedding storage reads src/memory/chunks/embeddings.rs, src/memory/tree/store/summaries.rs, src/memory/chunks/store_embed_tests.rs	Chunk and summary queries, batch reads, and uncovered-work detection match recognized signature variants.

Gmail synchronization resilience

Layer / File(s)	Summary
Provider-aware page-size retry src/memory/sync/composio/page_size.rs, src/memory/sync/composio/orchestrator.rs, src/memory/sync/composio/*tests.rs	Sources can declare page-size parameters. Payload-too-large responses retry with smaller requests.
Gmail executor and canonical email adapter src/memory/sync/composio/gmail.rs, src/memory/sync/composio/gmail_tests.rs	Gmail uses an ActionExecutor, declares max_results, and renders messages as canonical Markdown.

Storage bootstrap and source validation

Layer / File(s)	Summary
Transient database bootstrap retry src/memory/chunks/recovery.rs, src/memory/chunks/connection.rs, src/memory/chunks/recovery_tests.rs	Cold-start classification includes SQLITE_IOERR_FSTAT. Initialization retries matching failures with bounded exponential backoff.
Composio max-items validation src/memory/sources/types.rs, src/memory/sources/types_tests.rs	max_items is accepted for Composio and RSS sources and rejected for other source kinds.

Estimated code review effort: 4 (Complex) | ~75 minutes

Possibly related PRs

• tinyhumansai/tinycortex#66: Adds earlier signature-specific embedding APIs extended by this PR.
• tinyhumansai/tinycortex#74: Introduces signature-scoped embedding reads extended with variant matching.
• tinyhumansai/tinycortex#125: Modifies related re-embedding coverage logic.

Sequence Diagram(s)

sequenceDiagram
  participant SyncOrchestrator
  participant GmailSyncPipeline
  participant ActionExecutor
  participant MemoryStore
  SyncOrchestrator->>GmailSyncPipeline: request page with max_results
  GmailSyncPipeline->>ActionExecutor: execute Gmail request
  ActionExecutor-->>GmailSyncPipeline: payload-too-large response
  GmailSyncPipeline->>ActionExecutor: retry with reduced max_results
  ActionExecutor-->>SyncOrchestrator: successful message page
  SyncOrchestrator->>MemoryStore: store canonical Markdown document

Loading

Suggested reviewers: senamakel

Poem

I’m a rabbit with signatures neat,
Matching old spellings beneath every seat.
Gmail pages shrink when they grow wide,
Canonical Markdown hops inside.
Cold starts retry, then settle just right.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the three primary changes: Gmail canonicalisation, embedding signature unification, and adaptive page retries.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR fixes three independent but related bugs uncovered during a Gmail sync investigation: raw MIME JSON was being stored instead of readable message text, embedding signature spellings were inconsistent between the tree sidecar and namespace store (silently partitioning one vector space), and a single oversized Gmail page was permanently blocking sync runs. All three fixes are backward-compatible and include regression tests.

• Gmail canonicalisation (gmail.rs): replaces to_string_pretty(&item.raw) storage with a proper call to email::canonicalise, routing through email_clean::clean_body so the actual message words reach the chunk store.
• Signature unification (signature.rs, embeddings.rs, summaries.rs): introduces a new signature module that canonicalises all writes to provider=…;model=…;dims=… and expands reads via signature_variants to match the legacy {model}@{dims} spelling — making stranded tree chunks visible again without rewriting any on-disk rows.
• Adaptive page retry (page_size.rs, orchestrator.rs): when the provider returns a 413/Upstream_PayloadTooLarge error, the orchestrator halves the source's page_size_arg_key argument and retries, down to a floor of 1, and makes the accepted size sticky for the rest of the run.

Confidence Score: 5/5

All three changes are backward-compatible and safe to merge.

The Gmail fix replaces a clearly broken storage path; the signature unification is non-destructive (read-only on disk, legacy rows become visible without rewriting); the page retry is opt-in per source and sets the sticky override only on a confirmed successful response. 1298 tests including 27 new ones cover every changed path.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
src/memory/chunks/signature.rs	New module unifying two signature spellings. parse_signature handles both canonical and legacy forms. signature_variants expands for IN-clause reads. Thorough test coverage.
src/memory/chunks/embeddings.rs	All per-signature reads now expand via signature_variants and match with an IN clause. tree_active_signature now emits the canonical provider-inclusive form.
src/memory/sync/composio/gmail.rs	Core fix: canonical_markdown replaces raw JSON storage with a properly canonicalised Markdown document. Deterministic sent-at timestamp prevents re-chunking on undated messages.
src/memory/sync/composio/orchestrator.rs	Adaptive retry loop added inside the per-page loop. page_size_override is correctly set only after a page succeeds.
src/memory/sync/composio/page_size.rs	New module with is_payload_too_large, shrink_page_size (halves with floor of 1), and apply_page_size. Word-boundary check prevents false-positives from numbers containing '413'.
src/memory/chunks/connection.rs	Added 3-attempt exponential-backoff retry for transient cold-start I/O errors.
src/memory/chunks/recovery.rs	Added SQLITE_IOERR_FSTAT (1802) to the transient cold-start codes with accompanying test.
src/memory/config.rs	Adds provider: String field to EmbeddingConfig with #[serde(default)]. Backward-compatible.
src/memory/sources/types.rs	Widens the max_items validation guard from RSS-only to RSS + Composio, fixing a UI-to-store desync bug.
src/memory/tree/store/summaries.rs	Summary embedding reads now use signature_variants + IN-clause, giving legacy-spelled summary vectors the same visibility fix as chunk vectors.

_{Reviews (3): Last reviewed commit: "fix(sync): record the accepted page size..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

src/memory/chunks/embeddings.rs (1)

456-507: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

Split embeddings query helpers before the 500-line cap.

src/memory/chunks/embeddings.rs is now 518 lines, so extract the signature/embedding query helpers into a focused sibling module such as src/memory/chunks/embeddings_query.rs.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/memory/chunks/embeddings.rs` around lines 456 - 507, Extract the
signature/embedding query helpers, including
get_chunk_embeddings_for_signature_batch and its supporting query logic, from
embeddings.rs into a focused sibling module such as embeddings_query.rs. Update
module declarations and imports so existing callers retain the same behavior and
public API, and keep embeddings.rs below the 500-line limit.

Source: Coding guidelines

src/memory/sync/composio/gmail.rs (1)

280-296: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cc recipients are always dropped from the canonical email.

cc is hardcoded to Vec::new() (Line 288) even though email::canonicalise renders a Cc: line when present. If the upstream Gmail response reshaper surfaces Cc data on the message payload, it's silently lost here, meaning content someone was Cc'd on won't be searchable via that header. Worth confirming whether the reshaped envelope carries Cc data at all, and if so, extracting it the same way to is handled.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/memory/sync/composio/gmail.rs` around lines 280 - 296, The email_thread
function discards Cc recipients by always assigning an empty vector. Confirm the
reshaped Gmail message payload’s Cc field and extract it using the same
recipient parsing approach as message_recipients for to, preserving the
populated Cc values in EmailMessage.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/memory/sync/composio/orchestrator.rs`:
- Around line 497-545: Split the page-size retry helpers MIN_PAGE_SIZE,
is_payload_too_large, apply_page_size, and shrink_page_size out of
orchestrator.rs into a focused sibling module such as page_size.rs, re-exporting
or importing them for existing callers. Move their tests into a dedicated
page_size_tests.rs module while preserving current behavior and keeping
orchestrator.rs below the 500-line limit.
- Around line 508-517: Update is_payload_too_large so the 413 detection only
matches status-code-like occurrences rather than any unanchored “413” substring.
Preserve the existing payloadtoolarge, payload_too_large, and too-large
payload/response checks while preventing unrelated identifiers, amounts, or
timestamps from triggering retries.

---

Nitpick comments:
In `@src/memory/chunks/embeddings.rs`:
- Around line 456-507: Extract the signature/embedding query helpers, including
get_chunk_embeddings_for_signature_batch and its supporting query logic, from
embeddings.rs into a focused sibling module such as embeddings_query.rs. Update
module declarations and imports so existing callers retain the same behavior and
public API, and keep embeddings.rs below the 500-line limit.

In `@src/memory/sync/composio/gmail.rs`:
- Around line 280-296: The email_thread function discards Cc recipients by
always assigning an empty vector. Confirm the reshaped Gmail message payload’s
Cc field and extract it using the same recipient parsing approach as
message_recipients for to, preserving the populated Cc values in EmailMessage.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5ba9d842-8a29-4ad2-8ac1-c0ead2d8e11b

📥 Commits

Reviewing files that changed from the base of the PR and between e5753bb and 35e81c1.

📒 Files selected for processing (11)

• src/memory/chunks/embeddings.rs
• src/memory/chunks/mod.rs
• src/memory/chunks/signature.rs
• src/memory/chunks/signature_tests.rs
• src/memory/chunks/store_embed_tests.rs
• src/memory/config.rs
• src/memory/sync/composio/gmail.rs
• src/memory/sync/composio/gmail_tests.rs
• src/memory/sync/composio/orchestrator.rs
• src/memory/sync/composio/orchestrator_tests.rs
• src/memory/tree/store/summaries.rs

[yh928]

Pushed 0d531d8 — all three findings applied, plus a build break I hit while verifying them.

page_size_override set to the candidate, not the confirmed size (greptile) — fixed. You were right that the final value was correct only by coincidence of ordering. The shrink now records into a local attempted_page_size, and the run's sticky override is assigned where the response succeeds. With 25 → 12 → 6 the override is written once, with 6, instead of being left holding a rejected size at every intermediate step.

Bare 413 substring (CodeRabbit) — fixed. mentions_status_413 requires the digits to stand alone (no adjacent digit on either side), so a message id, an amount, or a timestamp carrying 413 no longer costs a shrink-and-retry cycle before the real error surfaces. Covered by a_number_that_merely_contains_413_is_not_a_status_code, which asserts both directions (HTTP 413, (413), returned 413. match; 4130, 1413, 94137, 1782891413 do not).

500-line cap (CodeRabbit) — fixed. MIN_PAGE_SIZE, is_payload_too_large, apply_page_size, shrink_page_size moved to page_size.rs with page_size_tests.rs; the existing helper tests moved with them. orchestrator.rs is back under the cap.

One thing the review did not catch, found while verifying: the default-feature build was broken. The chunk store's cold-start retry (connection.rs) calls tracing::debug! unconditionally, but tracing is optional and only arrives with sync — so cargo build --lib with default features failed to compile. CI stayed green because it builds with the feature on. The call is now #[cfg(feature = "sync")]-gated, matching how direct_ingest.rs does it. Also fixed one cloned_ref_to_slice_refs lint in a test file this PR touches.

Verification: 1301 lib tests green with --features sync, the default-feature build compiles, and clippy is clean in both configurations.

[coderabbitai]

♻️ Duplicate comments (1)

src/memory/sync/composio/orchestrator.rs (1)

512-515: 🛠️ Refactor suggestion | 🟠 Major

Continue the split so orchestrator.rs stays below 500 lines.

The page-size extraction moved the helper logic out of orchestrator.rs, but the file still reaches Line 515. Extract another cohesive orchestration block before merge.

As per path instructions, “Avoid letting any source file grow beyond 500 lines; split behavior into focused modules before that point.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/memory/sync/composio/orchestrator.rs` around lines 512 - 515, Continue
refactoring orchestrator.rs so it remains under 500 lines by extracting another
cohesive orchestration block into a focused module, while preserving the
existing behavior and module interfaces. Update the relevant declarations and
imports, including the cfg(test) tests module wiring, so the extracted logic
remains accessible and orchestrator.rs contains only its core orchestration
responsibilities.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/memory/sync/composio/orchestrator.rs`:
- Around line 512-515: Continue refactoring orchestrator.rs so it remains under
500 lines by extracting another cohesive orchestration block into a focused
module, while preserving the existing behavior and module interfaces. Update the
relevant declarations and imports, including the cfg(test) tests module wiring,
so the extracted logic remains accessible and orchestrator.rs contains only its
core orchestration responsibilities.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7f417b00-3eab-4a1c-aa1d-bbe168b986a2

📥 Commits

Reviewing files that changed from the base of the PR and between 5801e80 and 0d531d8.

📒 Files selected for processing (7)

• src/memory/chunks/connection.rs
• src/memory/chunks/store_embed_tests.rs
• src/memory/sync/composio/mod.rs
• src/memory/sync/composio/orchestrator.rs
• src/memory/sync/composio/orchestrator_tests.rs
• src/memory/sync/composio/page_size.rs
• src/memory/sync/composio/page_size_tests.rs

💤 Files with no reviewable changes (1)

• src/memory/sync/composio/orchestrator_tests.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• src/memory/chunks/store_embed_tests.rs
• src/memory/chunks/connection.rs