Repository navigation
feat: Evernote v10 crawler + cross-cutting infra (sortKey, attachment OCR, subdirConfig) - #125
Merged
Merged
Conversation
Imports notes, notebooks, tags, and attachments from a local Evernote v10
install. Decodes the binary RTE blob via the yjs library so notes render
in the same order Evernote shows them — text, images, and PDFs interleave
exactly as authored. Notebook membership is structured metadata (not a
tag), notebooks are pickable from the desktop "Add Collection" dialog
with note counts and aggregate sizes, and notes sort newest-first via a
new `EntityData.sortKey` field included in the entity hash so manifest /
sync / clone all stay consistent.
Cross-cutting changes outside the crawler:
- `EntityData.assets[].text` + new FTS `attachment_text` column for OCR.
bm25 weights now title=10 / tags=5 / body=1 / attachment=0.25, and
legacy collections without the column keep working via a column-presence
check rather than a destructive drop+recreate.
- `EntityData.sortKey` (default = title) propagates through hash, sync,
publish to D1, and the file-tree sort in both serve.ts and the worker.
- `FolderConfig.subdirConfig` lets a theme push a default config into
every direct subfolder (used so each Evernote notebook folder inherits
`sort: DESC`).
- Default attachment size cap raised from 10 MB to 100 MB; PDFs >10 MB
were silently dropped before, leaving broken `` references.
- UI stays responsive during sync — markdown pane and file tree keep
showing previous content while new data loads (small inline indicator
instead of a blocking "Loading…" placeholder).
- `<img>` override renders PDF refs as `<iframe key={url}>` so navigating
between two PDF notes reliably re-loads the embed.
- Electron `webPreferences.plugins: true` so the desktop app's PDF viewer
works inline.
- Markdown-special chars (`*`, `(`, `)`, `!`, `'`) percent-encoded in
attachment URLs so filenames like `11292025_*000779*…` don't break
markdown link parsing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… AGENTS.md Closes the publish-side gap left by #121 / #125: the local SearchIndexer gained an `attachment_text` column with a non-destructive migration, but `publish.ts` was still creating the old 6-column D1 FTS so OCR'd text disappeared on the published worker. Three coordinated changes: - `publish.ts`: detect existing FTS schema with PRAGMA table_info; if `attachment_text` is missing on republish, drop + recreate the FTS table and force-include every entity in the push set so all FTS rows re-INSERT (FTS5 doesn't support ALTER TABLE ADD COLUMN). Fresh collections get the new schema directly via CREATE VIRTUAL TABLE IF NOT EXISTS. INSERT prefix and value rows now carry `attachment_text` joined from `data.assets[].text` — same shape SyncEngine builds for the local FTS. - `wrangler-api.ts`: new `queryD1Rows` helper exposing the result body from D1 query API so the publish path can read PRAGMA output. - `worker/db/search.ts`: bm25 expression bumped to 7 weights matching the local indexer (title 10 / tags 5 / body 1 / attachment 0.25). Tags weight bumped from 2 → 5 along the way, also matching the local change in #121 that the worker had silently missed. AGENTS.md gains a "Local ↔ worker parity rule" section under Publishing to Cloudflare. It captures the principle that schema, query, ranking, manifest, and rendered-output changes must land in the same PR for both the local and worker code paths, with a per-category checklist of layers to update. Documents that `CREATE TABLE IF NOT EXISTS` is idempotent and FTS5 needs explicit drop+recreate to evolve. Mandates that test plans for these changes include a publish to a scratch worker. Closes #126 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the inline `CREATE TABLE IF NOT EXISTS` / `ALTER TABLE` / PRAGMA-checks scattered across `client.ts`, `indexer.ts`, and `publish.ts` with a single migrations framework. Each schema change is now an append-only numbered migration with a matching entry in `SCHEMA.md`. The runner records the latest applied id in the `metadata` table under `schema.version`; subsequent calls do one indexed SELECT and return. Two parallel migration lists since the local SQLite schema and the published Cloudflare D1 schema diverge in places (FTS `collection_name`, backfill loops, `id AUTOINCREMENT`). Both live under `packages/core/src/db/migrations/`: - `LOCAL_MIGRATIONS` — sync, runs from `getCollectionDb()` and from the `SearchIndexer` constructor as a defensive re-check - `WORKER_MIGRATIONS` — async, runs from `publishCollections()` against D1 via the new `queryD1Rows` helper, with a `FTS_RESET_FROM_BELOW` marker that signals when the publish flow needs to re-INSERT every FTS row (because v2 dropped + recreated the table) Migration unit test covers fresh DB → all migrations applied, already-current → no-op, partial state → only newer ids run, and the final FTS shape ends up with the expected columns. `AGENTS.md` updated under "Publishing to Cloudflare" with a "Schema migrations — single source of truth" section that mandates the pattern: every schema change goes through migrations, every change is documented in SCHEMA.md, every change ships a publish smoke test, FTS column additions need drop+recreate (FTS5 has no ALTER ADD COLUMN), and the local + worker lists must update in lockstep. The in-process cache from the first cut was dropped after it broke test isolation when DB files get deleted + recreated at the same path. A SELECT against an indexed metadata key is well under a microsecond locally, and publish-time D1 round-trips are rare enough that the optimization isn't worth the correctness hazard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion Two changes that shrink the schema divergence to its essential minimum: - **Local migration v4** drops `collection_name` from `entities_fts` (written but never queried — vestigial from a never-shipped multi- collection FTS plan; collection scoping is implicit since each collection has its own DB file). Drop+recreate since FTS5 lacks ALTER ADD COLUMN; SyncEngine re-INSERTs rows on next sync. - **Local migration v4** also rebuilds `entities` without `AUTOINCREMENT`. The historical local DDL used `INTEGER PRIMARY KEY AUTOINCREMENT`; the worker has plain `INTEGER PRIMARY KEY`. We never depended on the no-id-reuse guarantee AUTOINCREMENT provides (FTS rows are always deleted before being re-INSERTed). Migration detects via sqlite_master.sql, builds a correctly-shaped clone, copies rows preserving ids, drops + renames, rebuilds indexes — wrapped in BEGIN/COMMIT. - Shared `ENTITIES_TABLE_DDL` + `ENTITIES_INDEX_DDL` extracted into `migrations/shared.ts`. Worker v1 and local v4 both use them so the table shape comes from one place. Local v1 keeps its historical AUTOINCREMENT-using SQL (it's the captured pre-migration state). Result: after v4, the `entities` table and `entities_fts` schema are **identical** between local and worker. Remaining divergence is intentional and runtime-specific: - `sync_errors` — local-only (sync runs locally) - `r2_manifest` — worker-only (local has no R2) `SearchIndexer` simplifies — the `hasAttachmentTextColumn` flag and the two-branch `updateIndex` / `search` are gone; migrations always bring the FTS to the latest shape so the conditional logic is unreachable. bm25 expression collapses to one form: `1.0, 1.0, 1.0, 10.0, 1.0, 5.0, 0.25` (matching `worker/src/db/search.ts`); `snippet()` column index adjusts from 5 to 4 since `collection_name` is no longer at index 0. New migration tests cover: final 7-col FTS shape, AUTOINCREMENT removal, and row-preservation on the v4 entities rebuild including index recreation. SCHEMA.md updated to describe v4, the shared DDL constants, and the intentional local↔worker differences. AGENTS.md unchanged — the "Schema migrations — single source of truth" rule already captures this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`r2_manifest` was mentioned in AGENTS.md and a couple of migration-module comments as if it were a real D1 table; it isn't. Stale R2 cleanup is done by listing the bucket directly via \`listR2Objects\` and diffing against what was just uploaded. Same with \"tags\" and \"links\" — those live inside \`entities.data\` JSON, never as separate tables. After this fix the only real intentional schema difference between the local and worker DBs is \`sync_errors\` (local-only, since the worker never syncs). Documented as such. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vboctor
added a commit
that referenced
this pull request
Apr 29, 2026
… AGENTS.md Closes the publish-side gap left by #121 / #125: the local SearchIndexer gained an `attachment_text` column with a non-destructive migration, but `publish.ts` was still creating the old 6-column D1 FTS so OCR'd text disappeared on the published worker. Three coordinated changes: - `publish.ts`: detect existing FTS schema with PRAGMA table_info; if `attachment_text` is missing on republish, drop + recreate the FTS table and force-include every entity in the push set so all FTS rows re-INSERT (FTS5 doesn't support ALTER TABLE ADD COLUMN). Fresh collections get the new schema directly via CREATE VIRTUAL TABLE IF NOT EXISTS. INSERT prefix and value rows now carry `attachment_text` joined from `data.assets[].text` — same shape SyncEngine builds for the local FTS. - `wrangler-api.ts`: new `queryD1Rows` helper exposing the result body from D1 query API so the publish path can read PRAGMA output. - `worker/db/search.ts`: bm25 expression bumped to 7 weights matching the local indexer (title 10 / tags 5 / body 1 / attachment 0.25). Tags weight bumped from 2 → 5 along the way, also matching the local change in #121 that the worker had silently missed. AGENTS.md gains a "Local ↔ worker parity rule" section under Publishing to Cloudflare. It captures the principle that schema, query, ranking, manifest, and rendered-output changes must land in the same PR for both the local and worker code paths, with a per-category checklist of layers to update. Documents that `CREATE TABLE IF NOT EXISTS` is idempotent and FTS5 needs explicit drop+recreate to evolve. Mandates that test plans for these changes include a publish to a scratch worker. Closes #126 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an Evernote v10 crawler that imports notes, notebooks, tags, and attachments from a local install — no API tokens, decoded straight from Evernote's local SQLite + Yjs CRDT files. Notes render in their original Evernote order (text, images, and PDFs interleaved), notebook membership is structured metadata (not a tag), and notebooks are pickable from the desktop "Add Collection" form with note counts and aggregate sizes.
The work also lands four cross-cutting infra changes that the Evernote crawler exercises but are useful to any future source:
EntityData.sortKey— crawler-controlled file-tree ordering (default = title), folded into the entity hash so manifest / sync / clone all stay in stepattachment_textFTS column +assets[].text— makes OCR'd text inside images and PDFs searchable, with non-destructive migration for existing collectionsFolderConfig.subdirConfig— themes can declare a default config that gets propagated into every dynamic subfolder they createIssues closed
Closes #119, #120, #121, #122, #123, #124.
Test plan
Risks / call-outs
SyncEngine.checkVersionCompatbehaviour: bumping the Evernote crawler's minor version triggers a re-render but does NOT re-fetch attachments. Users with notes affected by the old 10 MB cap needfink sync <name> --fullto pick up the previously-skipped files. Documented in Raise default attachment size cap from 10 MB to 100 MB #123.<a>fallback link there. Real browsers + the local desktop app (after enabling `webPreferences.plugins: true`) work fine.🤖 Generated with Claude Code