projects consolidate over-merges unrelated projects (data-loss risk); cwd-based detection creates duplicate projects for one logical project

[Pep190272]

Summary

Two related problems around project identity:

1. Duplicate projects are created from a single logical project because project detection is derived from the current working directory. On a case-insensitive filesystem (Windows/macOS) the same path yields differently-cased project names (foo vs Foo), and running several products out of one shared parent directory (e.g. the user home) makes distinct products collapse onto that directory.
2. projects consolidate (both modes) auto-groups unrelated projects and would merge them into a single canonical project. Because a merge moves observations between projects, an accidental run causes silent data loss / cross-contamination that is hard to undo.

Environment: engram v1.16.3, Windows 10, MCP + local binary, single-user local install.

Reproduction / Evidence

A real local install accumulated 43 projects, where a large fraction are variants of the same logical project. Names below are anonymized placeholders; the casing/separator patterns are reproduced verbatim.

• Case-only duplicates: alpha (236 obs) and Alpha (222 obs); beta and Beta; home-user and Home-User.
• Separator/case variants of one project: gamma-project, Gamma-Project, Gamma_project.

Defect A — projects consolidate --all groups unrelated projects

engram projects consolidate --all --dry-run produced groups like:

Group: [.tool-x, Client-Site, Home-User, channel-integration,
        client-site, tool-x, home-user]  → merge into "client-site"
Group: [product-web, product-seo]        → merge into "product-web"
Group: [product-clean, other-site.com]   → merge into "other-site.com"

These are distinct projects (a client website, a personal/global scope, a channel integration, and separate products) being proposed for a single merge based on loose name similarity.

Defect B — narrow projects consolidate merges by "shared directory"

Run from the user home directory (C:\Users\<user>):

Detected project: "channel-integration"

Found similar project names:
  [1] home-user      116 obs  (shared directory)
  [2] Client-Site     82 obs  (shared directory)
  [3] tool-x           9 obs  (shared directory)

[dry-run] Would merge 3 project(s) into "channel-integration"

Three unrelated projects are proposed for merge solely because they were all worked on from the same home directory — a normal workflow when a user launches an agent from ~ for many different products.

Expected behavior

• Project-name derivation should be canonicalized (at minimum case-folded on case-insensitive filesystems) so the same path never yields both foo and Foo.
• consolidate should not treat "similar name" or "shared directory" alone as sufficient to merge. Auto-merge should be limited to provably equivalent names (e.g. case/normalization-equivalent), and everything else should require explicit, per-pair confirmation.
• There should be a surgical, explicit merge: engram projects merge <source> <target> for a single source→target, so users can clean up known duplicates without invoking a fuzzy grouping heuristic across the whole store.

Impact

• Users accumulate many phantom duplicate projects, fragmenting recall and inflating retrieval cost.
• The only available cleanup tool (consolidate) is unsafe to run unattended: a single --all invocation can merge unrelated projects and clients into one, with no straightforward undo. The safe workaround today is "do nothing and rely on cross-project search", which leaves the fragmentation in place.

Suggested direction

1. Canonicalize project keys on detection (case-fold + trim + normalize separators), especially on Windows/macOS.
2. Restrict consolidate auto-grouping to normalization-equivalent names; drop "shared directory" as a merge signal for distinctly-named projects.
3. Add projects merge <source> <target> for explicit, auditable single-pair merges.

Happy to provide the full anonymized project list and dry-run output if useful.