Feat: 添加未提取记忆补偿+遗忘曲线门控 - #69
Open
TriDefender wants to merge 32 commits into
Open
Conversation
可以重新尝试提取之前未提取的记忆节点，在正式提取前会让用户确认。
添加模仿艾宾浩斯遗忘曲线的门控机制，长期不用的节点会被deprecate掉（非硬性删除，可被重新激活）避免过时噪声影响检索结果
移植了adoresever@1fdec04 src/store/store.ts - CommunitySummary 接口新增 memberSignature: string | null - upsertCommunitySummary() 增加 memberSignature 参数，MERGE 时写入（沿用 embedding 的 CASE 保留模式：传 null 不清空旧值） - getCommunitySummary() / getAllCommunitySummaries() 返回 memberSignature - 新增 getCommunitySummaryBySignature()：按签名查社区（ORDER BY updatedAt DESC LIMIT 1），带回 embedding 供复用 src/graph/community.ts - 新增导出 buildCommunityMemberSignature()：成员 ID 排序后 sha1（与上游逐字等价） - summarizeCommunities() 循环内两层短路： 1. 签名未变且摘要非空 → 跳过，不调 LLM 2. 其他社区存在相同签名 + 非空摘要 → 复用其 summary + embedding，不调 LLM 3. 否则走原有 LLM 生成路径，upsert 时写入签名
This commit aims to tackle more finegrained memory control on cron sessions: Does a repeating cron session generates 'false popularity' on specific memory nodes? Now it enables you to: - disable memory extraction for cron runs - disable entirely graph functions, so it will not inject context information (it may change workflows, use this with caution since models may lack crucial information. you're advised to test the workflow before hand) - skip session end actions on cron
- 修掉了原来多会话并发结束会并发跑两次全局维护的竞态 - 防止插件启动时检测不到embeddings就整个生命周期回退到FTS
数据损坏/丢失： - gm_maintain 与后台维护链并发（validatedCount 双计、communityId 互踩）： scheduleMaintenance 改为类型化单飞 promise，全部维护入口收敛同一互斥点 - computeGlobalPageRank：GDS write 成功后 drop/topK 失败不再回落 1/(i+1) 覆盖真实 PageRank（drop 失败单独吞，宁泄漏临时投影） - summarizeCommunities：prune 挪到签名复用之后——先 prune 会删掉重编号 场景下的捐赠者社区，导致每轮维护全量重算 LLM 摘要+embedding - graph-memory extract：批边界 await 所有 syncEmbed，closeDriver 不再 丢失最后一批在途向量（markExtracted 后不可自愈） - PUT /nodes：改 type 与非法边清理（EDGE_DIRECTION_RULES 生成谓词）包进 单 executeWrite 事务；name 重名预检 409、空名 400 功能失效： - db.ts：删除不可达的 driver 重连分支（驱动 5.x session() 从不抛错）， getSession 永远优先模块级 _driver 单例 - llm.ts：OAuth 缓存按 oauthPath mtime 失效（CLI 刷新 token 后网关 下一次调用即生效，无需重启）；Anthropic 遍历找 text 块（thinking 块 在前不再误报 empty）；openai/anthropic 路径补 fetchRetry（429/5xx） - recall：双路径 Promise.all 并行 + query 向量算一次共享 + assemble 复用 未变 prompt 的缓存（4 embed/4 投影 → 1 embed/2 并行遍历） - flushMessageBuffer 成功后补排 maintenance（熔断跳过的一轮不再丢失） 竞态/一致性： - upsertNode：撞 *_name 唯一约束退回更新路径（幂等）；records[0] 守卫 - upsertEdge：三段查询合并为单条 apoc.merge.relationship（消除并发重复边） - session_end finalize 包进 withExtractLock；gm_record/gm_link 溯源统一 sessionId；crud 内联 normalizeName 改用 store 导入（受一致性测试保护） 性能： - detectDuplicates：N 次向量查询折叠为单条 UNWIND+CALL - applyDecay：Math.max spread 改 reduce（防超大节点集爆栈） - assemble：2N 次串行边查询改 edgesTouching 单次批量；社区摘要并发预取 - communityRepresentatives 加 totalLimit 封顶；graphWalk clamp maxDepth - recall 超时 30s 退避（withBudget 不取消底层查询，防堆积） 配置/测试： - schema recallMaxNodes 3→6 对齐代码默认；freshTailCount 接线到 sliceLastTurn（默认 5=原硬编码 KEEP_TURNS，行为不变） - integration.recall/routes 支持 NEO4J_TEST_URI（消除 7687 硬编码） - 新增集成用例：TASK→EVENT 改 type 边清理、name 重名 409、社区摘要
Contributor
Author
|
都是经过production环境 测试过的 |
某些宿主版本会在未 dispose 的情况下重复调用 register()（热重载/配置变更）： 重复注册 hook/工具/路由让每轮工作翻倍，且新引擎实例持有自己那份 per-session 提取锁，与旧实例并行时同一 session 的提取互斥被打破。现在重复 register() 只重绑 ContextEngine 工厂并告警；dispose() 清空标记后才允许真正的重载。 守卫放在 CLI 元数据注册之后，保持 cli-metadata 模式不加载运行时的契约。 顺带：llm.baseUrl / embedding.baseUrl 小写拼写归一到 baseURL（显式 baseURL 优先），与上游 52db397 对齐。
移植自上游 adoresever#96（SQLite → Neo4j Cypher）。原则不变：上下文压缩不构成 删除持久证据的授权 —— 默认 keep=all 零行为变化，必须显式 opt-in。 - keep=referenced：只删已提取完成的消息（知识已固化进图谱）；v2.0 无消息级 出处边，extracted=true 即等价上游'无 gm_node_sources 引用'前置条件 - keep=recent：叠加时间窗（每 session 最近 N 轮真实用户发言 + 最近 N 天） - 每 batch 有界（默认 500 行/维护周期），dryRun 只报候选不删 - 候选选择与删除同一写事务，DELETE 前按 extracted 重新校验 - 挂在 runMaintenance 尾部；register 时预校验配置（非法 fail closed 回退 keep=all） - openclaw.plugin.json configSchema 声明 messageRetention 测试安全加固（本地误连真实库事故的整改）： - integration.retention.test.ts 必须显式提供 NEO4J_TEST_URI（不默认 7687） - 隔离守卫：库内存在非测试前缀 GmMessage 即拒绝运行 - ci.yml 显式设置 NEO4J_TEST_URI（ephemeral runner 安全） - 本地测试容器固定 7688 端口
…of a ContextEngine object.. 修复如下问题： 07:54:21 [context-engine] Context engine "graph-memory-pro" degraded to "legacy" for this logical turn: Context engine "graph-memory-pro" factory returned null instead of a ContextEngine object.. The "legacy" engine will handle only this turn; configuration is unchanged, and "graph-memory-pro" will be retried next turn.
…ing (#1) Dead code found by cross-referencing every export (production, CLI, tests) on the desktop-2.0 codebase: Remove entirely (zero callers): - vectorSearch(): legacy compat wrapper, only ever wrapped vectorSearchWithScore(); nothing (production or tests) calls it - getAllVectors(): superseded by the Neo4j vector index — dedup now uses db.index.vector.queryNodes instead of pulling all embeddings - getAllCommunitySummaries(): no callers anywhere - updatePageranks(): production writes pagerank via GDS (gds.pageRank.write / uniform fallback) directly in pagerank.ts; this batch writer was only referenced by one integration test, which now sets pagerank via Cypher directly (same coverage of the topNodes read path) De-export (still used internally, no external importers): - ScoredNode / ScoredCommunity return types (store.ts) - LlmConfig (llm.ts — cli.ts only imports ReasoningEffort) - RecallOptions / RecallTimeField (recall.ts) Verification: tsc clean; vitest 239 passed / 76 skipped (Neo4j integration gated behind NEO4J_INTEGRATION=1). The one failing test (installer-upgrade, spawnSync bash ETIMEDOUT) fails identically on the unmodified branch — sandbox networking, unrelated to this change. Co-authored-by: TriDefender <173548745+TriDefender@users.noreply.github.com>
A1. Consolidate duplicated fetchRetry into src/engine/http.ts
- llm.ts / embed.ts each carried a near-identical copy that had
already drifted: the llm side lost the network-error retry branch
during an earlier refactor. Shared module restores it for both.
- Timeout errors are now a typed HttpTimeoutError; per-caller
semantics preserved via retryOnTimeout (llm: false — a 60s
timeout must not quadruple worst-case latency; embed: true).
- Exhausted status retries now return the last response instead of
a generic "failed after retries" error, so callers surface
status-specific messages.
- Add unit tests (8) covering status retry, network-error retry,
timeout semantics, and label formatting.
A2. extract.ts imports normalizeName from store.ts instead of carrying
a byte-identical copy guarded by a cross-file consistency test.
A3. New src/tokens.ts (CHARS_PER_TOKEN + estimateTokens); index.ts,
recall.ts and assemble.ts now share the /3 heuristic.
A4. Drop RecallResult.tokenEstimate — computed in three places, never
read by anyone (updated 4 test fixtures accordingly).
B1. store.ts uid(): crypto.randomUUID() replaces the hand-rolled
Date.now()+Math.random() suffix.
B3(partial). index.ts: extractAssistantText/extractUserText share
textFromBlocks(); llm.ts/store.ts block-parsing left as-is
(different input shapes).
Deliberately out of scope: oauth.ts toBase64Url wrapper (cosmetic),
JWT/SSE hand-rolled parsers (zero-dependency by design).
tsc clean; vitest 241 passed / 76 skipped. The one failure
(installer-upgrade, spawnSync bash ETIMEDOUT) is a pre-existing sandbox
networking limitation, reproduced on the unmodified branch.
Fixes #2
refactor: dedupe shared implementations — shared fetchRetry with restored network-error retry, single normalizeName, unified token estimate (Fixes #2)
…mory reembed) Switching embedding models leaves old-model vectors in place: recall/dedup guard with `WHERE embedding IS NOT NULL`, so stale vectors silently degrade (or break the dedup UNWIND on dimension change), and `graph-memory extract` backfill only covers new nodes. - embed.ts: `createEmbedder()` returns single `embed` + batched `embedBatch` (shared probe; input[]/texts[] bodies, 2s-per-text timeout scaling); pure `parseBatchEmbeddingResponse()` reassembles data[] by index and validates count/shape. `createEmbedFn` kept as a single-text wrapper. - store.ts: embedding stats, clearAllEmbeddings (removes embedding AND contentHash so syncEmbed cannot short-circuit on the stale hash), ID-cursor-paginated embedding targets (SKIP pagination breaks once the NULL set shrinks), saveCommunityEmbedding, getVectorIndexDimensions, dropVectorIndexes. - cli-reembed.ts: `graph-memory reembed` — probe dims vs vector index dims (abort on mismatch, `--recreate-index` to drop/recreate via initSchema), void, cursor-paginated batch rebuild (buildNodeEmbeddingText reused so contentHash semantics match runtime syncEmbed), per-item fallback when a batch call fails. Flags: --dry-run / --yes / --batch <n> / --recreate-index. - cli.ts: register the reembed subcommand. - tests: reembed.test.ts (planReembed + batch response parsing, 14 cases); integration.neo4j.test.ts +2 live-DB cases (void/cursor/community/index dims). - docs: README.md / README_CN.md re-embedding sections.
Deprecation只保留三种： 1、节点不被调用根据时间流逝下调重要性直至被标记失效 2、手动断联并且标记过时 3、被合并（也就是变成类似占位符的东西仅维持图结构) delete只保留一种： 1、Deprecate后六十天内没有再被查出来用就删
…eprecatedAt backfill 两处评审修复： - autoDeprecateNodes MATCH 加 status:'active' 守卫：评分快照与批量写入的 时间窗内刚被手动弃用的节点不再被覆盖为 deprecatedBy='decay' + 新 deprecatedAt（既改变复活语义又重置 60 天 purge 时钟） - initSchema 启动时幂等补写：为缺 deprecatedAt 的存量 deprecated 节点 钉死 deprecatedAt = coalesce(updatedAt, createdAt)，否则 upsertNode 对 manual/merge 弃用节点的 updatedAt bump 会无限推迟 purge（永远删不掉） - 新增两个集成测试（状态守卫 + 存量补写/漂移回归/幂等），docs/decay.md 同步
- 提取模式 cfg.extract.mode：per-turn（默认）/ batched 攒批提取，LLM 调用 次数降为 ~1/N，session_end 冲洗尾批 - trivial 轮本地预筛（turn-filter）：清洗后为空/无意义词表/超短无技术词的 轮次直接 markExtracted(producedKnowledge=false)，省一次完整 completion - 查询向量 LRU 缓存（query-cache）：同会话重复召回复用 query embedding， 零 embedding API 调用（db 模式不入缓存） - 批量向量读写：getVectorHashes/saveVectors 一次 UNWIND 替代 N 次单节点 往返（syncEmbedBatch 用） - finalize 阶梯触发 + 社区摘要 top-k 稳定签名，削减 session_end LLM 开销 - 测试：turn-filter/query-cache/extract-cost-guards 单测 + 消融实验 + 真实服务 embedding E2E（EMBED_E2E 门控）
代码审查发现(专项扫描) 确认重复 13 处、疑似重叠 11 处，同时有 4 个既有约定(t tokens/http/normalizeName/uid)验证干净。 已实施的精修(12 文件，-374/+247 行) 修复真实缺陷:exchangeAuthorizationCode(OAuth 登录)原是裸 fetch 无超时，CLI 登录有卡死风险；现与 refreshOAuthSession 共用新提取的 postTokenRequest,统一走 http.ts 的 fetchWithTimeout,错误消息格式保持不变。 消灭最高危的三份复制：提取结果持久化体(upsertNode → syncEmbedBatch → upsertEdge)在 index.ts×2 + cli-extract.ts×1 逐字重复且已漂移，收敛为新的 src/extractor/persist.ts(awaitEmbedSync 开关保留运行时 fire-and-forget / CLI 必须等待的语义差异)。 边类型白名单 8 处 → 1 处：store.ts 5 条 Cypher、projection.ts、crud.ts、index.ts 的 TypeBox union 全部改为从 EDGE_TYPES 派生——以后加新边类型只改 types.ts 一处。 store.ts 边行投影 5 份 → EDGE_ROW_RETURN + mapEdgeRecords;REST 删边下沉为 deleteEdgeById/deleteEdges,路由层不再写 Cypher(响应新增 deleted 计数，additive)。 oauth/llm 错误样板 7 份 → throwForStatus(http.ts 新导出，消息格式逐字保留)；llm.ts 内联的 Responses output_text 遍历改为复用 oauth.ts 导出的解析函数(多段文本从无分隔拼接统一为 "\n" 连接)。 其他：pagerank.ts 三份 top-20 读取收敛为 readTopKByPagerank;index.ts 两处内联 top-pagerank Cypher 改用 store.topNodes;saveVector 接受预计算 hash 消灭 md5 双算；修复 commit-turn 测试 mock 里的幽灵字段；给 recall 双路径相反的 tiebreak 顺序补了说明注释(有意差异，消融 harness 镜像了它)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
本PR添加了两个新功能：
openclaw graph-memory这个命令行handle触发对未提取的记忆重新入库+向量化，避免某些技术原因（llm超时等）遗漏过往记忆；