Skip to content

refactor(cache): change CacheFS to query-driven caching - #6346

Draft
baojun-zhang wants to merge 3 commits into
volcengine:mainfrom
baojun-zhang:cache-refactor
Draft

baojun-zhang wants to merge 3 commits into
volcengine:mainfrom
baojun-zhang:cache-refactor

Conversation

@baojun-zhang

@baojun-zhang baojun-zhang commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

将cachefs 对象从文件状态记录驱动(file/dir/subtree 模式)优化为查询结果记录驱动(path + op)
旧版本问题:

  1. 使用file/dir/subtree 对象模式缓存. file/dir 缓存文件系统对应内容,subtree 维护目录版本关系。如果 10w 文件 + 1w 目录近期被访问,最多会生成约10w + 1w + 1w = 12w 个 cache key
  2. 文件全部内容会被缓存到 redis 中 ,相比文件系统直接读取, 收益不大,成本变高
  3. 通过 subtree 维护file/dir之间的关系, 文件变更需要同时考虑考虑 file key、父目录 dir key、祖先 subtree key, 逻辑复杂
  4. 对检索不友好, 比如一次grep 为了收集结果扫了 1000 个文件,可能额外生成约 1000 个file key,并把这 1000 个文件正文放进 Redis。 ls/tree/glob/ 同理

新版:

  1. 不缓存文件正文
  2. 新版按path + op 建 key, 不再因文件正文读取天然膨胀。
  3. ragfs 层查询参数驱动,更容易命中
  4. 失效逻辑简单, 按受影响 path 删除相关 query bucket cache 即可

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • 将 RAGFS CacheFS 从file/dir/subtree 对象缓存改为path + op 查询结果缓存。
  • 不再缓存文件正文,只缓存stat 、read_dir 、tree_directory 、glob_directory 、grep 的查询结果。
  • 使用 Redis HASH bucket 存储查询结果,用 request hash 作为 field,并在命中后校验 request,避免错读缓存。
  • 写操作后异步失效缓存,通过UNLINK 删除 query bucket,并用全局 ZSET index 清理后代 bucket。
  • 排除.path.ovlock 、.exact.ovlock.* 等 PathLock 内部文件,避免进入查询缓存。
  • 为 Redis / Dynamic / Memory CacheRuntime 补齐 HASH、ZSET、UNLINK 原语,并同步更新指标、测试和文档。

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

Additional Notes

@baojun-zhang
baojun-zhang marked this pull request as draft October 20, 2026 10:06
@baojun-zhang baojun-zhang changed the title Cache refactor refactor(cache): change CacheFS to query-driven caching Oct 20, 2026
@ZaynJarvis ZaynJarvis added the kernel Core server, runtime, storage, retrieval, SDK, and CLI. label Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Core server, runtime, storage, retrieval, SDK, and CLI.

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants