-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat(drivers): add a driver that divides large files into multiple chunks #1153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces a new "chunk" driver that splits large files into multiple smaller parts with size limitations. The driver acts as middleware similar to the existing alias and crypt drivers, filtering chunk files from listings and providing a unified RangeReader that concatenates all chunks when accessing files.
- Adds a complete chunk driver implementation with file splitting/merging capabilities
- Implements all standard driver operations (List, Link, Move, Rename, Copy, Remove, Put) with chunk-aware logic
- Registers the new driver in the driver registry
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| drivers/chunk/meta.go | Driver metadata and configuration with Addition struct defining chunk parameters |
| drivers/chunk/driver.go | Complete driver implementation handling file chunking operations and range reading |
| drivers/all.go | Registration of the new chunk driver import |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ILoveScratch2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
性能可以优化,不过没什么太大问题
怎么优化? |
|
思路可以优化一下,如果按照文件名+后缀的形式区分分配,不可避免的会导致在重命名或移动时(特别是大文件,分块过多的情况下)发送大量的请求,并且因为与网盘的交互基本上没法做到事务,这种涉及大量文件的操作极有可能会因为网络波动或其他原因导致分块的操作丢失。我的建议是以文件夹作为最小的移动或重命名单位,保证对网盘的操作只有一次,并且可以使用文件的hash作为文件名,配合上后缀还可以做断点续传,原始文件名可以编码后作为文件夹的名称。具体细节你可以自己完善 |
那做的就和这个不一样了。主要是扫描分块消耗性能远比普通文件大 |
的确,这也是问题,特别是link基本上没有利用上缓存的优势基本上是重新扫了一遍。其实可以自定义Object对象将一些信息存下在link是就可以避免重新获取,直接断言类型把数据取出来就行。并且link提前获取所有link对于网盘来说如果取消下载的话代价有点太高了 |
有缓存的,chunk的model.Link有 |
我指的是某些情况下可能并不需要读取完整的文件(比如只读取头部)。如果每个文件都要加载完整的link的话就会导致在不必要的情况下发送 (文件数*分块数)个获取link的http请求,对于某些api比较拉跨的网盘缓存未命中的情况下会等待的比较久 |
564d2bd to
6636abd
Compare
6636abd to
9c6dd2b
Compare
|
上传失败时,要删除分片文件夹吗 @KirCute @Suyunmeng |
要删吧 |
你“有的网盘”是蓝奏云吧 |
你说的文件名限制是不是指后缀限制啊,CustomExt那个字段可以自定义分片文件的后缀 |
Ext这个名字还是太抽象了(虽然已经提了中文翻译,但是英文还是ext三个字母) |
|
@bingqingpeng |
我已经把所有回复都删除了 |
Description / 描述
添加了一个将文件划分为限制了大小的若干块的驱动,以文件夹的形式保存分片文件,该文件夹名称为
[openlist_chunk]源文件名,分片文件支持自定义后缀以应对云盘对文件类型的限制。这是一个与
alias和crypt类似的中间件驱动,调用该驱动的List方法得到的文件列表,如果文件夹的前缀为[openlist_chunk]将会被映射为文件,该大小为所有分片文件大小之和,调用Link方法得到的是一个拼接了所有文件分片的RangeReader。Motivation and Context / 背景
Closes #1074
How Has This Been Tested? / 测试
测试了后端存储为
local、uss和lanzou的情况,暂时没有发现问题Checklist / 检查清单
我已阅读 CONTRIBUTING 文档。
go fmtor prettier.我已使用
go fmt或 prettier 格式化提交的代码。我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
我已相应更新了相关仓库(若适用)。