feat: 分块上传优化 #2813 #2816

zzdjx · 2024-12-03T08:02:23Z

No description provided.

owenlxu · 2024-12-04T01:54:16Z

...neric/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/GenericController.kt

@@ -68,10 +68,13 @@ import com.tencent.bkrepo.generic.constant.HEADER_UPLOAD_ID
 import com.tencent.bkrepo.generic.pojo.BatchDownloadPaths
 import com.tencent.bkrepo.generic.pojo.BlockInfo
 import com.tencent.bkrepo.generic.pojo.CompressedFileInfo
+import com.tencent.bkrepo.generic.pojo.NewBlockInfo


这里新启用一个controller吧，不要和原来代码放在一起了

owenlxu · 2024-12-04T06:47:29Z

.../common/common-mongo/src/main/kotlin/com/tencent/bkrepo/common/mongo/dao/AbstractMongoDao.kt

@@ -135,6 +135,15 @@ abstract class AbstractMongoDao<E> : MongoDao<E> {
        return determineMongoTemplate().updateFirst(query, update, determineCollectionName(query))
    }

+    fun updateBlock(query: Query, update: Update, clazz: Class<E>): UpdateResult


为什么这个方法放在AbstractMongoDao

owenlxu · 2024-12-04T07:01:22Z

...main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/AbstractArtifactRepository.kt

@@ -133,6 +137,19 @@ abstract class AbstractArtifactRepository : ArtifactRepository {
        }
    }

+


这里新加的方法如果其他地方没有复用，建议不在AbstractArtifactRepository类中加方法

owenlxu · 2024-12-04T07:07:36Z

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

+    override fun updateBlock(blockNode: TBlockNode) {
+        with(blockNode) {
+            val criteria = Criteria.where(TBlockNode::id.name).`is`(id)
+                .and(TBlockNode::createdBy.name).`is`(createdBy)


这个update没必要写这个多条件吧

owenlxu · 2024-12-04T07:13:37Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

@@ -167,6 +219,51 @@ class UploadService(
        logger.info("User[${SecurityUtils.getPrincipal()}] complete upload [$artifactInfo] success.")
    }

+    fun completeNewBlockUpload(


这里是不是可以在一行

yaoxuwan · 2024-12-10T03:36:55Z

...ackend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/model/NodeAttribute.kt

@@ -0,0 +1,22 @@
+package com.tencent.bkrepo.generic.model
+
+data class NodeAttribute(


不要复制重复代码，可以放到common-metadata

yaoxuwan · 2024-12-10T07:13:23Z

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt

+                val blockCriteria = buildCriteria(projectId, repoName, fullPath, deleteTime)
+                val node = nodeDao.findOne(Query(blockCriteria))
+                if (node?.sha256 == FAKE_SHA256) {
+                    nodeBaseService.blockNodeService.deleteBlocks(projectId, repoName, fullPath)


blocknode删除现在是异步的，通过消息队列发送事件到fs-server，fs-server删除。
可以整改下，在服务内部异步删除

cnlkl · 2024-12-11T07:02:23Z

...ata-service/src/main/kotlin/com/tencent/bkrepo/common/metadata/dao/blocknode/BlockNodeDao.kt

-class BlockNodeDao : HashShardingMongoDao<TBlockNode>()
+class BlockNodeDao : HashShardingMongoDao<TBlockNode>(){
+
+        fun updateBlock(query: Query, update: Update): UpdateResult {


这里为什么要自己写一个update方法，不能直接使用父类的updateFirst(query,update)吗

cnlkl · 2024-12-11T07:27:38Z

...ice/src/main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/ArtifactRepository.kt

+     *
+     * @param context 构件上传上下文
+     */
+    fun newUpload(context: ArtifactUploadContext) {}


只有generic支持的分块上传不建议方这，这里主要定义的是所有依赖源共用的接口

cnlkl · 2024-12-11T07:29:52Z

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt

@@ -59,7 +59,7 @@ import java.time.LocalDateTime
 /**
 * 节点删除接口实现
 */
-open class NodeDeleteSupport(
+open class NodeDeleteSuspport(


这里为什么改类名

cnlkl · 2024-12-11T07:36:24Z

...metadata-service/src/main/kotlin/com/tencent/bkrepo/common/metadata/util/NodeDeleteHelper.kt

    ): Criteria {
        val normalizedFullPath = PathUtils.normalizeFullPath(fullPath)
        val normalizedPath = PathUtils.toPath(normalizedFullPath)
        val escapedPath = PathUtils.escapeRegex(normalizedPath)
        val criteria = where(TNode::projectId).isEqualTo(projectId)
            .and(TNode::repoName).isEqualTo(repoName)
-            .and(TNode::deleted).isEqualTo(null)
+            .and(TNode::deleted).isEqualTo(deletedDate)


这里原来是没限制deletedDate字段值的，现在没传deletedDate参数时会只查询出该字段为null的值

cnlkl · 2024-12-11T09:16:55Z

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

+        @RequestAttribute userId: String,
+        @ArtifactPathVariable artifactInfo: GenericArtifactInfo,
+    ): Response<NodeDetail> {
+        uploadService.newStartBlockUpload(userId, artifactInfo)


generic中上传和fs-server的使用场景不太一样，generic偏向完整文件的上传下载，这里如果一开始就创建node会导致文件未完成上传就对用户可见

cnlkl · 2024-12-11T09:21:07Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

+            md5 = FAKE_MD5,
+            operator = userId,
+            size = getLongHeader(HEADER_FILE_SIZE).takeIf { it > 0L }
+    ?: throw ErrorCodeException(GenericMessageCode.BLOCK_HEAD_NOT_FOUND) ,


这里写在一行不好阅读

getLongHeader(HEADER_FILE_SIZE).takeIf { it > 0L } ?: throw ErrorCodeException(GenericMessageCode.BLOCK_HEAD_NOT_FOUND)

cnlkl · 2024-12-11T09:22:52Z

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

+        return ResponseBuilder.success()
+    }
+
+    @Permission(ResourceType.REPO, PermissionAction.READ)


这里为什么校验repo权限而不是node权限

cnlkl · 2024-12-11T09:27:05Z

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

+        @RequestAttribute userId: String,
+        @ArtifactPathVariable artifactInfo: GenericArtifactInfo,
+    ): Response<Void> {
+        uploadService.abortNewBlockUpload(userId, artifactInfo)


这里如果是覆盖上传的情况，中断后会把原文件也删掉

cnlkl · 2024-12-11T09:33:30Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

+            operator = userId,
+            size = getLongHeader(HEADER_FILE_SIZE).takeIf { it > 0L }
+    ?: throw ErrorCodeException(GenericMessageCode.BLOCK_HEAD_NOT_FOUND) ,
+            overwrite = getBooleanHeader(HEADER_OVERWRITE),


覆盖上传时，会导致一次下载可能同时下载到新文件和旧文件的数据

cnlkl · 2024-12-11T09:48:13Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

+        var offset = 0L  // 用于记录当前偏移量
+
+        // 遍历块信息列表，创建对应的块节点
+        blockInfoList.forEach { blockInfo ->


上传分块的时候有办法知道blockSize然后计算出endPos吗，这样就不用最后再批量更新一次了

zacYL · 2024-12-12T02:41:57Z

docs/apidoc/generic/SeparateBlock.md

+## 初始化分块上传
+
+- API: POST /generic/separate/block/{project}/{repo}/{path}
+- API 名称: start_block_upload


这个文档和apidoc-user下的文档有啥区别

zacYL · 2024-12-12T02:55:42Z

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

+        instance = AuditInstanceRecord(
+            resourceType = NODE_RESOURCE,
+            instanceIds = "#artifactInfo?.getArtifactFullPath()",
+            instanceNames = "#artifactInfo?.getArtifactFullPath()"


只有最终上传成功生成节点才需要记录审计日志

feat: 分块上传优化 TencentBlueKing#2813

8642550

zzdjx requested review from cnlkl, yaoxuwan and felixncheng as code owners December 3, 2024 08:02

zzdjx requested review from owenlxu and removed request for cnlkl, felixncheng and yaoxuwan December 3, 2024 08:02

fix: 分块上传优化-问题修复 TencentBlueKing#2813

5cbe04d

owenlxu reviewed Dec 4, 2024

View reviewed changes

zzdjx added 3 commits December 6, 2024 16:36

Merge remote-tracking branch 'upstream/master' into dev_kurt

613b9f8

fix: 分块上传优化-代码调整+单测 TencentBlueKing#2813

a80c07a

fix: 分块上传优化-代码调整 TencentBlueKing#2813

d65bc84

zzdjx requested review from yaoxuwan, cnlkl, zacYL and felixncheng December 9, 2024 07:52

yaoxuwan reviewed Dec 10, 2024

View reviewed changes

zzdjx added 2 commits December 10, 2024 15:01

Merge remote-tracking branch 'upstream/master' into dev_kurt

8009fcf

fix: NodeAttribute抽离公共放到metadata TencentBlueKing#2813

c21d488

yaoxuwan reviewed Dec 10, 2024

View reviewed changes

zzdjx added 3 commits December 11, 2024 15:16

fix: 新增Generic下Block事件监听 TencentBlueKing#2813

318f65a

fix: 代码规范 TencentBlueKing#2813

ba35f7f

fix: 错误修复 TencentBlueKing#2813

89a7905

cnlkl requested changes Dec 11, 2024

View reviewed changes

zacYL reviewed Dec 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: 分块上传优化 #2813 #2816

feat: 分块上传优化 #2813 #2816

zzdjx commented Dec 3, 2024

owenlxu Dec 4, 2024

owenlxu Dec 4, 2024

owenlxu Dec 4, 2024

owenlxu Dec 4, 2024

owenlxu Dec 4, 2024

yaoxuwan Dec 10, 2024 •

edited

Loading

yaoxuwan Dec 10, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

cnlkl Dec 11, 2024

zacYL Dec 12, 2024

zacYL Dec 12, 2024

		@@ -133,6 +137,19 @@ abstract class AbstractArtifactRepository : ArtifactRepository {
		}
		}

		@@ -0,0 +1,22 @@
		package com.tencent.bkrepo.generic.model

		data class NodeAttribute(

feat: 分块上传优化 #2813 #2816

Are you sure you want to change the base?

feat: 分块上传优化 #2813 #2816

Conversation

zzdjx commented Dec 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaoxuwan Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaoxuwan Dec 10, 2024 •

edited

Loading