Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: explicitly specify column groups for storage v2 api #39790

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

shaoting-huang
Copy link
Contributor

@shaoting-huang shaoting-huang commented Feb 11, 2025

  • use the new packed reader and writer api to be compatible with current etcd meta
  • For the new packed writer API: column groups and paths are explicitly defined by users and won't split column groups by memory in storage v2. Packed writer follows the user-defined column groups to split arrow record and write into the corresponding file path.
  • For the new packed reader API: read paths are explicitly defined by users.
    related: [Feature]: storage v2 support #39173

@sre-ci-robot sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. area/compilation labels Feb 11, 2025
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Feb 11, 2025
Copy link
Contributor

mergify bot commented Feb 11, 2025

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Copy link
Contributor

mergify bot commented Feb 11, 2025

@shaoting-huang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link

codecov bot commented Feb 11, 2025

Codecov Report

Attention: Patch coverage is 81.55340% with 19 lines in your changes missing coverage. Please review.

Project coverage is 81.66%. Comparing base (15c8798) to head (b6b6555).
Report is 57 commits behind head on master.

Current head b6b6555 differs from pull request most recent head f267af7

Please upload reports for the commit f267af7 to get more accurate results.

Files with missing lines Patch % Lines
internal/storagev2/packed/packed_writer.go 70.00% 5 Missing and 4 partials ⚠️
internal/storagev2/packed/packed_reader.go 62.50% 3 Missing and 3 partials ⚠️
internal/storagev2/packed/util.go 50.00% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #39790       +/-   ##
===========================================
+ Coverage   69.39%   81.66%   +12.27%     
===========================================
  Files         302     1187      +885     
  Lines       27077   179033   +151956     
===========================================
+ Hits        18790   146211   +127421     
- Misses       8287    26837    +18550     
- Partials        0     5985     +5985     
Components Coverage Δ
Client 79.24% <ø> (∅)
Core ∅ <ø> (∅)
Go 81.77% <81.55%> (∅)
Files with missing lines Coverage Δ
internal/storage/serde_events.go 75.03% <ø> (ø)
internal/storage/serde_events_v2.go 61.71% <100.00%> (ø)
internal/util/initcore/init_core.go 89.60% <100.00%> (ø)
internal/storagev2/packed/util.go 50.00% <50.00%> (ø)
internal/storagev2/packed/packed_reader.go 72.72% <62.50%> (ø)
internal/storagev2/packed/packed_writer.go 78.18% <70.00%> (ø)

... and 1483 files with indirect coverage changes

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shaoting-huang
To complete the pull request process, please assign tedxu after the PR has been reviewed.
You can assign the PR to them by writing /assign @tedxu in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Feb 13, 2025
Signed-off-by: shaoting-huang <[email protected]>
Copy link
Contributor

mergify bot commented Feb 13, 2025

@shaoting-huang go-sdk check failed, comment rerun go-sdk can trigger the job again.

1 similar comment
Copy link
Contributor

mergify bot commented Feb 14, 2025

@shaoting-huang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: shaoting-huang <[email protected]>
Signed-off-by: shaoting-huang <[email protected]>
@mergify mergify bot removed the ci-passed label Feb 15, 2025
Copy link
Contributor

mergify bot commented Feb 15, 2025

@shaoting-huang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Signed-off-by: shaoting-huang <[email protected]>
Copy link
Contributor

mergify bot commented Feb 16, 2025

@shaoting-huang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@shaoting-huang shaoting-huang changed the title feat: storage v2 packed api integration feat: explicitly specify column groups for storage v2 api Feb 17, 2025
Copy link
Contributor

mergify bot commented Feb 17, 2025

@shaoting-huang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Copy link
Contributor

mergify bot commented Feb 18, 2025

@shaoting-huang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link
Contributor

mergify bot commented Feb 18, 2025

@shaoting-huang cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

Copy link
Contributor

mergify bot commented Feb 18, 2025

@shaoting-huang E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/compilation dco-passed DCO check passed. kind/feature Issues related to feature request from users size/XL Denotes a PR that changes 500-999 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants