Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vine: imbalanced temp file replication #4056

Closed
JinZhou5042 opened this issue Feb 5, 2025 · 1 comment
Closed

vine: imbalanced temp file replication #4056

JinZhou5042 opened this issue Feb 5, 2025 · 1 comment
Labels

Comments

@JinZhou5042
Copy link
Member

JinZhou5042 commented Feb 5, 2025

In the following run I intended to set a large number of temp file replications. Although I didn’t encounter any issues like tasks being forsaken or slowdowns in the end, the imbalance in temp file replications concerns me a bit.

If the workflow runs long enough, every file will eventually reach its expected replication count. But within a specific time period, some files get replicated really well, while others are under-replicated, which could increase the risk of submitting recovery tasks.

The x-axis represents numerical file index while the y-axis represents the number of replicas for each file. File pruning was disabled.

Image

@JinZhou5042
Copy link
Member Author

solved with #4076, this was because some source workers holding a specific replica just cannot transfer more...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

No branches or pull requests

1 participant