refactor(server): extract data transfer crate (#1243)

[lzjqsdd]

Summary

Extract a new curvine-data-transfer crate for server-side data transfer orchestration and shared UFS/RPC plumbing. The existing curvine-server, curvine-worker, and curvine-master module paths now re-export from the new crate where needed, so behavior and public call sites stay stable.

Issue Describe / Design

Related to #1243.

Design decisions:

• Move transfer copy orchestration, worker-side load task execution, UFS client/factory/manager, JobWorkerClient, and RpcContext into crates/server/curvine-data-transfer.
• Keep existing server/worker/master compatibility module paths as thin re-exports to avoid behavior changes.
• Replace server-side uses of the public curvine-client facade with direct dependencies on curvine-client-core, curvine-job-client, and curvine-unified-fs.
• Feature-gate transfer server and store backend dependencies so non-transfer-server users of curvine-data-transfer do not pull in web or SQL backend crates.
• Leave master-private legacy JobManager / LoadJobRunner in curvine-master because they depend on MasterFilesystem, MountManager, and journal/master state; those modules now consume data-transfer UFS/RPC plumbing directly.

Changes

Module / File	Change	Impact on existing behavior
crates/server/curvine-data-transfer	New crate with module roots, UFS backend features, and transfer server/store features.	None; new owner for moved orchestration code.
crates/server/curvine-data-transfer/Cargo.toml	Makes axum, curvine-web, rusqlite, mysql, and uuid optional behind transfer-* features.	Non-transfer-server dependents avoid these direct dependencies.
curvine-server/src/transfer/*	Moved transfer store/planner/scheduler/service/server modules to curvine-data-transfer; curvine-server enables transfer-server.	None; curvine_server::transfer::* remains via re-export.
curvine-worker/src/worker/task/*	Moved worker load task runner/context/store/manager to curvine-data-transfer.	None; curvine_worker::worker::task::* remains via re-export.
UFS/RPC plumbing	Moved UfsClient, UfsManager, UfsFactory, JobWorkerClient, and RpcContext to curvine-data-transfer.	None; old master/server re-export paths remain.
Cargo dependencies	Removed server-side curvine-client facade dependencies and wired direct lower-level crates.	Reduces internal facade dependency surface; no runtime behavior change.

Test verified

Test case	Result	Notes
make format	PASS	Run on exact HEAD eca53b2f2196c57b65ec69951879fece74c4faf9.
cargo check -p curvine-data-transfer	PASS	Verifies default worker/common path without transfer server dependencies.
cargo check -p curvine-data-transfer --features transfer	PASS	Verifies transfer core feature.
cargo check -p curvine-data-transfer --features transfer-store-sqlite	PASS	Verifies SQLite store feature.
cargo check -p curvine-data-transfer --features transfer-store-mysql	PASS	Verifies MySQL store feature.
cargo check -p curvine-data-transfer --features transfer-web	PASS	Verifies transfer web feature.
cargo check -p curvine-data-transfer --features transfer-server	PASS	Verifies full transfer server path.
cargo check -p curvine-master -p curvine-worker -p curvine-server -p curvine-web	PASS	Validates adjusted dependents.
cargo test -p curvine-data-transfer --features transfer-server	PASS	5 passed.
cargo test -p curvine-server --test load_job_submit_test	PASS	20 passed.
cargo tree -p curvine-data-transfer --features opendal-s3 --edges normal --depth 1	PASS	Direct dependency list excludes axum, curvine-web, rusqlite, mysql, and uuid.

Dependencies

• Related PRs: None
• Related issues: [BUG]: curvine-cli should not require RDMA runtime libraries on non-RDMA clients #1243
• Blocks / blocked by: None

[Copilot]

Pull request overview

This PR extracts server-side data transfer orchestration and shared UFS/RPC plumbing into a new curvine-data-transfer crate, then updates curvine-server, curvine-worker, curvine-master, and curvine-web to depend on (and re-export from) that crate while removing direct usage of the higher-level curvine-client facade.

Changes:

• Introduce crates/server/curvine-data-transfer containing transfer store backends, scheduler/service/handlers, worker load-task runner, UFS factory/manager, JobWorkerClient, and RpcContext.
• Replace curvine-client facade dependencies/usages with lower-level crates (curvine-client-core, curvine-job-client, curvine-unified-fs) across server/worker/master/web.
• Keep existing module paths stable via thin pub use ...::* re-exports (e.g., curvine_server::transfer::*, curvine_worker::worker::task::*).

Reviewed changes

Copilot reviewed 35 out of 52 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
Cargo.toml	Add curvine-data-transfer to workspace members/default-members and workspace deps.
Cargo.lock	Record new crate + updated dependency graph after refactor.
crates/server/curvine-data-transfer/Cargo.toml	Define new crate, features, and dependencies.
crates/server/curvine-data-transfer/src/lib.rs	New crate root exporting common/transfer/worker + JobWorkerClient/RpcContext.
crates/server/curvine-data-transfer/src/job_worker_client.rs	Moved JobWorkerClient RPC wrapper for worker interactions.
crates/server/curvine-data-transfer/src/rpc_context.rs	Moved RpcContext used by server-side RPC handling/auditing.
crates/server/curvine-data-transfer/src/common/mod.rs	New common module root re-exporting UFS plumbing.
crates/server/curvine-data-transfer/src/common/ufs_client.rs	Moved UFS client implementation to the new crate.
crates/server/curvine-data-transfer/src/common/ufs_factory.rs	Moved UFS/worker client factory to the new crate.
crates/server/curvine-data-transfer/src/common/ufs_manager.rs	Moved mount/UFS manager to the new crate.
crates/server/curvine-data-transfer/src/worker/mod.rs	New worker module root.
crates/server/curvine-data-transfer/src/worker/task/mod.rs	Worker task module root (LoadTaskRunner/TaskManager/TaskStore/TaskContext).
crates/server/curvine-data-transfer/src/worker/task/load_task_runner.rs	Worker-side load execution updated to use core/job-client/unified-fs crates.
crates/server/curvine-data-transfer/src/worker/task/task_manager.rs	Worker task orchestration updated to new client crates.
crates/server/curvine-data-transfer/src/worker/task/task_store.rs	New task store (plus supersede/cancel semantics test).
crates/server/curvine-data-transfer/src/worker/task/task_context.rs	New task context/progress tracking.
crates/server/curvine-data-transfer/src/transfer/mod.rs	Transfer module root wiring stores/scheduler/service/handlers.
crates/server/curvine-data-transfer/src/transfer/store.rs	Define TransferStore trait and update structs.
crates/server/curvine-data-transfer/src/transfer/backend.rs	Backend wrapper with per-operation metrics and “store unavailable” detection.
crates/server/curvine-data-transfer/src/transfer/cluster_cache.rs	Cluster metadata cache updated to use curvine-client-core.
crates/server/curvine-data-transfer/src/transfer/job_snapshot.rs	Job mount snapshot resolution/loading logic.
crates/server/curvine-data-transfer/src/transfer/planner.rs	Planner updated to use curvine-client-core.
crates/server/curvine-data-transfer/src/transfer/service.rs	Transfer API/service implementation (submit/status/list/watch/report).
crates/server/curvine-data-transfer/src/transfer/scheduler.rs	Scheduler logic (lease, planning, dispatching, stale recovery, cancel).
crates/server/curvine-data-transfer/src/transfer/handler.rs	ORPC handler routing transfer RPCs to the service.
crates/server/curvine-data-transfer/src/transfer/router_handler.rs	Axum router for health/ready/metrics endpoints.
crates/server/curvine-data-transfer/src/transfer/transfer_server.rs	Transfer server wiring: store backend, cache, scheduler, RPC + web servers.
crates/server/curvine-data-transfer/src/transfer/memory_store.rs	In-memory TransferStore backend implementation.
crates/server/curvine-data-transfer/src/transfer/sqlite_store.rs	SQLite TransferStore backend implementation (moved into new crate).
crates/server/curvine-data-transfer/src/transfer/mysql_store.rs	MySQL TransferStore backend implementation (moved into new crate).
crates/server/curvine-data-transfer/src/transfer/metrics.rs	Transfer metrics registry + helpers.
crates/server/curvine-data-transfer/src/transfer/tests/planner_test.rs	Planner tests relocated under the new crate.
curvine-server/src/transfer/mod.rs	Replace in-crate transfer module with re-export from curvine-data-transfer.
curvine-server/src/common/mod.rs	Replace common UFS plumbing with re-export from curvine-data-transfer::common.
curvine-server/src/test/mini_cluster.rs	Switch CurvineFileSystem import to curvine-client-core.
curvine-server/tests/load_task_runner_fault_test.rs	Switch JobMasterClient import to curvine-job-client.
curvine-server/Cargo.toml	Replace curvine-client facade with direct deps + feature forwarding to data-transfer.
curvine-worker/src/worker/task/mod.rs	Replace worker task module with re-export from curvine-data-transfer.
curvine-worker/src/worker/replication/worker_replication_manager.rs	Switch imports from curvine-client to curvine-client-core.
curvine-worker/src/worker/block/master_client.rs	Switch imports from curvine-client to curvine-client-core.
curvine-worker/src/worker/block/block_actor.rs	Switch imports from curvine-client to curvine-client-core.
curvine-worker/src/lib.rs	Re-export UfsFactory/RpcContext from curvine-data-transfer.
curvine-worker/Cargo.toml	Replace curvine-client facade + curvine-master dependency with curvine-client-core + curvine-data-transfer.
curvine-web/src/router/load_handler.rs	Switch JobMasterClient import to curvine-job-client.
curvine-web/Cargo.toml	Replace curvine-client facade with curvine-job-client.
curvine-master/src/master/mod.rs	Re-export RpcContext from curvine-data-transfer.
curvine-master/src/master/journal/ufs_loader.rs	Switch MountValue import to curvine-unified-fs.
curvine-master/src/master/job/mod.rs	Re-export JobWorkerClient from curvine-data-transfer.
curvine-master/src/master/job/job_runner.rs	Switch MountValue import to curvine-unified-fs.
curvine-master/src/master/job/job_manager.rs	Switch MountValue import to curvine-unified-fs.
curvine-master/src/common/mod.rs	Re-export UfsFactory from curvine-data-transfer.
curvine-master/Cargo.toml	Replace curvine-client facade with curvine-data-transfer + curvine-unified-fs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.