Is your feature request related to a problem?
During block replication under simulated worker failure, a master raft node can receive or install a 0-byte snapshot while local metadata already contains files. journal_loader refuses the empty snapshot and the master process aborts instead of recovering or rejecting the snapshot safely.
Master logs refusing to apply empty snapshot over filesystem with 4 files, then journal-writer/channel closed and the master aborts; the replication simulated-worker-failure test fails.
Reproduction:
- Check out Curvine commit 54185ed.
- Run cargo nextest run -p curvine-tests test_replication_with_simulated_worker_failure.
- Observe master abort with refusing to apply empty snapshot over non-empty filesystem.
Impact
Multi-master clusters can lose a master during replication/failover scenarios, breaking availability and replication tests.
Describe the solution you'd like
An empty snapshot must not tear down a master that already has non-empty filesystem state. The node should reject the snapshot safely and remain available for replication.
In journal/raft snapshot apply, reject or safely ignore empty snapshots when local state is non-empty without aborting the master process; add a regression around simulated worker failure under multi-master election.
Additional context
Seen on daily profile with cleanup passed. Distinct from older FileNotFound symptom on the same test name. Related open discussion exists around raft hard-state safety (#1368) but the decisive error here is empty snapshot apply. Confidence medium pending confirmation the empty snapshot is produced by product code rather than test teardown races.
Root-cause hypothesis: Root-cause hypothesis: during replication_test simulated worker failure, a master raft node attempts to apply a 0-byte snapshot over a non-empty filesystem and journal_loader refuses, aborting the master and failing the test.
Evidence summary: test_replication_with_simulated_worker_failure log shows raft_node stop: refusing to apply empty snapshot ... (0 bytes) over filesystem with 4 files and 0 dirs(journal_loader.rs:516), then journal-writer/channel closed and master abort.
Related issues: #1207
Acceptance / test gaps:
- test_replication_with_simulated_worker_failure passes without master abort
- Empty snapshot apply against non-empty state does not terminate the master process
Is your feature request related to a problem?
During block replication under simulated worker failure, a master raft node can receive or install a 0-byte snapshot while local metadata already contains files. journal_loader refuses the empty snapshot and the master process aborts instead of recovering or rejecting the snapshot safely.
Master logs refusing to apply empty snapshot over filesystem with 4 files, then journal-writer/channel closed and the master aborts; the replication simulated-worker-failure test fails.
Reproduction:
Impact
Multi-master clusters can lose a master during replication/failover scenarios, breaking availability and replication tests.
Describe the solution you'd like
An empty snapshot must not tear down a master that already has non-empty filesystem state. The node should reject the snapshot safely and remain available for replication.
In journal/raft snapshot apply, reject or safely ignore empty snapshots when local state is non-empty without aborting the master process; add a regression around simulated worker failure under multi-master election.
Additional context
Seen on daily profile with cleanup passed. Distinct from older FileNotFound symptom on the same test name. Related open discussion exists around raft hard-state safety (#1368) but the decisive error here is empty snapshot apply. Confidence medium pending confirmation the empty snapshot is produced by product code rather than test teardown races.
Root-cause hypothesis: Root-cause hypothesis: during replication_test simulated worker failure, a master raft node attempts to apply a 0-byte snapshot over a non-empty filesystem and journal_loader refuses, aborting the master and failing the test.
Evidence summary: test_replication_with_simulated_worker_failure log shows raft_node stop: refusing to apply empty snapshot ... (0 bytes) over filesystem with 4 files and 0 dirs(journal_loader.rs:516), then journal-writer/channel closed and master abort.
Related issues: #1207
Acceptance / test gaps: