eventservice: scan retry can lose spilled inserts from large UK updates #6415

What did you do?

Found a potential silent data-loss path while reviewing #6392, the release-8.5 backport of #5511. It also exists on master.

This is a code-review finding, not an observed production incident or an executed fault-injection reproducer. The SQL below needs the specified fault/timing to trigger the bug; SQL alone is not sufficient.

Use new-architecture TiCDC with a MySQL/TiDB sink, transaction-atomicity=none, and the default 1 MiB large-transaction scan threshold. Prepare upstream data and wait until all three rows are replicated:

CREATE DATABASE spill_retry_demo;
USE spill_retry_demo;
CREATE TABLE t (
  id INT PRIMARY KEY,
  uk INT NOT NULL,
  payload MEDIUMTEXT NOT NULL,
  UNIQUE KEY uk_idx (uk)
);
INSERT INTO t VALUES
  (1, 1, REPEAT('a', 1048576)),
  (2, 2, REPEAT('a', 1048576)),
  (3, 3, REPEAT('a', 1048576));

Then execute one transaction:

BEGIN;
UPDATE spill_retry_demo.t
SET uk = uk + 100, payload = REPEAT('b', 1048576);
COMMIT;

Each row's raw KV size exceeds the scan threshold. A UK update is split into delete + deferred insert. Consider this sequence:

1. The first scan emits delete(id=1), spills insert(id=1, uk=101), and the broker saves row cursor P1 after row 1.

2. The next original-row scan attempts to append row 2's insert to spill. Make that write fail once, while keeping the process and dispatcher/epoch alive. A concrete possible cause is insufficient disk space or quota; an illustrative underlying error is:

   write <data-dir>/eventservice/eventservice-large-txn-insert-<id>.spill: no space left on device
   

   This is an example, not a captured log. The write error propagates as ErrSpillFileOp to the scanner. For deterministic testing, inject a one-shot append error after confirming P1 publication; do not fill a production disk.

3. The current error handler deletes the entire spill, including row 1's insert. The broker keeps P1 unchanged.

4. After the I/O condition clears, the same dispatcher retries from after P1. Rows 2 and 3 can complete, but row 1's insert is neither retained nor regenerated. Deleting the spill can itself free space, potentially allowing the retry to succeed.

Do not issue further updates to these rows before verification, since a later successful UK update could restore a missing row.

What did you expect to see?

After recovery and checkpoint catch-up, this query on both sides should return all three rows:

SELECT id, uk FROM spill_retry_demo.t ORDER BY id;
-- 1  101
-- 2  102
-- 3  103

A failed scan must retain the deferred inserts belonging to already-published fragments.

What did you see instead?

Under the sequence above, the downstream can be missing (1, 101) even though replication resumes.

Root cause: scanner error handling unconditionally calls cleanupLargeTxnState() on original-scan or iterator Close errors. The broker returns on error without rolling back the previously published row cursor. The cursor and spill therefore no longer describe the same progress.

Normal fragment interruption is not an error. Restart/reset recovery is also different: this scenario requires an in-place retry on the same live dispatcher.

Suggested fix

Use a scanner-local per-attempt checkpoint (Plan B):

• Before an original-row scan, save the existing spill's byte offset and phase/state metadata.
• On scan or iterator Close failure, truncate only this attempt's appended tail and restore the entry phase; preserve earlier fragments' inserts. If this attempt created the spill, clean up only that new state.
• If rollback fails, retain the original recovery target and complete rollback before scanning/appending/draining again. Keep existing drain-phase rollback behavior.
• Add deterministic regressions for first-fragment success followed by append/Close failure, including partial writes and rollback failure; assert no missing or duplicate inserts after retry.

Versions of the cluster

• Affected code verified on master 006b1df37a241e1e960d36c4395c7f01d76254a2 (includes eventservice: split large scanned transactions #5511).
• Also present in the unmerged release-8.5 backport eventservice: split large scanned transactions (#5511) #6392 at ec03b549f933b922e2d626fdb0c10ef18b72392c. This is not a claim that an already-released v8.5 build contains this backport.
• Upstream TiDB/TiKV runtime versions: not applicable to this static review; no cluster reproduction has been run for this issue.