Skip to content

Commit 69e1110

Browse files
zacharysierakowskizacharysierakowski
committed
integration tests
1 parent 9e752b8 commit 69e1110

7 files changed

Lines changed: 266 additions & 6 deletions

File tree

‎localtests/move-tables/atomic-multi-table-cutover/create.sql‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
-- Atomic multi-table cutover test.
2+
--
3+
-- Two tables of identical shape with a correlation column `txn_id`. The test
4+
-- workload (see test.sh) commits transactions that write the SAME txn_id into
5+
-- BOTH tables. Because all migrated tables are renamed in a single atomic
6+
-- `RENAME TABLE t1 TO ..., t2 TO ...` at cutover, every such transaction lands on
7+
-- the target entirely or not at all -- so the target tables must hold exactly the
8+
-- same set of txn_ids. A regression to per-table sequential RENAME would split a
9+
-- boundary transaction and leave an orphan.
10+
11+
drop table if exists gh_ost_test;
12+
create table gh_ost_test (
13+
id int(11) NOT NULL AUTO_INCREMENT,
14+
txn_id int(11) NOT NULL,
15+
payload varchar(32) NOT NULL,
16+
PRIMARY KEY (id),
17+
KEY txn_ix (txn_id)
18+
);
19+
20+
insert into gh_ost_test (txn_id, payload) values
21+
(0, 'seed'), (0, 'seed'), (0, 'seed'), (0, 'seed'), (0, 'seed');
22+
23+
drop table if exists gh_ost_test_other;
24+
create table gh_ost_test_other (
25+
id int(11) NOT NULL AUTO_INCREMENT,
26+
txn_id int(11) NOT NULL,
27+
payload varchar(32) NOT NULL,
28+
PRIMARY KEY (id),
29+
KEY txn_ix (txn_id)
30+
);
31+
32+
insert into gh_ost_test_other (txn_id, payload) values
33+
(0, 'seed'), (0, 'seed'), (0, 'seed'), (0, 'seed'), (0, 'seed');

‎localtests/move-tables/atomic-multi-table-cutover/tables.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
gh_ost_test
2+
gh_ost_test_other

‎localtests/move-tables/atomic-multi-table-cutover/test.sh‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
#!/bin/bash
2+
# Atomic multi-table cutover test.
3+
#
4+
# A workload commits transactions that each write the SAME txn_id into BOTH
5+
# migrated tables, in a tight loop, right up to the cutover. Because gh-ost
6+
# renames every migrated table in ONE atomic `RENAME TABLE t1 TO ..., t2 TO ...`,
7+
# each cross-table transaction lands on the target entirely or not at all.
8+
#
9+
# Verification is deterministic (final-state set + checksum comparison, no timing
10+
# assertions): the set of txn_ids in target gh_ost_test must exactly equal the set
11+
# in target gh_ost_test_other. A regression from the atomic multi-table RENAME to
12+
# a per-table sequential RENAME splits a boundary transaction across the two
13+
# tables and leaves an orphan txn_id, failing this test.
14+
15+
database=test
16+
17+
build_binary
18+
build_ghost_command
19+
20+
######################################################################################################
21+
### Drive cross-table transactions, then cut over while they are still committing
22+
######################################################################################################
23+
24+
echo "⚙️ Starting cross-table transaction workload..."
25+
26+
# Each iteration commits one transaction touching BOTH tables with a shared
27+
# txn_id. The loop runs with no delay until the tables are renamed at cutover
28+
# (the INSERT then errors and the loop exits), so cross-table transactions are
29+
# committing continuously while the cutover happens. Note: we do NOT (and cannot,
30+
# from a shell) control whether a transaction is literally mid-commit at the
31+
# RENAME instant -- that is timing-dependent. Correctness is asserted
32+
# deterministically on the final state below (no orphaned txn_id across the pair
33+
# + per-table checksums), not on hitting that instant.
34+
(
35+
n=1
36+
while true; do
37+
mysql-exec source primary $database -e \
38+
"START TRANSACTION; \
39+
INSERT INTO gh_ost_test (txn_id, payload) VALUES ($n, 'a'); \
40+
INSERT INTO gh_ost_test_other (txn_id, payload) VALUES ($n, 'b'); \
41+
COMMIT;" 2>/dev/null || break
42+
n=$((n + 1))
43+
done
44+
) &
45+
workload_pid=$!
46+
47+
# Remove the postpone flag so cutover proceeds while the workload is still
48+
# committing cross-table transactions. This is a best-effort time-based overlap,
49+
# not a guarantee that a transaction is mid-commit at the exact RENAME; the
50+
# atomicity guarantee is verified on the final target state below.
51+
(
52+
sleep 4
53+
echo "⏩ Sending unpostpone cutover"
54+
rm $postpone_cutover_flag_file &> /dev/null
55+
) &
56+
57+
echo > $test_logfile
58+
bash -c "$cmd" >> $test_logfile 2>&1
59+
ghost_result=$?
60+
61+
kill $workload_pid &> /dev/null
62+
63+
if [ $ghost_result -ne 0 ]; then
64+
echo "ERROR: gh-ost should have succeeded but did not. ($ghost_result)"
65+
return 1
66+
fi
67+
68+
echo -e "\n\n\n\n\n"
69+
70+
######################################################################################################
71+
### Validate atomicity + data integrity (read primaries to avoid replication lag)
72+
######################################################################################################
73+
74+
echo "⚙️ Validating atomic multi-table cutover..."
75+
76+
# Sanity: the workload must have landed cross-table rows on the target, otherwise
77+
# the atomicity assertion below would be vacuously true.
78+
paired=$(mysql-exec target primary $database -sNe "SELECT COUNT(*) FROM gh_ost_test WHERE txn_id > 0;")
79+
if [ -z "$paired" ] || [ "$paired" -lt 1 ]; then
80+
echo "ERROR: workload produced no cross-table rows on target; test would be vacuous."
81+
return 1
82+
fi
83+
84+
# Atomicity invariant: every cross-table transaction landed entirely or not at
85+
# all, i.e. the txn_id sets match across the two target tables (no orphans).
86+
orphans_a=$(mysql-exec target primary $database -sNe \
87+
"SELECT COUNT(*) FROM gh_ost_test t1 WHERE t1.txn_id > 0 \
88+
AND NOT EXISTS (SELECT 1 FROM gh_ost_test_other t2 WHERE t2.txn_id = t1.txn_id);")
89+
orphans_b=$(mysql-exec target primary $database -sNe \
90+
"SELECT COUNT(*) FROM gh_ost_test_other t2 WHERE t2.txn_id > 0 \
91+
AND NOT EXISTS (SELECT 1 FROM gh_ost_test t1 WHERE t1.txn_id = t2.txn_id);")
92+
93+
if [ "$orphans_a" != "0" ] || [ "$orphans_b" != "0" ]; then
94+
echo "ERROR: non-atomic cutover: ${orphans_a} txn_id(s) in gh_ost_test missing from gh_ost_test_other; ${orphans_b} the other way."
95+
return 1
96+
fi
97+
98+
# Full data integrity: each migrated table matches its source rollback handle.
99+
for table_name in gh_ost_test gh_ost_test_other; do
100+
src_checksum=$(mysql-exec source primary $database -ss -e "SELECT * FROM _${table_name}_del ORDER BY id" | md5sum)
101+
dst_checksum=$(mysql-exec target primary $database -ss -e "SELECT * FROM ${table_name} ORDER BY id" | md5sum)
102+
if [ "$src_checksum" != "$dst_checksum" ]; then
103+
echo "ERROR: checksum mismatch on ${table_name} between source _del and target."
104+
return 1
105+
fi
106+
done
107+
108+
echo "✅ Atomic multi-table cutover validated: ${paired} cross-table transactions, no orphans, checksums match."

‎localtests/move-tables/multiple-three-concurrent-writes/create.sql‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
-- Three tables with distinct schemas, primary-key types, and row counts. This
2+
-- exercises the multi-table move-tables path at its widest: per-table
3+
-- runtime state, per-table query builders, interleaved row copy where the tables
4+
-- finish at different times, and a single atomic multi-table RENAME at cutover.
5+
--
6+
-- These three tables are the canonical superset used by the manual harness
7+
-- (script/move-tables/setup, reset, insert-source-primary-loop). The `single`
8+
-- and `multiple-two` localtest fixtures move subsets of them.
9+
10+
drop table if exists gh_ost_test;
11+
create table gh_ost_test (
12+
id bigint(20) NOT NULL AUTO_INCREMENT,
13+
column1 int(11) NOT NULL,
14+
column2 smallint(5) unsigned NOT NULL,
15+
column3 mediumint(8) unsigned NOT NULL,
16+
column4 tinyint(3) unsigned NOT NULL,
17+
column5 int(11) NOT NULL,
18+
column6 int(11) NOT NULL,
19+
PRIMARY KEY (id),
20+
KEY c12_ix (column1, column2)
21+
) auto_increment=1;
22+
23+
insert into gh_ost_test values
24+
(NULL, 1001, 100, 500000, 10, 1700000001, 1700000002),
25+
(NULL, 1002, 200, 600000, 20, 1700000003, 1700000004),
26+
(NULL, 1003, 300, 700000, 30, 1700000005, 1700000006),
27+
(NULL, 1004, 400, 800000, 40, 1700000007, 1700000008),
28+
(NULL, 1005, 500, 900000, 50, 1700000009, 1700000010),
29+
(NULL, 1006, 600, 1000000, 60, 1700000011, 1700000012),
30+
(NULL, 1007, 700, 1100000, 70, 1700000013, 1700000014),
31+
(NULL, 1008, 800, 1200000, 80, 1700000015, 1700000016),
32+
(NULL, 1009, 900, 1300000, 90, 1700000017, 1700000018),
33+
(NULL, 1010, 1000, 1400000, 100, 1700000019, 1700000020),
34+
(NULL, 1011, 1100, 1500000, 110, 1700000021, 1700000022),
35+
(NULL, 1012, 1200, 1600000, 120, 1700000023, 1700000024),
36+
(NULL, 1013, 1300, 1700000, 130, 1700000025, 1700000026),
37+
(NULL, 1014, 1400, 1800000, 140, 1700000027, 1700000028),
38+
(NULL, 1015, 1500, 1900000, 150, 1700000029, 1700000030),
39+
(NULL, 1016, 1600, 2000000, 160, 1700000031, 1700000032),
40+
(NULL, 1017, 1700, 2100000, 170, 1700000033, 1700000034),
41+
(NULL, 1018, 1800, 2200000, 180, 1700000035, 1700000036),
42+
(NULL, 1019, 1900, 2300000, 190, 1700000037, 1700000038),
43+
(NULL, 1020, 2000, 2400000, 200, 1700000039, 1700000040),
44+
(NULL, 1021, 2100, 2500000, 210, 1700000041, 1700000042),
45+
(NULL, 1022, 2200, 2600000, 220, 1700000043, 1700000044),
46+
(NULL, 1023, 2300, 2700000, 230, 1700000045, 1700000046),
47+
(NULL, 1024, 2400, 2800000, 240, 1700000047, 1700000048),
48+
(NULL, 1025, 2500, 2900000, 250, 1700000049, 1700000050);
49+
50+
drop table if exists gh_ost_test_other;
51+
create table gh_ost_test_other (
52+
uid int(11) NOT NULL,
53+
name varchar(64) NOT NULL,
54+
amount decimal(10,2) NOT NULL,
55+
created_at datetime NOT NULL,
56+
PRIMARY KEY (uid),
57+
UNIQUE KEY name_uq (name)
58+
);
59+
60+
insert into gh_ost_test_other values
61+
(1, 'alpha', 10.50, '2024-01-01 10:00:00'),
62+
(2, 'bravo', 20.75, '2024-01-02 11:00:00'),
63+
(3, 'charlie', 30.00, '2024-01-03 12:00:00'),
64+
(4, 'delta', 40.25, '2024-01-04 13:00:00'),
65+
(5, 'echo', 50.50, '2024-01-05 14:00:00'),
66+
(6, 'foxtrot', 60.75, '2024-01-06 15:00:00'),
67+
(7, 'golf', 70.00, '2024-01-07 16:00:00'),
68+
(8, 'hotel', 80.25, '2024-01-08 17:00:00'),
69+
(9, 'india', 90.50, '2024-01-09 18:00:00'),
70+
(10, 'juliet', 100.75, '2024-01-10 19:00:00'),
71+
(11, 'kilo', 110.00, '2024-01-11 20:00:00'),
72+
(12, 'lima', 120.25, '2024-01-12 21:00:00');
73+
74+
drop table if exists gh_ost_test_third;
75+
create table gh_ost_test_third (
76+
code varchar(32) NOT NULL,
77+
label varchar(128) NOT NULL,
78+
score double NOT NULL,
79+
updated_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
80+
PRIMARY KEY (code),
81+
KEY score_ix (score)
82+
);
83+
84+
insert into gh_ost_test_third (code, label, score) values
85+
('code_1', 'label_1', 1.5),
86+
('code_2', 'label_2', 2.5),
87+
('code_3', 'label_3', 3.5),
88+
('code_4', 'label_4', 4.5),
89+
('code_5', 'label_5', 5.5),
90+
('code_6', 'label_6', 6.5),
91+
('code_7', 'label_7', 7.5),
92+
('code_8', 'label_8', 8.5);

‎localtests/move-tables/multiple-three-concurrent-writes/on_test.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
# Three-table move with sustained DML on all three tables during the
4+
# copy. insert-source-primary-loop auto-detects every seeded fixture
5+
# (gh_ost_test, gh_ost_test_other, gh_ost_test_third) and writes to all of them,
6+
# so each migrated table sees concurrent inserts while gh-ost copies and drains.
7+
# The harness then validates per-table structure + content checksums (source
8+
# `_<table>_del` vs target), which deterministically proves every concurrent
9+
# write was captured on the target.
10+
DATABASE=test script/move-tables/insert-source-primary-loop 100 0.01 100 &
11+
sleep 5 && kill $!

‎localtests/move-tables/multiple-three-concurrent-writes/tables.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
gh_ost_test
2+
gh_ost_test_other
3+
gh_ost_test_third

‎script/move-tables/README.md‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -93,13 +93,24 @@ script/move-tables/teardown
9393

9494
The same fixtures back the CI integration tests, run via
9595
`localtests/move-tables-test.sh [filter]`. Each test directory under
96-
`localtests/move-tables/` is self-contained (its own `create.sql` + `tables.txt`):
97-
98-
- `single` — moves 1 table (`gh_ost_test`)
99-
- `multi` — moves 2 tables (`gh_ost_test`, `gh_ost_test_other`)
100-
- `three` — moves 3 tables (`gh_ost_test`, `gh_ost_test_other`, `gh_ost_test_third`)
96+
`localtests/move-tables/` is self-contained (its own `create.sql` + `tables.txt`,
97+
plus an optional `on_test.sh` for concurrent workload or `test.sh` for a fully
98+
custom scenario):
99+
100+
- `single` — moves 1 table, idle source
101+
- `single-concurrent-writes` — moves 1 table with sustained DML during copy
102+
- `single-with-hooks` — moves 1 table and asserts the hook env vars
103+
- `multiple-two` — moves 2 tables, idle source
104+
- `multiple-three` — moves 3 tables, idle source
105+
- `multiple-three-concurrent-writes` — moves 3 tables with sustained DML on all three
106+
- `atomic-multi-table-cutover` — moves 2 tables while committing cross-table
107+
transactions up to cutover; asserts the atomic multi-table RENAME leaves no
108+
orphaned rows across the pair
109+
- `resume-panic-on-row-copy`, `resume-panic-before-drain-complete`,
110+
`resume-panic-before-on-success-hook` — crash mid-run via a failpoint, then
111+
`--resume` to completion
101112

102113
Run a single scenario by name, e.g.:
103114
```bash
104-
localtests/move-tables-test.sh three
115+
localtests/move-tables-test.sh multiple-three
105116
```

0 commit comments

Comments
 (0)