Skip to content

Commit 4da92e1

Browse files
tarofacetaroface
committed
address review comments
1 parent 5357ac2 commit 4da92e1

13 files changed

+135
-127
lines changed

‎src/current/_includes/molt/fetch-data-load-output.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@
6666
~~~ json
6767
{"level":"info","type":"summary","fetch_id":"f5cb422f-4bb4-4bbd-b2ae-08c4d00d1e7c","num_tables":3,"tables":["migration_schema.employees","migration_schema.payments","migration_schema.payments"],"cdc_cursor":"4c658ae6-e8ad-11ef-8449-0242ac140006:1-29","net_duration_ms":6752.847625,"net_duration":"000h 00m 06s","time":"2024-03-18T12:30:37-04:00","message":"fetch complete"}
6868
~~~
69+
70+
This message includes a `cdc_cursor` value. You must set `--defaultGTIDSet` this value when starting [`replication-only` mode](#replicate-changes-to-cockroachdb).
6971
</section>
7072

7173
<section class="filter-content" markdown="1" data-scope="oracle">
@@ -76,7 +78,7 @@
7678

7779
{% if page.name == "migrate-data-load-replicate-only.md" %}
7880
<section class="filter-content" markdown="1" data-scope="oracle">
79-
The following message contains the appropriate `--backfillFromSCN` and `--scn` values to use in [`replication-only` mode](#replicate-changes-to-cockroachdb):
81+
The following message contains the appropriate `--backfillFromSCN` and `--scn` values to set when [starting`replication-only` mode](#replicate-changes-to-cockroachdb):
8082

8183
{% include_cached copy-clipboard.html %}
8284
~~~

‎src/current/_includes/molt/fetch-replicator-flags.md

Lines changed: 41 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,13 @@ In the `molt fetch` command, use `--replicator-flags` to pass options to the inc
88
</section>
99

1010
<section class="filter-content" markdown="1" data-scope="mysql">
11-
| Flag | Description |
12-
|--------------------|----------------------------------------------------------------------------------------------------------------|
13-
| `--defaultGTIDSet` | **Required.** Default GTID set for changefeed. |
14-
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
15-
16-
Replication from MySQL requires `--defaultGTIDSet`, which specifies the MySQL GTID set. To find your GTID record, run the following command on MySQL:
17-
18-
{% include_cached copy-clipboard.html %}
19-
~~~ sql
20-
SELECT source_uuid, min(interval_start), max(interval_end)
21-
FROM mysql.gtid_executed
22-
GROUP BY source_uuid;
23-
~~~
11+
| Flag | Description |
12+
|--------------------|-------------------------------------------------------------------------------------------------------------------------------------|
13+
| `--defaultGTIDSet` | **Required.** Default GTID set for changefeed. |
14+
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
15+
| `--userscript` | Path to a userscript that enables table filtering from MySQL sources. Refer to [Table filter userscript](#table-filter-userscript). |
16+
17+
Replication from MySQL requires `--defaultGTIDSet`, which sets the starting GTID for replication. You can find this value in the `cdc_cursor` field of the `fetch complete` message after the [initial data load](#load-data-into-cockroachdb) completes.
2418
</section>
2519

2620
<section class="filter-content" markdown="1" data-scope="oracle">
@@ -31,30 +25,11 @@ GROUP BY source_uuid;
3125
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
3226
| `--userscript` | Path to a userscript that enables table filtering from Oracle sources. Refer to [Table filter userscript](#table-filter-userscript). |
3327

34-
Replication from Oracle requires `--scn` and `--backfillFromSCN`, which specify the snapshot SCN and the earliest active transaction SCN, respectively. To find these values, refer to the message `replication-only mode should include the following replicator flags` in the [data load](#load-data-into-cockroachdb) output.
35-
</section>
36-
37-
{% elsif page.name == "migrate-failback.md" %}
38-
| Flag | Description |
39-
|--------------------|--------------------------------------------------------------------------------------------------------------------------------------|
40-
| `--stagingSchema` | **Required.** Staging schema name for the changefeed checkpoint table. |
41-
| `--tlsCertificate` | Path to the server TLS certificate for the webhook sink. Refer to [Secure failback for changefeed](#secure-changefeed-for-failback). |
42-
| `--tlsPrivateKey` | Path to the server TLS private key for the webhook sink. Refer to [Secure failback for changefeed](#secure-changefeed-for-failback). |
43-
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
44-
45-
- Failback requires `--stagingSchema`, which specifies the staging schema name used as a checkpoint. MOLT Fetch [logs the staging schema name]({% link molt/migrate-data-load-replicate-only.md %}#replicate-changes-to-cockroachdb) when it starts replication:
46-
47-
~~~ shell
48-
staging database name: _replicator_1749699789613149000
49-
~~~
50-
51-
- When configuring a [secure changefeed](#secure-changefeed-for-failback) for failback, you **must** include `--tlsCertificate` and `--tlsPrivateKey`, which specify the paths to the server certificate and private key for the webhook sink connection.
52-
53-
<section class="filter-content" markdown="1" data-scope="oracle">
54-
- Oracle migrations using `--table-filter` must also include `--userscript`. Refer to [Table filter userscript]({% link molt/migrate-data-load-replicate-only.md %}#table-filter-userscript).
28+
Replication from Oracle requires `--scn` and `--backfillFromSCN`, which specify the snapshot SCN and the earliest active transaction SCN, respectively. You can find these values in the message `replication-only mode should include the following replicator flags` after the [initial data load](#load-data-into-cockroachdb) completes.
5529
</section>
5630

5731
{% elsif page.name == "migrate-replicate-only.md" %}
32+
5833
| Flag | Description |
5934
|-------------------|----------------------------------------------------------------------------------------------------------------|
6035
| `--stagingSchema` | **Required.** Staging schema name for the changefeed checkpoint table. |
@@ -66,23 +41,52 @@ Resuming replication requires `--stagingSchema`, which specifies the staging sch
6641
staging database name: _replicator_1749699789613149000
6742
~~~
6843

44+
<section class="filter-content" markdown="1" data-scope="mysql">
45+
{{site.data.alerts.callout_info}}
46+
When using `--table-filter`, you must also include `--userscript`. Refer to [Table filter userscript]({% link molt/migrate-data-load-replicate-only.md %}?filters=mysql#table-filter-userscript).
47+
{{site.data.alerts.end}}
48+
</section>
49+
6950
<section class="filter-content" markdown="1" data-scope="oracle">
7051
{{site.data.alerts.callout_info}}
71-
Oracle migrations using `--table-filter` must also include `--userscript`. Refer to [Table filter userscript]({% link molt/migrate-data-load-replicate-only.md %}#table-filter-userscript).
52+
When using `--table-filter`, you must also include `--userscript`. Refer to [Table filter userscript]({% link molt/migrate-data-load-replicate-only.md %}?filters=oracle#table-filter-userscript).
7253
{{site.data.alerts.end}}
7354
</section>
7455

75-
{% elsif page.name == "migrate-data-load-and-replication" %}
56+
{% elsif page.name == "migrate-data-load-and-replication.md" %}
57+
7658
| Flag | Description |
7759
|-----------------|----------------------------------------------------------------------------------------------------------------|
7860
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
7961

62+
<section class="filter-content" markdown="1" data-scope="mysql">
63+
{{site.data.alerts.callout_info}}
64+
When using `--table-filter`, you must also include `--userscript`. Refer to [Table filter userscript](#table-filter-userscript).
65+
{{site.data.alerts.end}}
66+
</section>
67+
8068
<section class="filter-content" markdown="1" data-scope="oracle">
8169
{{site.data.alerts.callout_info}}
82-
Oracle migrations using `--table-filter` must also include `--userscript`. Refer to [Table filter userscript](#table-filter-userscript).
70+
When using `--table-filter`, you must also include `--userscript`. Refer to [Table filter userscript](#table-filter-userscript).
8371
{{site.data.alerts.end}}
8472
</section>
8573

74+
{% elsif page.name == "migrate-failback.md" %}
75+
| Flag | Description |
76+
|--------------------|--------------------------------------------------------------------------------------------------------------------------------------|
77+
| `--stagingSchema` | **Required.** Staging schema name for the changefeed checkpoint table. |
78+
| `--tlsCertificate` | Path to the server TLS certificate for the webhook sink. Refer to [Secure failback for changefeed](#secure-changefeed-for-failback). |
79+
| `--tlsPrivateKey` | Path to the server TLS private key for the webhook sink. Refer to [Secure failback for changefeed](#secure-changefeed-for-failback). |
80+
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
81+
82+
- Failback requires `--stagingSchema`, which specifies the staging schema name used as a checkpoint. MOLT Fetch [logs the staging schema name]({% link molt/migrate-data-load-replicate-only.md %}#replicate-changes-to-cockroachdb) when it starts replication:
83+
84+
~~~ shell
85+
staging database name: _replicator_1749699789613149000
86+
~~~
87+
88+
- When configuring a [secure changefeed](#secure-changefeed-for-failback) for failback, you **must** include `--tlsCertificate` and `--tlsPrivateKey`, which specify the paths to the server certificate and private key for the webhook sink connection.
89+
8690
{% else %}
8791
| Flag | Description |
8892
|-----------------|----------------------------------------------------------------------------------------------------------------|

‎src/current/_includes/molt/fetch-schema-table-filtering.md

Lines changed: 9 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -16,49 +16,14 @@ When migrating from Oracle, you **must** include `--schema-filter` to name an Or
1616
~~~
1717
--schema-filter 'migration_schema'
1818
~~~
19+
</section>
1920

20-
#### Table filter userscript
21+
{% if page.name != "migrate-bulk-load.md" %}
22+
<section class="filter-content" markdown="1" data-scope="mysql">
23+
{% include molt/fetch-table-filter-userscript.md %}
24+
</section>
2125

22-
When migrating only a subset of tables using `--table-filter`, you **must** supply a userscript that filters change events to those tables. For example, save the following as `table_filter.ts`:
23-
24-
~~~ ts
25-
import * as api from "replicator@v1";
26-
27-
// List the source tables (matching case) to include in replication
28-
const allowedTables = ["EMPLOYEES", "PAYMENTS", "ORDERS"];
29-
30-
// Update this to your target CockroachDB database and schema name
31-
api.configureSource("defaultdb.migration_schema", {
32-
dispatch: (doc: Document, meta: Document): Record<Table, Document[]> | null => {
33-
// Replicate only if the table matches one of the allowed tables
34-
if (allowedTables.includes(meta.table)) {
35-
let ret: Record<Table, Document[]> = {};
36-
ret[meta.table] = [doc];
37-
return ret;
38-
}
39-
// Ignore all other tables
40-
return null;
41-
},
42-
deletesTo: (doc: Document, meta: Document): Record<Table, Document[]> | null => {
43-
// Optionally filter deletes the same way
44-
if (allowedTables.includes(meta.table)) {
45-
let ret: Record<Table, Document[]> = {};
46-
ret[meta.table] = [doc];
47-
return ret;
48-
}
49-
return null;
50-
},
51-
});
52-
~~~
53-
54-
Then pass the userscript to MOLT Fetch using `--replicator-flags`:
55-
56-
~~~ shell
57-
molt fetch \
58-
--source $SOURCE \
59-
--target $TARGET \
60-
--schema-filter schema_to_migrate \
61-
--table-filter 'employees|payments|orders' \
62-
--replicator-flags "--userscript table_filter.ts"
63-
~~~
64-
</section>
26+
<section class="filter-content" markdown="1" data-scope="oracle">
27+
{% include molt/fetch-table-filter-userscript.md %}
28+
</section>
29+
{% endif %}

‎src/current/_includes/molt/fetch-table-filter-userscript.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
#### Table filter userscript
2+
3+
When migrating only a subset of tables using `--table-filter`, you **must** supply a userscript that filters change events to those tables. For example, save the following as `table_filter.ts`:
4+
5+
~~~ ts
6+
import * as api from "replicator@v1";
7+
8+
// List the source tables (matching source names) to include in replication
9+
const allowedTables = ["EMPLOYEES", "PAYMENTS", "ORDERS"];
10+
11+
// Update this to your target CockroachDB database and schema name
12+
api.configureSource("defaultdb.migration_schema", {
13+
dispatch: (doc: Document, meta: Document): Record<Table, Document[]> | null => {
14+
// Replicate only if the table matches one of the allowed tables
15+
if (allowedTables.includes(meta.table)) {
16+
let ret: Record<Table, Document[]> = {};
17+
ret[meta.table] = [doc];
18+
return ret;
19+
}
20+
// Ignore all other tables
21+
return null;
22+
},
23+
deletesTo: (doc: Document, meta: Document): Record<Table, Document[]> | null => {
24+
// Optionally filter deletes the same way
25+
if (allowedTables.includes(meta.table)) {
26+
let ret: Record<Table, Document[]> = {};
27+
ret[meta.table] = [doc];
28+
return ret;
29+
}
30+
return null;
31+
},
32+
});
33+
~~~
34+
35+
Pass the userscript to MOLT Fetch with the `--userscript` [replication flag](#replication-flags):
36+
37+
~~~
38+
--replicator-flags "--userscript table_filter.ts"
39+
~~~

‎src/current/_includes/molt/molt-oracle-limitations.md renamed to ‎src/current/_includes/molt/molt-limitations.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,16 @@
1-
## Limitations
1+
{% if page.name != "migrate-bulk-load.md" %}
2+
### Limitations
23

4+
- Running DDL on the source or target while replication is in progress can cause replication failures.
5+
{% endif %}
6+
7+
<section class="filter-content" markdown="1" data-scope="oracle">
38
- Migrations must be performed from a single Oracle schema. You **must** include `--schema-filter` so that MOLT Fetch only loads data from the specified schema. Refer to [Schema and table filtering](#schema-and-table-filtering).
49
- Specifying `--table-filter` is also strongly recommended to ensure that only necessary tables are migrated from the Oracle schema. A userscript is required to use `--table-filter` with an Oracle source.
510

611
{% if page.name != "bulk-load.md" %}
712
- Replication will not work for tables or column names exceeding 30 characters. This is a [limitation of Oracle LogMiner](https://docs.oracle.com/en/database/oracle/oracle-database/21/sutil/oracle-logminer-utility.html#GUID-7594F0D7-0ACD-46E6-BD61-2751136ECDB4).
813

9-
- Running DDL on the source or target while replication is in progress can cause replication failures.
10-
1114
- Oracle LogMiner does not support the following data types:
1215
- User-defined types (UDTs)
1316
- Nested tables
@@ -19,4 +22,5 @@
1922
- If you are migrating LOB columns from Oracle 12c, use [AWS DMS Binary Reader](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html#CHAP_Source.Oracle.CDC) instead of LogMiner. Oracle LogMiner does not support LOB replication in 12c.
2023
{% endif %}
2124

22-
- Oracle advises against `LONG RAW` columns and [recommends converting them to `BLOB`](https://www.orafaq.com/wiki/LONG_RAW#History). `LONG RAW` can only store binary values up to 2GB, and only one `LONG RAW` column per table is supported.
25+
- Oracle advises against `LONG RAW` columns and [recommends converting them to `BLOB`](https://www.orafaq.com/wiki/LONG_RAW#History). `LONG RAW` can only store binary values up to 2GB, and only one `LONG RAW` column per table is supported.
26+
</section>

‎src/current/_includes/molt/molt-setup.md

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,3 @@
1-
## Before you begin
2-
3-
- Create a CockroachDB [{{ site.data.products.cloud }}]({% link cockroachcloud/create-your-cluster.md %}) or [{{ site.data.products.core }}]({% link {{ site.current_cloud_version }}/install-cockroachdb-mac.md %}) cluster.
4-
- Install the [MOLT (Migrate Off Legacy Technology)]({% link releases/molt.md %}#installation) tools.
5-
- Review the MOLT Fetch [best practices]({% link molt/molt-fetch.md %}#best-practices).
6-
- Review [Migration Strategy]({% link molt/migration-strategy.md %}).
7-
81
<div class="filters filters-big clearfix">
92
<button class="filter-button" data-scope="postgres">PostgreSQL</button>
103
<button class="filter-button" data-scope="mysql">MySQL</button>
@@ -15,12 +8,21 @@
158
{{site.data.alerts.callout_info}}
169
{% include feature-phases/preview.md %}
1710
{{site.data.alerts.end}}
11+
</section>
1812

19-
{% include molt/oracle-migration-prerequisites.md %}
13+
## Before you begin
2014

21-
{% include molt/molt-oracle-limitations.md %}
15+
- Create a CockroachDB [{{ site.data.products.cloud }}]({% link cockroachcloud/create-your-cluster.md %}) or [{{ site.data.products.core }}]({% link {{ site.current_cloud_version }}/install-cockroachdb-mac.md %}) cluster.
16+
- Install the [MOLT (Migrate Off Legacy Technology)]({% link releases/molt.md %}#installation) tools.
17+
- Review the MOLT Fetch [best practices]({% link molt/molt-fetch.md %}#best-practices).
18+
- Review [Migration Strategy]({% link molt/migration-strategy.md %}).
19+
20+
<section class="filter-content" markdown="1" data-scope="oracle">
21+
{% include molt/oracle-migration-prerequisites.md %}
2222
</section>
2323

24+
{% include molt/molt-limitations.md %}
25+
2426
## Prepare the source database
2527

2628
{% include molt/migration-prepare-database.md %}

‎src/current/_includes/molt/oracle-migration-prerequisites.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
## Prerequisites
1+
### Prerequisites
22

33
#### Oracle Instant Client
44

‎src/current/molt/migrate-bulk-load.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,11 +62,9 @@ Perform the bulk load of the source data.
6262

6363
{% include molt/fetch-data-load-output.md %}
6464

65-
<section class="filter-content" markdown="1" data-scope="postgres mysql">
6665
## Verify the data load
6766

6867
{% include molt/verify-output.md %}
69-
</section>
7068

7169
## Modify the CockroachDB schema
7270

‎src/current/molt/migrate-data-load-and-replication.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@ toc: true
55
docs_area: migrate
66
---
77

8+
{% assign tab_names_html = "Load and replicate;Replicate separately" %}
9+
{% assign html_page_filenames = "migrate-data-load-and-replication.html;migrate-data-load-replicate-only.html" %}
10+
11+
{% include filter-tabs.md tab_names=tab_names_html page_filenames=html_page_filenames page_folder="molt" %}
12+
813
Use `data-load-and-replication` mode to perform a one-time bulk load of source data and start continuous replication in a single command.
914

1015
{{site.data.alerts.callout_success}}
1116
You can also [load and replicate separately]({% link molt/migrate-data-load-replicate-only.md %}) using `data-load` and `replicate-only`.
1217
{{site.data.alerts.end}}
1318

14-
{% assign tab_names_html = "Load and replicate;Replicate separately" %}
15-
{% assign html_page_filenames = "migrate-data-load-and-replication.html;migrate-data-load-replicate-only.html" %}
16-
17-
{% include filter-tabs.md tab_names=tab_names_html page_filenames=html_page_filenames page_folder="molt" %}
18-
1919
{% include molt/molt-setup.md %}
2020

2121
## Load data into CockroachDB
@@ -52,7 +52,7 @@ Start the initial load of data into the target database. Continuous replication
5252
--table-filter 'employees|payments|orders' \
5353
--bucket-path 's3://migration/data/cockroach' \
5454
--table-handling truncate-if-exists \
55-
--replicator-flags '--metricsAddr :30005' \
55+
--replicator-flags '--metricsAddr :30005 --userscript table_filter.ts' \
5656
--mode data-load-and-replication
5757
~~~
5858
</section>

0 commit comments

Comments
 (0)