Skip to content

Commit

Permalink
feat: document live migration from TimescaleDB to Timescale (#2897)
Browse files Browse the repository at this point in the history
* docs: live migration from TimescaleDB to Timescale
* docs: update PG to TS live migration doc
  • Loading branch information
Harkishen-Singh authored Jan 5, 2024
1 parent bd9be6d commit 8fb6943
Show file tree
Hide file tree
Showing 6 changed files with 740 additions and 341 deletions.
48 changes: 48 additions & 0 deletions _partials/_migrate_live_migration_step2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import SetupSourceTarget from "versionContent/_partials/_migrate_set_up_source_and_target.mdx";

<SetupSourceTarget />

It's important to ensure that the `old_snapshot_threshold` value is set to the
default value of `-1` in your source database. This prevents PostgreSQL from
treating the data in a snapshot as outdated. If this value is set other than
`-1`, it might affect the existing data migration step.

To check the current value of `old_snapshot_threshold`, run the command:

```sh
psql -X -d $SOURCE -c 'show old_snapshot_threshold'
```

If the query returns something other than `-1`, you must change it.

If you have a superuser on a self-hosted database, run the following command:

```sh
psql -X -d $SOURCE -c 'alter system set old_snapshot_threshold=-1'
```

Otherwise, if you are using a managed service, use your cloud provider's
configuration mechanism to set `old_snapshot_threshold` to `-1`.

Next, you should set `wal_level` to `logical` so that the write-ahead log (WAL)
records information that is needed for logical decoding.

To check the current value of `wal_level`, run the command:

```sh
psql -X -d $SOURCE -c 'show wal_level'
```

If the query returns something other than `logical`, you must change it.

If you have a superuser on a self-hosted database, run the following command:

```sh
psql -X -d $SOURCE -c 'alter system set wal_level=logical'
```

Otherwise, if you are using a managed service, use your cloud provider's
configuration mechanism to set `wal_level` to `logical`.

Restart your database for the changes to take effect, and verify that the
settings are reflected in your database.
26 changes: 16 additions & 10 deletions migrate/live-migration/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,9 @@ import SourceTargetNote from "versionContent/_partials/_migrate_source_target_no
# Live migration

Live migration is a migration strategy to move a large amount of data
(100&nbsp;GB-10&nbsp;TB+) with low downtime (on the order of minutes of
downtime). It is significantly more complicated to execute than a migration
with downtime using [pg_dump/restore][pg-dump-and-restore], but supports more
use-cases and has less requirements than the [dual-write and backfill] method.
(100&nbsp;GB-10&nbsp;TB+) with low downtime (on the order of few minutes). It requires
more steps to execute than a migration with downtime using [pg_dump/restore][pg-dump-and-restore],
but supports more use-cases and has less requirements than the [dual-write and backfill] method.

<SourceTargetNote />

Expand All @@ -27,20 +26,27 @@ Roughly, it consists of four steps:
3. Copy data from source to target while capturing changes.
4. Apply captured changes from source to target.

Currently live migration only supports migrating from PostgreSQL, but we are
actively working on supporting TimescaleDB.

Live migration works well when:
- Large, busy tables have primary keys, or don't have many `UPDATE` or
`DELETE` statements.
- The insert workload does not exceed 20'000 rows per second, and
- The insert workload does not exceed 20,000 rows per second, and
inserts are batched. If your application exceeds this, you should use
the [dual-write and backfill] migration method.

For more information, consult the step-by-step migration guide:
For more information, refer to the step-by-step migration guide:

- [Live migration from PostgreSQL][from-postgres]
- [Live migration from TimescaleDB][from-timescaledb]

If you want to manually migrate data from PostgreSQL, refer to
[Live migration from PostgreSQL manually][live-migration-manual].

If you are migrating from AWS RDS to Timescale, you can refer to [this][live-migration-playbook] playbook
for a detailed migration guide.

[from-postgres]: /migrate/:currentVersion:/live-migration/live-migration-from-postgres/
[from-timescaledb]: /migrate/:currentVersion:/live-migration/live-migration-from-timescaledb/
[live-migration-manual]: /migrate/:currentVersion:/playbooks/live-migration-from-timescaledb-manually/
[pg-dump-and-restore]: /migrate/:currentVersion:/pg-dump-and-restore/
[dual-write and backfill]: /migrate/:currentVersion:/dual-write-and-backfill/
[dual-write and backfill]: /migrate/:currentVersion:/dual-write-and-backfill/
[live-migration-playbook]: /migrate/:currentVersion:/playbooks/rds-timescale-live-migration/
Loading

0 comments on commit 8fb6943

Please sign in to comment.