Skip to content

Add live migration from postgres #2759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import SourceTargetNote from "versionContent/_partials/_migrate_source_target_no
import DumpDatabaseRoles from "versionContent/_partials/_migrate_dual_write_dump_database_roles.mdx";
import Step6eTurnOnCompressionPolicies from "versionContent/_partials/_migrate_dual_write_6e_turn_on_compression_policies.mdx";
import Step6aThroughc from "versionContent/_partials/_migrate_dual_write_6a_through_c.mdx";
import ExplainPgDumpFlags from "versionContent/_partials/_migrate_explain_pg_dump_flags.mdx";

# Dual-write and backfill from PostgreSQL database

Expand Down
5 changes: 3 additions & 2 deletions migrate/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ migrating from PostgreSQL.

If you're using PostgreSQL, you may also have heard of logical replication
being the recommended strategy for migrations with low downtime. Currently,
TimescaleDB doesn't work with logical replication, so this is not a viable
option, but we are actively working on making this possible.
TimescaleDB supports logical replication only from sources that are using
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't true but this PR should address this.

PostgreSQL without Timescale. There's active work being made for supporting
Timescale sources.

If you're migrating from something other than PostgreSQL, and don't want to use
the dual-write and backfill approach, then the easiest way to move your data to
Expand Down
45 changes: 45 additions & 0 deletions migrate/live-migration/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: Live migration
excerpt: Migrate an entire database with low downtime
products: [cloud]
keywords: [migration, low-downtime, backup]
tags: [recovery, logical backup, replication]
---

# Live migration

Live migration is a migration strategy to move a large amount of data
(100 GB-10 TB+) with low downtime (on the order of minutes of
downtime). It is significantly more complicated to execute than a migration
with downtime using [pg_dump/restore][pg-dump-and-restore], but supports more
use-cases and has less requirements than the [dual-write and backfill] method.

<SourceTargetNote />

Live migration leverages Postgres' built-in replication functionality to
provide a seamless migration with very little application downtime.

Roughly, it consists of four steps:

1. Prepare and create replication slot in source database.
2. Copy schema from source to target, optionally enabling hypertables.
3. Copy data from source to target while capturing changes.
4. Apply captured changes from source to target.

Currently live migration only supports migrating from PostgreSQL, but we are
actively working on supporting TimescaleDB.

Live migration works well when:
- Large, busy tables have primary keys, or don't have many `UPDATE` or
`DELETE` statements.
- The insert workload does not exceed 20'000 rows per second, and
inserts are batched. If your application exceeds this, you should use
the [dual-write and backfill] migration method.

For more information, consult the step-by-step migration guide:

- [Live migration from PostgreSQL][from-postgres]

[from-postgres]: /migrate/:currentVersion:/live-migration/live-migration-from-postgres/
[pg-dump-and-restore]: /migrate/:currentVersion:/pg-dump-and-restore/
[dual-write and backfill]: /migrate/:currentVersion:/dual-write-and-backfill/
Loading