Skip to content

Commit

Permalink
Add live migrations to index page (#2763)
Browse files Browse the repository at this point in the history
* resolve conflicts

* update the live migrate page ref

* minor edits

---------

Co-authored-by: James Guthrie <[email protected]>
  • Loading branch information
VineethReddy02 and JamesGuthrie authored Oct 30, 2023
1 parent 8537fbe commit eb110e1
Showing 1 changed file with 48 additions and 15 deletions.
63 changes: 48 additions & 15 deletions migrate/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,36 +18,67 @@ which are:
- How much data you are migrating (megabytes, or terabytes?)
- Where you are migrating your data from (PostgreSQL, TimescaleDB, Influx, or MySQL?)

Below are the different migration options we offer. You can choose the one that best suits your requirements:

| Migration strategy | Supported sources | Use when | Downtime requirements |
|-------------------------------------------|----------------------------------|----------------------------------------------------------------------|--------------------------------|
| [pg_dump and pg_restore][pg-dump-restore] | Postgres, TimescaleDB | Downtime is okay | Requires some downtime |
| [Dual-write and backfill][dual-write] | Postgres, TimescaleDB and others | Append-only data, heavy insert workload (~20,000 inserts per second) | Optimized for minimal downtime |
| [Live migration][live-migration] | Postgres | Simplified end-to-end migration with almost zero downtime | Optimized for minimal downtime |

If you are using PostgreSQL or TimescaleDB and can afford to take your
application offline for a few hours, the simplest option is to migrate data
from another database into Timescale using PostgreSQL's `pg_dump` and
`pg_restore` commands.
`pg_restore` commands.

You can also use these tools to migrate your data from Managed Service for
TimescaleDB, from a self-hosted TimescaleDB instance, or from another
PostgreSQL database, including Amazon RDS. Consult the guide on [migrating with
pg_dump and pg_restore][pg-dump-restore].
PostgreSQL database, including Amazon RDS.

If you are looking for a low-downtime alternative (with downtime on the order
of minutes), you can choose one of our low downtime migration offerings:

1. Live migration
2. Dual-write and backfill

Live migration is Timescale's packaging of Postgres' logical replication
functionality, which leverages the excellent [pgcopydb] project from the
Postgres community. Live migration is an end-to-end solution which copies the
database schema and data, and then replicates changes in real-time from one
database to the other. At present, live migration only supports Postgres as the
source database (not TimescaleDB).

Dual-write and backfill is a method to write from your application to two
databases at once, and gives tooling and guidance to move your existing data
from the one database to the other. It is specifically catered for, and relies
on, your data being predominantly append-only time-series data. As such, it
comes with some caveats and prerequisites which live migration does not
(dual-write and backfill does not support executing `UPDATE` or `DELETE`
statements on your data). Additionally, it requires you to make changes to the
ingest pipeline of your application.

If you are looking for a low-downtime alternative (downtime on the order of
minutes), are willing to modify your ingestion code, and the bulk of your data
is stored in time-series tables, you can use the [dual-write and backfill][dual-write]
strategy for a low-downtime migration. This strategy also works if you are not
migrating from PostgreSQL.
We recommend using live migration if modifying your application logic to
perform dual writes is a significant effort on your end, or if you are using
`UPDATE` and `DELETE` statements on your time-series data. If your application
is writing more than 20'000 rows per second, you should use dual-write and
backfill instead.

If you're using PostgreSQL, you may also have heard of logical replication
being the recommended strategy for migrations with low downtime. Currently,
TimescaleDB supports logical replication only from sources that are using
PostgreSQL without Timescale. There's active work being made for supporting
Timescale sources.
We recommend using dual-write and backfill if modifying your ingest pipeline is
not an issue, and if you want to evaluate your existing solution side-by-side
with the new Timescale-based solution. Dual-write and backfill is the only
low-downtime solution if you're migrating from a database other than PostgreSQL.

If you're migrating from something other than PostgreSQL, and don't want to use
the dual-write and backfill approach, then the easiest way to move your data to
above listed migration options, then the easiest way to move your data to
Timescale is by exporting the data from your existing database as a `.csv` file,
and importing it with [timescaledb-parallel-copy][parallel-copy].

For other ingestion methods, see the [data ingest section][data-ingest].

If you encounter any difficulties while migrating, consult the
[troubleshooting] documentation, or open a support request.
[troubleshooting] documentation, open a support requestl, or take
your issue to the `#migration` channel in the [community slack](https://slack.timescale.com/),
where the developers of this migration method are there to help.

<OpenSupportRequest />

Expand All @@ -56,3 +87,5 @@ If you encounter any difficulties while migrating, consult the
[pg-dump-restore]: /migrate/:currentVersion:/pg-dump-and-restore/
[parallel-copy]: /use-timescale/:currentVersion:/ingest-data/import-csv/
[troubleshooting]: /migrate/:currentVersion:/troubleshooting/
[live-migration]: /migrate/:currentVersion:/live-migration/
[pgcopydb]: https://github.com/dimitri/pgcopydb

0 comments on commit eb110e1

Please sign in to comment.