diff --git a/_partials/_migrate_live_migrate_faq_all.md b/_partials/_migrate_live_migrate_faq_all.md index 9de2535d6f..8b7f51194e 100644 --- a/_partials/_migrate_live_migrate_faq_all.md +++ b/_partials/_migrate_live_migrate_faq_all.md @@ -13,9 +13,7 @@ a table, index, view, or materialized view. When you see you this error: ### FATAL: remaining connection slots are reserved for non-replication superuser connections This may happen when the number of connections exhaust `max_connections` defined in your target Timescale Cloud -service. By default, live-migration needs around ~10 connections on the source and ~20 connections on the target. -For information on tuning the number of connections during migration, see [Tune the target Timescale Cloud service][tune-connections]. - +service. By default, live-migration needs around ~6 connections on the source and ~12 connections on the target. ### Migration seems to be stuck with “x GB copied to Target DB (Source DB is y GB)” @@ -38,6 +36,24 @@ To resolve this issue: 1. When `migrate` has finished, manually refresh the materialized views you excluded. +### Restart migration from scratch after a non-resumable failure + +If the migration halts due to a failure, such as a misconfiguration of the source or target database, you may need to restart the migration from scratch. In such cases, you can reuse the original Timescale target instance created for the migration by utilizing the `--drop-if-exists` flag with the migrate command. + +This flag ensures that the existing target objects created by the previous migration are dropped, allowing the migration to proceed without trouble. + +Here’s an example command to restart the migration: + +```shell +docker run --rm -it --name live-migration-migrate \ + -e PGCOPYDB_SOURCE_PGURI=$SOURCE \ + -e PGCOPYDB_TARGET_PGURI=$TARGET \ + --pid=host \ + -v ~/live-migration:/opt/timescale/ts_cdc \ + timescale/live-migration:latest migrate --drop-if-exists +``` + +This approach provides a clean slate for the migration process while reusing the existing target instance. ### Inactive or lagging replication slots If you encounter an “Inactive or lagging replication slots” warning on your cloud provider console after using live-migration, it might be due to lingering replication slots created by the live-migration tool on your source database. @@ -80,6 +96,20 @@ Live-migration does not migrate table privileges. After completing Live-migratio 2. If the query is either UPDATE/DELETE, make sure the columns used on the WHERE clause have necessary indexes. 3. If the query is either UPDATE/DELETE on the tables which are converted as hypertables, make sure the REPLIDA IDENTITY(defaults to primary key) on the source is compatible with the target primary key. If not, create an UNIQUE index source database by including the hypertable partition column and make it as a REPLICA IDENTITY. Also, create the same UNIQUE index on target. +### ERROR: out of memory (or) Failed on request of size xxx in memory context "yyy" on Timescale instance + +This error occurs when the Out of Memory (OOM) guard is triggered due to memory allocations exceeding safe limits. It typically happens when multiple concurrent connections to the TimescaleDB instance are performing memory-intensive operations. For example, during live migrations, this error can occur when large indexes are being created simultaneously. + +The live-migration tool includes a retry mechanism to handle such errors. However, frequent OOM crashes may significantly delay the migration process. + +One of the following can be used to avoid the OOM errors: + +1. Upgrade to Higher Memory Spec Instances: To mitigate memory constraints, consider using a TimescaleDB instance with higher specifications, such as an instance with 8 CPUs and 32 GB RAM (or more). Higher memory capacity can handle larger workloads and reduce the likelihood of OOM errors. + +1. Reduce Concurrency: If upgrading your instance is not feasible, you can reduce the concurrency of the index migration process using the `--index-jobs=` flag in the migration command. By default, the value of `--index-jobs` matches the GUC max_parallel_workers. Lowering this value reduces the memory usage during migration but may increase the total migration time. + +By taking these steps, you can prevent OOM errors and ensure a smoother migration experience with TimescaleDB. + ### ERROR: invalid snapshot identifier: "xxxxxx" (or) SSL SYSCALL error: EOF detected on RDS This rare phenomenon may happen when: @@ -103,5 +133,4 @@ This rare phenomenon may happen when: Upgrade to better instances types until migration completes. -[tune-connections]: /migrate/:currentVersion:/live-migration/#tune-the-target-timescale-cloud-service [align-versions]: /migrate/:currentVersion:/live-migration/#align-the-version-of-timescaledb-on-the-source-and-target diff --git a/_partials/_migrate_live_setup_connection_strings.md b/_partials/_migrate_live_setup_connection_strings.md index 0f14a70fcb..c026525a69 100644 --- a/_partials/_migrate_live_setup_connection_strings.md +++ b/_partials/_migrate_live_setup_connection_strings.md @@ -7,3 +7,7 @@ export TARGET="postgres://tsdbadmin:@:/tsdb?sslmode=requir ``` You find the connection information for your Timescale Cloud service in the configuration file you downloaded when you created the service. + + +Avoid using connection strings that route through connection poolers like PgBouncer or similar tools. The live-migration tool requires a direct connection to the database to function properly. + diff --git a/_partials/_migrate_live_setup_environment.md b/_partials/_migrate_live_setup_environment.md index 1c68fffa90..14a407d381 100644 --- a/_partials/_migrate_live_setup_environment.md +++ b/_partials/_migrate_live_setup_environment.md @@ -1,7 +1,6 @@ import SetupConnectionStrings from "versionContent/_partials/_migrate_live_setup_connection_strings.mdx"; import MigrationSetupDBConnectionTimescaleDB from "versionContent/_partials/_migrate_set_up_align_db_extensions_timescaledb.mdx"; import TuneSourceDatabase from "versionContent/_partials/_migrate_live_tune_source_database.mdx"; -import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_live_setup_environment_target_config.mdx"; ## Set your connection strings @@ -22,15 +21,7 @@ import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_liv -## Tune the target Timescale Cloud service - - - - - - - [modify-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#modify-basic-parameters [mst-portal]: https://portal.managed.timescale.com/login [tsc-portal]: https://console.cloud.timescale.com/ -[configure-instance-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#configure-database-parameters \ No newline at end of file +[configure-instance-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#configure-database-parameters diff --git a/_partials/_migrate_live_setup_environment_awsrds.md b/_partials/_migrate_live_setup_environment_awsrds.md index 70167ec96f..874bdcb14b 100644 --- a/_partials/_migrate_live_setup_environment_awsrds.md +++ b/_partials/_migrate_live_setup_environment_awsrds.md @@ -1,7 +1,6 @@ import SetupConnectionStrings from "versionContent/_partials/_migrate_live_setup_connection_strings.mdx"; import MigrationSetupDBConnectionPostgresql from "versionContent/_partials/_migrate_set_up_align_db_extensions_postgres_based.mdx"; import TuneSourceDatabaseAWSRDS from "versionContent/_partials/_migrate_live_tune_source_database_awsrds.mdx"; -import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_live_setup_environment_target_config.mdx"; ## Set your connection strings @@ -23,14 +22,5 @@ import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_liv -## Tune the target Timescale Cloud service - - - - - - - - [modify-parameters]: /use-timescale/:currentVersion:/configuration/customize-configuration/#modify-basic-parameters -[mst-portal]: https://portal.managed.timescale.com/login \ No newline at end of file +[mst-portal]: https://portal.managed.timescale.com/login diff --git a/_partials/_migrate_live_setup_environment_mst.md b/_partials/_migrate_live_setup_environment_mst.md index 5790fbf0f5..3c29264a5a 100644 --- a/_partials/_migrate_live_setup_environment_mst.md +++ b/_partials/_migrate_live_setup_environment_mst.md @@ -1,7 +1,6 @@ import SetupConnectionStrings from "versionContent/_partials/_migrate_live_setup_connection_strings.mdx"; import MigrationSetupDBConnectionTimescaleDB from "versionContent/_partials/_migrate_set_up_align_db_extensions_timescaledb.mdx"; import TuneSourceDatabaseMST from "versionContent/_partials/_migrate_live_tune_source_database_mst.mdx"; -import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_live_setup_environment_target_config.mdx"; ## Set your connection strings @@ -22,14 +21,6 @@ import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_liv -## Tune the target Timescale Cloud service - - - - - - - [modify-parameters]: /use-timescale/:currentVersion:/configuration/customize-configuration/#modify-basic-parameters -[mst-portal]: https://portal.managed.timescale.com/login \ No newline at end of file +[mst-portal]: https://portal.managed.timescale.com/login diff --git a/_partials/_migrate_live_setup_environment_postgres.md b/_partials/_migrate_live_setup_environment_postgres.md index aa391eaccf..2b87975668 100644 --- a/_partials/_migrate_live_setup_environment_postgres.md +++ b/_partials/_migrate_live_setup_environment_postgres.md @@ -1,7 +1,6 @@ import SetupConnectionStrings from "versionContent/_partials/_migrate_live_setup_connection_strings.mdx"; import MigrationSetupDBConnectionPostgresql from "versionContent/_partials/_migrate_set_up_align_db_extensions_postgres_based.mdx"; import TuneSourceDatabasePostgres from "versionContent/_partials/_migrate_live_tune_source_database_postgres.mdx"; -import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_live_setup_environment_target_config.mdx"; ## Set your connection strings @@ -23,15 +22,8 @@ import MigrateSetupTargetEnvironment from "versionContent/_partials/_migrate_liv -## Tune the target Timescale Cloud service - - - - - - [modify-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#modify-basic-parameters [mst-portal]: https://portal.managed.timescale.com/login [tsc-portal]: https://console.cloud.timescale.com/ -[configure-instance-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#configure-database-parameters \ No newline at end of file +[configure-instance-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#configure-database-parameters diff --git a/_partials/_migrate_live_setup_environment_target_config.md b/_partials/_migrate_live_setup_environment_target_config.md deleted file mode 100644 index 462f728b1d..0000000000 --- a/_partials/_migrate_live_setup_environment_target_config.md +++ /dev/null @@ -1,31 +0,0 @@ -1. **Disable connection pooling** - - By default, Timescale Cloud services are created without connection pools. Ensure - that connection pooling is disabled for your source database. - -1. **Setup the connections for migration** - - By default, Timescale Cloud services allow a maximum of 100 concurrent connections. If you are - migrating a lot of data: - - 1. In [Timescale Console][tsc-portal], select `Database parameters` and search for `max_connections`. - 2. Hover the mouse over the value column, click the pencil icon, then change the value. - - By default, for each table, Live-migration consumes 10 connections to the source database and 28 - connections to your target Timescale Cloud service. - - - Source: `2` + (value of `--table-jobs`, default value is `8`) = 10 - - Target: `4` + (`2` * (value of `--table-jobs`, default value is `8`)) + (value of `--index-jobs`, default value is `8`) = 28 - - Set the `--table-jobs` and `--index-jobs` flags to reduce the number of connection when you - [run the migrate command](#migrate-your-data-then-start-downtime). - - 3. Click outside the row, then click `Apply changes and restart`. - - Your service restarts. - - -[modify-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#modify-basic-parameters -[mst-portal]: https://portal.managed.timescale.com/login -[tsc-portal]: https://console.cloud.timescale.com/ -[configure-instance-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/#configure-database-parameters