diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md deleted file mode 100644 index b4017b661..000000000 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -Title: Prepare AWS Aurora and PostgreSQL for RDI -aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/my-sql-mariadb/ -alwaysopen: false -categories: -- docs -- integrate -- rs -- rdi -description: Prepare AWS Aurora/PostgreSQL databases to work with RDI -group: di -linkTitle: Prepare AWS Aurora/PostgreSQL -summary: Redis Data Integration keeps Redis in sync with the primary database in near - real time. -type: integration -weight: 5 ---- - -Follow the steps in the sections below to prepare an -[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) -database to work with RDI. - -## 1. Create a parameter group - -In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), -navigate to **Parameter groups > Create parameter group**. You will see the panel shown -below: - -{{Create parameter group panel}} - -Enter the following information: - -| Name | Value | -| :-- | :-- | -| **Parameter group name** | rdi-aurora-pg | -| **Description** | Enable logical replication for RDI | -| **Engine Type** | Aurora PostgreSQL | -| **Parameter group family** | aurora-postgresql15 | -| **Type** | DB Cluster Parameter Group | - -Select **Create** to create the parameter group. - -## 2. Edit the parameter group - -Navigate to **Parameter groups** in the console. Select the `rdi-aurora-pg` -group you have just created and then select **Edit** . You will see this panel: - -{{Edit parameter group panel}} - -Search for the `rds.logical_replication` parameter and set its value to 1. Then, -select **Save Changes**. - -## 3. Select the new parameter group - -Go back to your target database on the RDS console, select **Modify** and then -scroll down to **Additional Configuration**. Set -the **DB Cluster Parameter Group** to the value `rdi-aurora-pg` that you have just added: - -{{Additional Configuration panel}} diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md new file mode 100644 index 000000000..061188eb1 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/_index.md @@ -0,0 +1,22 @@ +--- +Title: Prepare AWS RDS and Aurora databases for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to prepare AWS RDS and Aurora databases for RDI. +group: di +linkTitle: Prepare AWS RDS and Aurora +summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration. +hideListLinks: false +type: integration +weight: 5 +--- + +You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html). + +The pages in this section give detailed instructions to get your source +database ready for Debezium to use: \ No newline at end of file diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md new file mode 100644 index 000000000..211aeae8c --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md @@ -0,0 +1,78 @@ +--- +Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Enable CDC features in your source databases +group: di +hideListLinks: false +linkTitle: Prepare AWS Aurora/RDS MySQL +summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration. +type: integration +weight: 2 +--- + +Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database. +database to work with RDI. + +## Create and apply parameter group + +RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group. + +1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings: + + | Name | Value | + | :-- | :-- | + | **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` | + | **Description** | (Optional) Enter a description for the parameter group | + | **Engine Type** | Choose **Aurora MySQL** for Aurora MySQL or **MySQL Community** for AWS RDS MySQL. | + | **Parameter group family** | Choose **aurora-mysql8.0** for Aurora MySQL or **mysql8.0** for AWS RDS MySQL. | + + Select **Create** to create the parameter group. + +1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters: + + | Name | Value | + | :-- | :-- | + | `binlog_format` | `ROW` | + | `binlog_row_image` | `FULL` | + + Select **Save Changes** to apply the changes to the parameter group. + +1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created. + + Select **Save changes** to apply the parameter group to the new database. + +## Create Debezium user + +The Debezium connector needs a user account to connect to MySQL. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to your database as an admin user and create a new user for the connector: + + ```sql + CREATE USER ''@'%' IDENTIFIED BY ''; + ``` + + Replace `` and `` with a username and password for the new user. + + The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host. + +1. Grant the user the necessary permissions: + + ```sql + GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO ''@'%'; + ``` + + Replace `` with the username of the Debezium user. + +1. Finalize the user's permissions: + + ```sql + FLUSH PRIVILEGES; + ``` diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md new file mode 100644 index 000000000..ec2dee520 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md @@ -0,0 +1,82 @@ +--- +Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI +aliases: +- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/ +- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/ +- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rc +- rdi +description: Prepare AWS Aurora PostgreSQL databases to work with RDI +group: di +linkTitle: Prepare AWS Aurora PostgreSQL +summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration. +type: integration +weight: 1 +--- + +Follow the steps in the sections below to prepare an +[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html) +database to work with RDI. + +## Create and apply parameter group + +RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group. + +1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings: + + | Name | Value | + | :-- | :-- | + | **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` | + | **Description** | (Optional) Enter a description for the parameter group | + | **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. | + | **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. | + + Select **Create** to create the parameter group. + +1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters: + + | Name | Value | + | :-- | :-- | + | `rds.logical_replication` | `1` | + + Select **Save Changes** to apply the changes to the parameter group. + +1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created. + + Select **Save changes** to apply the parameter group to your database. + +## Create Debezium user + +The Debezium connector needs a user account to connect to PostgreSQL. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector: + + ```sql + CREATE ROLE WITH LOGIN PASSWORD '' VALID UNTIL 'infinity'; + ``` + + Replace `` and `` with a username and password for the new user. + +1. Grant the user the necessary replication permissions: + + ```sql + GRANT rds_replication TO ; + ``` + + Replace `` with the username of the Debezium user. + +1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database: + + ```sql + GRANT SELECT ON ALL TABLES IN SCHEMA TO ; + ``` + + Replace `` with the username of the Debezium user and `` with the schema name. + diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md new file mode 100644 index 000000000..7f96b1948 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md @@ -0,0 +1,100 @@ +--- +Title: Prepare Microsoft SQL Server on AWS RDS for RDI +aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Enable CDC features in your source databases +group: di +hideListLinks: false +linkTitle: Prepare Microsoft SQL Server on AWS RDS +summary: Prepare Microsoft SQL Server on AWS RDS databases to work with Redis Data Integration. +type: integration +weight: 3 +--- + +Follow the steps in the sections below to prepare a [Microsoft SQL Server on AWS RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.SQLServer.html) database to work with RDI. + +## Create the Debezium user + +The Debezium connector needs a user account to connect to SQL Server. This +user must have appropriate permissions on all databases where you want Debezium +to capture changes. + +1. Connect to your database as an admin user and create a new user for the connector: + + ```sql + USE master + GO + CREATE LOGIN WITH PASSWORD = '' + GO + USE + GO + CREATE USER FOR LOGIN + GO + ``` + + Replace `` and `` with a username and password for the new user and replace `` with the name of your database. + +1. Grant the user the necessary permissions: + + ```sql + USE master + GO + GRANT VIEW SERVER STATE TO + GO + USE + GO + EXEC sp_addrolemember N'db_datareader', N'' + GO + ``` + + Replace `` with the username of the Debezium user and replace `` with the name of your database. + +## Enable CDC on the database + +Change Data Capture (CDC) must be enabled for the database and for each table you want to capture. + +1. Enable CDC for the database by running the following command: + + ```sql + EXEC msdb.dbo.rds_cdc_enable_db '' + GO + ``` + + Replace `` with the name of your database. + +1. Enable CDC for each table you want to capture by running the following commands: + + ```sql + USE + GO + EXEC sys.sp_cdc_enable_table + @source_schema = N'', + @source_name = N'', + @role_name = N'', + @supports_net_changes = 0 + GO + ``` + + Replace `` with the name of your database, `` with the name of the schema containing the table, `
` with the name of the table, and `` with the name of a new role that will be created to manage access to the CDC data. + + {{< note >}} +The value for `@role_name` can’t be a fixed database role, such as `db_datareader`. +Specifying a new name will create a corresponding database role that has full access to the +captured change data. + {{< /note >}} + +1. Add the Debezium user to the CDC role: + + ```sql + USE + GO + EXEC sp_addrolemember N'', N'' + GO + ``` + + Replace `` with the name of the role you created in the previous step and replace `` with the username of the Debezium user. \ No newline at end of file diff --git a/content/operate/rc/databases/rdi/setup.md b/content/operate/rc/databases/rdi/setup.md index f62675ee0..5242a4ca7 100644 --- a/content/operate/rc/databases/rdi/setup.md +++ b/content/operate/rc/databases/rdi/setup.md @@ -35,11 +35,15 @@ weight: 1 Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{}}) to find a list of supported source databases and database versions. See [Prepare source databases]({{}}) to find steps for your database type: -- [MySQL and mariaDB]({{}}) -- [Oracle]({{}}) -- [SQL Server]({{}}) -- [AWS Aurora PostgreSQL]({{}}) -- [AWS RDS PostgreSQL]({{}}) +- Hosted on an AWS EC2 instance: + - [MySQL and mariaDB]({{}}) + - [Oracle]({{}}) + - [SQL Server]({{}}) + - [PostgreSQL]({{}}) +- Hosted on AWS RDS or AWS Aurora: + - [AWS Aurora PostgreSQL and AWS RDS PostgreSQL]({{}}) + - [AWS Aurora MySQL and AWS RDS MySQL]({{}}) + - [AWS RDS SQL Server]({{}}) See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.