Skip to content

RC RDI: Add more setup databases pages #1544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
Title: Prepare AWS RDS and Aurora databases for RDI
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Learn how to prepare AWS RDS and Aurora databases for RDI.
group: di
linkTitle: Prepare AWS RDS and Aurora
summary: Prepare AWS Aurora and AWS RDS databases to work with Redis Data Integration.
hideListLinks: false
type: integration
weight: 5
---

You can use RDI with databases on [AWS Relational Database Service (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) and [AWS Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraOverview.html).

The pages in this section give detailed instructions to get your source
database ready for Debezium to use:
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
---
Title: Prepare AWS Aurora MySQL/AWS RDS MySQL for RDI
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Enable CDC features in your source databases
group: di
hideListLinks: false
linkTitle: Prepare AWS Aurora/RDS MySQL
summary: Prepare AWS Aurora MySQL and AWS RDS MySQL databases to work with Redis Data Integration.
type: integration
weight: 2
---

Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database.
database to work with RDI.

## Create and apply parameter group

RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.

1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:

| Name | Value |
| :-- | :-- |
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-mysql` |
| **Description** | (Optional) Enter a description for the parameter group |
| **Engine Type** | Choose **Aurora MySQL** for Aurora MySQL or **MySQL Community** for AWS RDS MySQL. |
| **Parameter group family** | Choose **aurora-mysql8.0** for Aurora MySQL or **mysql8.0** for AWS RDS MySQL. |

Select **Create** to create the parameter group.

1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:

| Name | Value |
| :-- | :-- |
| `binlog_format` | `ROW` |
| `binlog_row_image` | `FULL` |

Select **Save Changes** to apply the changes to the parameter group.

1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.

Select **Save changes** to apply the parameter group to the new database.

## Create Debezium user

The Debezium connector needs a user account to connect to MySQL. This
user must have appropriate permissions on all databases where you want Debezium
to capture changes.

1. Connect to your database as an admin user and create a new user for the connector:

```sql
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
```

Replace `<username>` and `<password>` with a username and password for the new user.

The `%` means that the user can connect from any client. If you want to restrict the user to connect only from the RDI host, replace `%` with the IP address of the RDI host.

1. Grant the user the necessary permissions:

```sql
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO '<username>'@'%';
```

Replace `<username>` with the username of the Debezium user.

1. Finalize the user's permissions:

```sql
FLUSH PRIVILEGES;
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
Title: Prepare AWS Aurora PostgreSQL/AWS RDS PostgreSQL for RDI
aliases:
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aur-pgsql/
- /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql/
- /integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rc
- rdi
description: Prepare AWS Aurora PostgreSQL databases to work with RDI
group: di
linkTitle: Prepare AWS Aurora PostgreSQL
summary: Prepare AWS Aurora PostgreSQL databases to work with Redis Data Integration.
type: integration
weight: 1
---

Follow the steps in the sections below to prepare an
[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html)
database to work with RDI.

## Create and apply parameter group

RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.

1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:

| Name | Value |
| :-- | :-- |
| **Parameter group name** | Enter a suitable parameter group name, like `rdi-aurora-pg` or `rdi-rds-pg` |
| **Description** | (Optional) Enter a description for the parameter group |
| **Engine Type** | Choose **Aurora PostgreSQL** for Aurora PostgreSQL or **PostgreSQL** for AWS RDS PostgreSQL. |
| **Parameter group family** | Choose **aurora-postgresql15** for Aurora PostgreSQL or **postgresql13** for AWS RDS PostgreSQL. |

Select **Create** to create the parameter group.

1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:

| Name | Value |
| :-- | :-- |
| `rds.logical_replication` | `1` |

Select **Save Changes** to apply the changes to the parameter group.

1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.

Select **Save changes** to apply the parameter group to your database.

## Create Debezium user

The Debezium connector needs a user account to connect to PostgreSQL. This
user must have appropriate permissions on all databases where you want Debezium
to capture changes.

1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector:

```sql
CREATE ROLE <username> WITH LOGIN PASSWORD '<password>' VALID UNTIL 'infinity';
```

Replace `<username>` and `<password>` with a username and password for the new user.

1. Grant the user the necessary replication permissions:

```sql
GRANT rds_replication TO <username>;
```

Replace `<username>` with the username of the Debezium user.

1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:

```sql
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <username>;
```

Replace `<username>` with the username of the Debezium user and `<schema>` with the schema name.

Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
---
Title: Prepare Microsoft SQL Server on AWS RDS for RDI
aliases: /integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Enable CDC features in your source databases
group: di
hideListLinks: false
linkTitle: Prepare Microsoft SQL Server on AWS RDS
summary: Prepare Microsoft SQL Server on AWS RDS databases to work with Redis Data Integration.
type: integration
weight: 3
---

Follow the steps in the sections below to prepare a [Microsoft SQL Server on AWS RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.SQLServer.html) database to work with RDI.

## Create the Debezium user

The Debezium connector needs a user account to connect to SQL Server. This
user must have appropriate permissions on all databases where you want Debezium
to capture changes.

1. Connect to your database as an admin user and create a new user for the connector:

```sql
USE master
GO
CREATE LOGIN <username> WITH PASSWORD = '<password>'
GO
USE <database>
GO
CREATE USER <username> FOR LOGIN <username>
GO
```

Replace `<username>` and `<password>` with a username and password for the new user and replace `<database>` with the name of your database.

1. Grant the user the necessary permissions:

```sql
USE master
GO
GRANT VIEW SERVER STATE TO <username>
GO
USE <database>
GO
EXEC sp_addrolemember N'db_datareader', N'<username>'
GO
```

Replace `<username>` with the username of the Debezium user and replace `<database>` with the name of your database.

## Enable CDC on the database

Change Data Capture (CDC) must be enabled for the database and for each table you want to capture.

1. Enable CDC for the database by running the following command:

```sql
EXEC msdb.dbo.rds_cdc_enable_db '<database>'
GO
```

Replace `<database>` with the name of your database.

1. Enable CDC for each table you want to capture by running the following commands:

```sql
USE <database>
GO
EXEC sys.sp_cdc_enable_table
@source_schema = N'<schema>',
@source_name = N'<table>',
@role_name = N'<role>',
@supports_net_changes = 0
GO
```

Replace `<database>` with the name of your database, `<schema>` with the name of the schema containing the table, `<table>` with the name of the table, and `<role>` with the name of a new role that will be created to manage access to the CDC data.

{{< note >}}
The value for `@role_name` can’t be a fixed database role, such as `db_datareader`.
Specifying a new name will create a corresponding database role that has full access to the
captured change data.
{{< /note >}}

1. Add the Debezium user to the CDC role:

```sql
USE <database>
GO
EXEC sp_addrolemember N'<role>', N'<username>'
GO
```

Replace `<role>` with the name of the role you created in the previous step and replace `<username>` with the username of the Debezium user.
14 changes: 9 additions & 5 deletions content/operate/rc/databases/rdi/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,15 @@ weight: 1
Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC). See [Prerequisites]({{<relref "/operate/rc/databases/rdi#prerequisites">}}) to find a list of supported source databases and database versions.

See [Prepare source databases]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/">}}) to find steps for your database type:
- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}})
- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}})
- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}})
- [AWS Aurora PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aur-pgsql">}})
- [AWS RDS PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}})
- Hosted on an AWS EC2 instance:
- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}})
- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}})
- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}})
- [PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}})
- Hosted on AWS RDS or AWS Aurora:
- [AWS Aurora PostgreSQL and AWS RDS PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql">}})
- [AWS Aurora MySQL and AWS RDS MySQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql">}})
- [AWS RDS SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver">}})

See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.

Expand Down