Skip to content

Commit 2fbdc60

Browse files
LINxianshengLINxiansheng
authored
Merge pull request #71 from oceanbase/xiaofeng_branch
增加英文版入门教程第四章
2 parents 7460bfa + eac4a46 commit 2fbdc60

File tree

18 files changed

+1872
-3
lines changed

18 files changed

+1872
-3
lines changed

‎docs/user_manual/quick_starts_and_hands_on_practices_in_english/chapter_04_migration_and_synchronization_oceanbase/00_introduction.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: Introduction
3+
weight: 1
4+
---
5+
6+
# 4.0 Introduction
7+
8+
This chapter describes how to migrate and synchronize data in OceanBase Database.
9+
10+
## Contact us
11+
12+
You can contact us in the following ways:
13+
14+
* Forum on the official website of OceanBase Database Community Edition: [https://ask.oceanbase.com/](https://ask.oceanbase.com/)
15+
16+
* GitHub page for reporting issues of OceanBase Database Community Edition: [https://github.com/oceanbase/oceanbase/issues](https://github.com/oceanbase/oceanbase/issues)
17+
18+
* DingTalk group ID: 33254054
19+
20+
> **Note**
21+
>
22+
> You can click [here](https://open.oceanbase.com/course/275) to learn the supporting course of this tutorial — From Beginner to Pro: A Guide for DBAs.

‎docs/user_manual/quick_starts_and_hands_on_practices_in_english/chapter_04_migration_and_synchronization_oceanbase/01_compatibility_introduction.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
---
2+
title: Compatibility between OceanBase Database and MySQL
3+
weight: 2
4+
---
5+
6+
# 4.1 Compatibility between OceanBase Database and MySQL
7+
8+
To migrate or synchronize data between heterogeneous databases, you need to first learn about the compatibility between these databases in terms of, for example, data types, character sets, collations, and indexes. Data migration or synchronization between incompatible databases will inevitably fail. That is why you need to grasp features and characteristics of source databases, and check whether they are compatible with destination databases before migration, and if not, whether other better alternatives are available. This topic describes the compatibility between OceanBase Database V4.2.1 and MySQL 8.x. You can check the compatibility between OceanBase Database and other databases by referring to this topic.
9+
10+
OceanBase Database in MySQL mode is compatible with most features and statements of MySQL 5.7 or 8.0. In this topic, differences between OceanBase Database in MySQL mode and native MySQL are described in the following aspects: data types, strings, procedural language (PL) features, system views, character sets, collations, indexes, SQL_MODE, partitions, and backup and restore.
11+
12+
> **Note**
13+
>
14+
> For information about compatibility between OceanBase Database of other versions and MySQL, see [Compatibility with MySQL](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001103402) in OceanBase Database documentation of the target versions.
15+
16+
## Data types
17+
18+
OceanBase Database supports all data types of MySQL except the `SERIAL` type.
19+
20+
## Length comparison of string types
21+
22+
| Type | MySQL 8.0 | OceanBase Database V4.2.1 |
23+
| ---------- | ------------------------------- | --------------------- |
24+
| CHAR | 255 characters | 256 characters |
25+
| VARCHAR | 65,535 characters (around 16,383 characters in fact) | 262,144 characters |
26+
| BINARY | 255 bytes | 256 bytes |
27+
| VARBINARY | 65,535 bytes (around 65,532 characters in fact) | 1,048,576 bytes |
28+
| TINYBLOB | 255 bytes | 255 bytes |
29+
| BLOB | 65,535 bytes | 65,536 bytes |
30+
| MEDIUMBLOB | 16,777,215 bytes | 16,777,216 bytes |
31+
| LONGBLOB | 4,294,967,295 bytes (4 GB) | 536,870,911 bytes |
32+
| TINYTEXT | 255 bytes | 255 bytes |
33+
| TEXT | 65,535 bytes | 65,536 bytes |
34+
| MEDIUMTEXT | 16,777,215 bytes | 6,777,216 bytes |
35+
| LONGTEXT | 4,294,967,295 bytes (4 GB) | 536,870,911 bytes |
36+
37+
## PL features
38+
39+
OceanBase Database Community Edition is compatible with most PL features of MySQL. For more information, see [PL reference](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001104171).
40+
41+
OceanBase Database supports the following PL features:
42+
43+
* Data types
44+
45+
* Stored procedures
46+
47+
* Custom functions
48+
49+
* Triggers
50+
51+
* Exception handling
52+
53+
OceanBase Database also provides particular MySQL PL packages, such as `DBMS_RESOURCE_MANAGER`, `DBMS_STATS`, `DBMS_UDR`, `DBMS_XPLAN` and `DBMS_WORKLOAD_REPOSITORY`.
54+
55+
## System views
56+
57+
OceanBase Database implements most views of two internal databases: `information_schema` and `mysql`. However, due to the differences from MySQL in architecture, OceanBase Database cannot implement all the views of MySQL or ensure consistency with MySQL in definitions of all columns in the views.
58+
59+
For more information about the columns in system views, see [Overview](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001103435).
60+
61+
## Character sets and collations
62+
63+
This section describes only the character sets and collations supported by OceanBase Database Community Edition V4.2.1.
64+
65+
> **Notice**
66+
>
67+
> The supported character sets and collations may vary in different versions. If you use OceanBase Database Community Edition of a version other than V4.2.1, run the `show charset` and `show collation` commands to view the supported character sets and collations.
68+
69+
The following table describes the character sets and collations supported by OceanBase Database Community Edition.
70+
71+
| Character set | Collation | Description |
72+
| ------------ | ----------------------- | ------------------------------------------------------------------ |
73+
| utf8mb4 | utf8mb4_general_ci | A general collation. |
74+
| utf8mb4 | utf8mb4_bin | A binary collation. |
75+
| binary | binary | A binary collation. |
76+
| gbk | gbk_chinese_ci | A collation for Chinese. |
77+
| gbk | gbk_bin | A binary collation. |
78+
| utf16 | utf16_general_ci | A general collation. |
79+
| utf16 | utf16_bin | A binary collation. |
80+
| gb18030 | gb18030_chinese_ci | A collation for Chinese. |
81+
| gb18030 | gb18030_bin | A binary collation. |
82+
| latin1 | latin1_swedish_ci | A collation for Swedish/Finnish. |
83+
| latin1 | latin1_bin | A binary collation. |
84+
| gb18030_2022 | gb18030_2022_bin | A binary collation. |
85+
| gb18030_2022 | gb18030_2022_chinese_ci | A Pinyin collation for Chinese. The collation is case-insensitive. This is the default collation for this character set in MySQL mode. |
86+
| gb18030_2022 | gb18030_2022_chinese_cs | A Pinyin collation for Chinese. The collation is case-sensitive. |
87+
| gb18030_2022 | gb18030_2022_radical_ci | A radical stroke collation for Chinese. The collation is case-insensitive. |
88+
| gb18030_2022 | gb18030_2022_radical_cs | A radical stroke collation for Chinese. The collation is case-sensitive. |
89+
| gb18030_2022 | gb18030_2022_stroke_ci | A stroke collation for Chinese. The collation is case-insensitive. |
90+
| gb18030_2022 | gb18030_2022_stroke_cs | A stroke collation for Chinese. The collation is case-sensitive. |
91+
92+
For more information about character sets and collations, see [Character set and collation](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001106903).
93+
94+
## Indexes
95+
96+
The following table describes index types that are supported in MySQL but not in OceanBase Database.
97+
98+
| Index type | Index data structure | MySQL | OceanBase Database |
99+
| -------- | ------------ | ------------ | ---------------- |
100+
| Index extension | B-tree | Supported | Not supported |
101+
| Descending index | B-tree | Supported | Not supported |
102+
| Full-text index | B-tree | Supported | Not supported |
103+
| HASH index | B-tree | Supported | Not supported |
104+
| LOCK option | / | Supported | Not supported |
105+
| Index merge | B-tree | Supported | Not supported |
106+
107+
For more information about indexes, see [Indexes](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001106626).
108+
109+
## SQL_MODE
110+
111+
OceanBase Database V4.2.1 supports all SQL_MODE values that are supported by MySQL. For more information, see [sql_mode](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001105329).
112+
113+
## Partitions
114+
115+
The partition support feature of OceanBase Database is different from that of MySQL.
116+
117+
* OceanBase Database supports partitioning, template-based subpartitioning, and non-template-based subpartitioning. MySQL does not support non-template-based subpartitioning.
118+
119+
* OceanBase Database supports the following subpartitioning methods: HASH, KEY, RANGE, RANGE COLUMNS, LIST, and LIST COLUMNS. MySQL supports only HASH and KEY subpartitioning.
120+
121+
For more information, see [Create and manage partitions](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001105785).
122+
123+
## Backup and restore
124+
125+
OceanBase Database is compatible with some backup and restore features of MySQL. For example, OceanBase Database supports the following features:
126+
127+
* Full backup and incremental backup.
128+
129+
* Hot backup.
130+
131+
* Table-level restore.
132+
133+
OceanBase Database does not support the following features:
134+
135+
* Cluster-level backup or restore.
136+
137+
* Cold backup.
138+
139+
* Validation of backup data.
140+
141+
* Backup or restore for some databases within tenants.

‎docs/user_manual/quick_starts_and_hands_on_practices_in_english/chapter_04_migration_and_synchronization_oceanbase/02_ecological_tools_of_migration_synchronization.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
title: Ecosystem components for data migration and synchronization
3+
weight: 3
4+
---
5+
6+
# 4.2 Ecosystem components for data migration and synchronization
7+
8+
Database O&M engineers often need to migrate or synchronize data in the event of database replacement, IDC relocation, business testing, database upgrade, and other routine database O&M operations. You can perform operations such as inter-table data archiving, disk usage balancing, and resource unit migration within an OceanBase cluster by using simple commands. However, you need tools to, for example, synchronize data between clusters or with heterogeneous data sources.
9+
10+
This topic describes several frequently used data migration methods and tools.
11+
12+
The following table lists the migration solutions supported by OceanBase Database.
13+
14+
| Migration solution | Schema migration | Full data migration | Incremental migration | Data verification | Supported data source |
15+
| ------------------- | -------- | ------------ | ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
16+
| OceanBase Migration Service (OMS) | Supported | Supported | Supported | Supported | <ul><li>OceanBase</li><li>MySQL</li><li>MariaDB</li><li>PostgreSQL</li><li>GreenPlum</li><li>HBase</li><li>TiDB</li><li>Kafka</li><li>RocketMQ</li></ul> |
17+
| OBLogProxy | Supported | Not supported | Supported | Not supported | <ul><li>OceanBase</li><li>MySQL binlog tools</li><li>CDC tools</li><li>OBLogClient</li></ul> |
18+
| OBLOADER & OBDUMPER | Supported | Supported | Not supported | Not supported | OceanBase |
19+
| SQL statements | Supported | Supported | Not supported | Not supported | <ul><li>Mainstream databases</li><li>SQL text</li><li>CSV files</li></ul> |
20+
| DataX | Not supported | Supported | Not supported | Not supported | Many data sources are supported. For more information, see [official introduction](https://github.com/alibaba/DataX/blob/master/introduction.md) |
21+
| Canal | Supported | Supported | Supported | Not supported | Many data sources are supported. For more information, see [official introduction](https://github.com/alibaba/canal/wiki) |
22+
| Flink CDC | Supported | Supported | Supported | Not supported | Many data sources are supported. For more information, see [official introduction](https://github.com/apache/flink-cdc) |
23+
| SeaTunnel | Supported | Supported | Supported | Not supported | Many data sources are supported. For more information, see [official introduction](https://seatunnel.apache.org/docs/about/) |
24+
25+
## OMS
26+
27+
OMS is a service that supports data interaction between a homogeneous or heterogeneous data source and OceanBase Database. OMS provides the capabilities for online migration of existing data and real-time synchronization of incremental data.
28+
29+
OMS Community Edition provides a visualized and centralized management platform. You can migrate data in real time with simple configurations. OMS Community Edition aims to implement real-time data migration and synchronization from homogeneous or heterogeneous databases to OceanBase Database with low risks, low costs, and high efficiency.
30+
31+
For more information, see [OceanBase Migration Service](https://en.oceanbase.com/docs/oms-en).
32+
33+
## OBLogProxy
34+
35+
OBLogProxy is the incremental log proxy service of OceanBase Database. It establishes connections with OceanBase Database to read incremental logs and provides downstream services with change data capture (CDC) capabilities.
36+
37+
OBLogProxy supports the following two modes:
38+
39+
* The binlog mode of OBLogProxy is designed for compatibility with MySQL binlogs. It allows you to use MySQL binlog incremental parsing tools to synchronize OceanBase Database logs in real time. Thereby, you can smoothly use MySQL binlog tools with OceanBase Database.
40+
41+
* OBLogProxy in CDC mode subscribes to data changes in OceanBase Database and synchronizes these data changes to downstream services in real time for real-time or quasi-realtime data replication and synchronization.
42+
43+
For more information, see [OBLogProxy](https://en.oceanbase.com/docs/common-oceanbase-database-10000000001239311).
44+
45+
## OBLOADER & OBDUMPER
46+
47+
OBLOADER & OBDUMPER are client tools that are developed in Java. At present, they apply only to OceanBase Database. You can use OBLOADER to import the definition files and table data files of database objects in storage media to OceanBase Database, or use OBDUMPER to export them to storage media.
48+
49+
We recommend that you use OBLOADER in combination with OBDUMPER. OBLOADER is compatible with the CSV files exported by using client tools such as mysqldump and Mydumper. Therefore, you can use OBLOADER in data migration. You can integrate OBDUMPER into a database O&M system for logical backup. OBLOADER provides various types of built-in data preprocessing functions to improve its data import performance. The automatic fault tolerance mechanism ensures data import stability. Rich monitoring information is provided for you to observe the import performance and progress in real time.
50+
51+
For more information, see [OceanBase Loader and Dumper](https://en.oceanbase.com/docs/obloader-obdumper-en).
52+
53+
## SQL statements for data migration
54+
55+
Executing SQL statements is a common and simple way to export data to an external file or import data from an external file to a database. You can also use SQL statements to migrate data from one table to another and process the data accordingly.
56+
57+
Supported statements include SELECT INTO OUTFILE, LOAD DATA, and INSERT INTO. Direct load is also supported to accelerate data import. For more information, see 'Use SQL statements for data migration' in 'migration-and-synchronization-through-sql'.
58+
59+
## Other migration tools
60+
61+
### Canal
62+
63+
Canal parses the incremental logs of a MySQL database and provides incremental data subscription and consumption.
64+
65+
In its early days, Alibaba needed to synchronize data between the two IDCs deployed in China and the United States, and the incremental data was collected based on business triggers. Since 2010, the company has gradually switched the method to obtaining and synchronizing incremental data by parsing database logs, giving rise to a large number of database services that are based on the subscription and consumption of incremental logs. Canal supports using a MySQL 5.1.x, 5.5.x, 5.6.x, 5.7.x, or 8.0.x database as the source.
66+
67+
Canal works in the following way:
68+
69+
1. Canal disguises itself as a MySQL slave by simulating the communication protocol of the MySQL slave, and sends a dump request to the MySQL master.
70+
71+
2. The MySQL master receives the dump request, and pushes binary logs to the slave, which is Canal.
72+
73+
3. Canal parses the binary logs into a stream of bytes.
74+
75+
For more information, see the [introduction to Canal](https://github.com/alibaba/canal?tab=readme-ov-file).
76+
77+
### Flink CDC
78+
79+
Flink CDC is a streaming data processing technology based on Flink.
80+
81+
It monitors and captures changes of the data source, extracts the changes into the data streams of Flink, and sends the data streams to downstream systems in real time, so that downstream systems can process the data changes in a timely manner. Flink CDC supports multiple data sources, such as relational databases, NoSQL databases, and message queues.
82+
83+
For more information, see the [introduction to Flink CDC](https://github.com/apache/flink-cdc).
84+
85+
### DataX
86+
87+
DataX is a tool for offline synchronization between heterogeneous data sources. You can use it to stably and efficiently synchronize data between many types of heterogeneous data sources, such as relational databases (like MySQL, Oracle, and OceanBase Database), HDFS, Hive, ODPS, HBase, and FTP.
88+
89+
![DataX architecture](/img/user_manual/quick_starts_and_hands_on_practices_in_english/chapter_04_migration_and_synchronization_oceanbase/02_ecological_tools_of_migration_synchronization/001.png)
90+
91+
To address issues of data synchronization between heterogeneous data sources in a complex mesh topology, DataX introduces the star topology and serves as the transmission hub that connects to various data sources. This way, you can connect a new data source to DataX and start synchronizing data between the new data source and the existing data sources right away.
92+
93+
For more information, see [the introduction of DataX](https://github.com/alibaba/DataX/blob/master/introduction.md).
94+
95+
### SeaTunnel
96+
97+
SeaTunnel is an easy-to-use distributed data integration platform that supports real-time synchronization of massive amounts of data with ultra-high performance. It can stably and efficiently synchronize tens of billions of data records every day and has been deployed by nearly 100 enterprises for their production.
98+
99+
SeaTunnel is specifically developed for data integration and synchronization, and solves the following data integration challenges:
100+
101+
* Diverse data sources: While hundreds of types of incompatible data sources are in use in the industry, emerging technologies keep bringing new data source types. Users require a tool that efficiently supports most if not all of the data sources.
102+
103+
* Complex synchronization scenarios: For example, offline full synchronization, offline incremental synchronization, CDC data synchronization, real-time synchronization, and database synchronization are involved.
104+
105+
* High resource consumption: Existing data integration and synchronization tools often consume a great amount of computing resources or Java Database Connectivity (JDBC) resources for real-time synchronization of numerous small tables, which increases the resource costs of enterprises.
106+
107+
* Lack of quality monitoring: Data loss and duplication are common issues during the process of a data integration or synchronization task, which lacks monitoring, making it impossible to observe the data quality of the task.
108+
109+
* Complex technology stacks: Enterprises use different technical components, therefore users must develop corresponding synchronization programs for data integration.
110+
111+
* Difficulties in management and maintenance: As different underlying technical components, such as Flink and Spark, are used, offline and real-time synchronization processes are often separately developed and managed, making the synchronization management and maintenance more difficult.
112+
113+
The following diagram describes the workflow of SeaTunnel.
114+
115+
![SeaTunnel workflow](/img/user_manual/quick_starts_and_hands_on_practices_in_english/chapter_04_migration_and_synchronization_oceanbase/02_ecological_tools_of_migration_synchronization/002.png)
116+
117+
For more information, see the [introduction to SeaTunnel](https://seatunnel.apache.org/docs/about/).

0 commit comments

Comments
 (0)