Skip to content

Commit 2484c4a

Browse files
authored
Merge pull request #3211 from emqx/20251020-r60-fix-snowflake-typo
fix(snowflake): remove typo line
2 parents 3bc37f5 + 856acf0 commit 2484c4a

File tree

2 files changed

+42
-45
lines changed

2 files changed

+42
-45
lines changed

en_US/data-integration/snowflake.md

Lines changed: 24 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ EMQX utilizes the rule engine and Sink to forward device events and data to Snow
2424

2525
Snowpipe Streaming is currently a [preview feature](https://docs.snowflake.com/en/release-notes/preview-features) in Snowflake. It is available only for accounts hosted on AWS.
2626

27-
:::
27+
:::
2828

2929
After events and message data are written to the Snowflake, they can be accessed for a variety of business and technical purposes, including:
3030

@@ -101,7 +101,7 @@ The script automatically downloads the Snowflake ODBC `.deb` installation packag
101101
Run the following command to view the configurations in the `/etc/odbc.ini` file:
102102

103103
```
104-
emqx@emqx-0:~$ cat /etc/odbc.ini
104+
emqx@emqx-0:~$ cat /etc/odbc.ini
105105
106106
[snowflake]
107107
Description=SnowflakeDB
@@ -117,7 +117,7 @@ snowflake = SnowflakeDSIIDriver
117117
Run the following command to view the configurations in the `/etc/odbcinst.ini` file:
118118

119119
```
120-
emqx@emqx-0:~$ cat /etc/odbcinst.ini
120+
emqx@emqx-0:~$ cat /etc/odbcinst.ini
121121
122122
[ODBC Driver 18 for SQL Server]
123123
Description=Microsoft ODBC Driver 18 for SQL Server
@@ -171,13 +171,13 @@ To install and configure the Snowflake ODBC driver on macOS, follow these steps:
171171
[ODBC]
172172
Trace=no
173173
TraceFile=
174-
174+
175175
[ODBC Drivers]
176176
Snowflake = Installed
177-
177+
178178
[ODBC Data Sources]
179179
snowflake = Snowflake
180-
180+
181181
[Snowflake]
182182
Driver = /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib
183183
EOF
@@ -273,29 +273,29 @@ This includes:
273273
274274
```sql
275275
USE ROLE accountadmin;
276-
276+
277277
-- Create a database to store your data (if not exists)
278278
CREATE DATABASE IF NOT EXISTS testdatabase;
279-
279+
280280
-- Create a table to receive MQTT data
281281
CREATE OR REPLACE TABLE testdatabase.public.emqx (
282282
clientid STRING,
283283
topic STRING,
284284
payload STRING,
285285
publish_received_at TIMESTAMP_LTZ
286286
);
287-
287+
288288
-- Create a Snowflake stage for uploading files (aggregated mode only)
289289
CREATE STAGE IF NOT EXISTS testdatabase.public.emqx
290290
FILE_FORMAT = (TYPE = CSV PARSE_HEADER = TRUE FIELD_OPTIONALLY_ENCLOSED_BY = '"')
291291
COPY_OPTIONS = (ON_ERROR = CONTINUE PURGE = TRUE);
292-
292+
293293
-- Create a pipe for aggregated mode that copies from the stage
294294
CREATE PIPE IF NOT EXISTS testdatabase.public.emqx AS
295295
COPY INTO testdatabase.public.emqx
296296
FROM @testdatabase.public.emqx
297297
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
298-
298+
299299
-- Create a pipe for streaming mode (direct ingestion)
300300
CREATE PIPE IF NOT EXISTS testdatabase.public.emqxstreaming AS
301301
COPY INTO testdatabase.public.emqx (
@@ -312,8 +312,7 @@ This includes:
312312
$1:publish_received_at::TIMESTAMP_LTZ
313313
FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING'))
314314
);
315-
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
316-
315+
317316
```
318317
319318
- The `COPY INTO` inside the pipe ensures Snowflake automatically loads staged or streamed data into your table.
@@ -326,7 +325,7 @@ This includes:
326325
CREATE USER IF NOT EXISTS snowpipeuser
327326
PASSWORD = 'Snowpipeuser99'
328327
MUST_CHANGE_PASSWORD = FALSE;
329-
328+
330329
-- Bind the RSA public key to the user
331330
ALTER USER snowpipeuser SET RSA_PUBLIC_KEY = '
332331
<YOUR_PUBLIC_KEY_CONTENTS_LINE_1>
@@ -348,19 +347,19 @@ This includes:
348347
349348
```sql
350349
CREATE OR REPLACE ROLE snowpipe;
351-
350+
352351
-- Grant usage and read/write permissions
353352
GRANT USAGE ON DATABASE testdatabase TO ROLE snowpipe;
354353
GRANT USAGE ON SCHEMA testdatabase.public TO ROLE snowpipe;
355354
GRANT INSERT, SELECT ON testdatabase.public.emqx TO ROLE snowpipe;
356-
355+
357356
-- Aggregated mode requires access to stage and pipe
358357
GRANT READ, WRITE ON STAGE testdatabase.public.emqx TO ROLE snowpipe;
359358
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqx TO ROLE snowpipe;
360-
359+
361360
-- Streaming mode requires permissions on the streaming pipe
362361
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqxstreaming TO ROLE snowpipe;
363-
362+
364363
-- Link role to the user and set it as default
365364
GRANT ROLE snowpipe TO USER snowpipeuser;
366365
ALTER USER snowpipeuser SET DEFAULT_ROLE = snowpipe;
@@ -395,7 +394,7 @@ If you plan to use the aggregated upload mode in your Snowflake Sink, you need t
395394
- Or configure it in `/etc/odbc.ini`;
396395
397396
- If using key-pair authentication instead, leave this field blank.
398-
397+
399398
::: tip
400399
401400
Use either Password or Private Key for authentication, not both. If neither is configured here, ensure the appropriate credentials are set in `/etc/odbc.ini`.
@@ -404,14 +403,14 @@ If you plan to use the aggregated upload mode in your Snowflake Sink, you need t
404403
405404
- **Private Key Path**: The absolute file path to the private RSA key used for authenticating with Snowflake via ODBC. This path must be consistent across all nodes of the cluster. For example:
406405
`/etc/emqx/certs/snowflake_rsa_key.private.pem`.
407-
406+
408407
- **Private Key Password**: The password used to decrypt the private RSA key file, if the key is encrypted. Leave this field blank if the key was generated without encryption (i.e., with the `-nocrypt` option in OpenSSL).
409-
408+
410409
- **Proxy**: Configuration settings for connecting to Snowflake through an HTTP proxy server. HTTPS proxies are **not** supported. By default, no proxy is used. To enable proxy support, select the `Enable Proxy` and provide the following:
411410
412411
- **Proxy Host**: The hostname or IP address of the proxy server.
413412
- **Proxy Port**: The port number used by the proxy server.
414-
413+
415414
6. If you want to establish an encrypted connection, click the **Enable TLS** toggle switch. For more information about TLS connection, see [TLS for External Resource Access](../network/overview.md/#tls-for-external-resource-access). TLS must be enabled for streaming mode, as communication is over HTTPS.
416415
417416
7. Advanced settings (optional): See [Advanced Settings](#advanced-settings).
@@ -475,9 +474,9 @@ This section demonstrates how to create a rule in EMQX to process messages (e.g.
475474
476475
:::
477476
::: tip
478-
479-
For Snowflake integration, it is important that the selected fields exactly match the number of columns and their names of the table defined in Snowflake, so avoid adding extra fields or selecting from `*`.
480-
477+
478+
For Snowflake integration, it is important that the selected fields exactly match the number of columns and their names of the table defined in Snowflake, so avoid adding extra fields or selecting from `*`.
479+
481480
:::
482481
483482
@@ -597,4 +596,3 @@ This section delves into the advanced configuration options available for the Sn
597596
| **Query Mode** | Allows you to choose between `synchronous` or `asynchronous` request modes to optimize message transmission according to different requirements. In asynchronous mode, writing to Snowflake does not block the MQTT message publishing process. However, this may lead to clients receiving messages before they arrive at Snowflake. | `Asynchronous` |
598597
| **Batch Size** | Specifies the maximum size of data batches transmitted from EMQX to Snowflake in a single transfer operation. By adjusting the size, you can fine-tune the efficiency and performance of data transfer between EMQX and Snowflake.<br />If the "Batch Size" is set to "1," data records are sent individually, without being grouped into batches. | `100` |
599598
| **Inflight Window** | "In-flight queue requests" refer to requests that have been initiated but have not yet received a response or acknowledgment. This setting controls the maximum number of in-flight queue requests that can exist simultaneously during Sink communication with Snowflake. <br/>When **Request Mode** is set to `asynchronous`, the "Request In-flight Queue Window" parameter becomes particularly important. If strict sequential processing of messages from the same MQTT client is crucial, then this value should be set to `1`. | `100` |
600-

zh_CN/data-integration/snowflake.md

Lines changed: 18 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ EMQX 利用规则引擎和 Sink 将设备事件和数据转发到 Snowflake。
2424

2525
Snowpipe Streaming 当前是 Snowflake 的[预览功能](https://docs.snowflake.com/en/release-notes/preview-features),仅适用于部署在 AWS 上的账户。
2626

27-
:::
27+
:::
2828

2929
当事件和消息数据写入 Snowflake 后,可用于各种业务和技术用途,包括:
3030

@@ -118,13 +118,13 @@ scripts/install-snowflake-driver.sh
118118
[ODBC]
119119
Trace=no
120120
TraceFile=
121-
121+
122122
[ODBC Drivers]
123123
Snowflake = Installed
124-
124+
125125
[ODBC Data Sources]
126126
snowflake = Snowflake
127-
127+
128128
[Snowflake]
129129
Driver = /opt/snowflake/snowflakeodbc/lib/universal/libSnowflake.dylib
130130
EOF
@@ -219,29 +219,29 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
219219
220220
```sql
221221
USE ROLE accountadmin;
222-
222+
223223
-- 创建用于存储数据的数据库(如果不存在)
224224
CREATE DATABASE IF NOT EXISTS testdatabase;
225-
225+
226226
-- 创建用于接收 MQTT 数据的表
227227
CREATE OR REPLACE TABLE testdatabase.public.emqx (
228228
clientid STRING,
229229
topic STRING,
230230
payload STRING,
231231
publish_received_at TIMESTAMP_LTZ
232232
);
233-
233+
234234
-- 创建用于聚合模式的存储区,用于上传文件
235235
CREATE STAGE IF NOT EXISTS testdatabase.public.emqx
236236
FILE_FORMAT = (TYPE = CSV PARSE_HEADER = TRUE FIELD_OPTIONALLY_ENCLOSED_BY = '"')
237237
COPY_OPTIONS = (ON_ERROR = CONTINUE PURGE = TRUE);
238-
238+
239239
-- 创建用于聚合模式的管道,从存储区中复制数据
240240
CREATE PIPE IF NOT EXISTS testdatabase.public.emqx AS
241241
COPY INTO testdatabase.public.emqx
242242
FROM @testdatabase.public.emqx
243243
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
244-
244+
245245
-- 创建用于流式模式的管道,直接摄取数据
246246
CREATE PIPE IF NOT EXISTS testdatabase.public.emqxstreaming AS
247247
COPY INTO testdatabase.public.emqx (
@@ -258,8 +258,7 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
258258
$1:publish_received_at::TIMESTAMP_LTZ
259259
FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING'))
260260
);
261-
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
262-
261+
263262
```
264263
265264
- `COPY INTO` 语句确保 Snowflake 能自动将存储区或 Streaming 中的数据加载到目标表。
@@ -272,7 +271,7 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
272271
CREATE USER IF NOT EXISTS snowpipeuser
273272
PASSWORD = 'Snowpipeuser99'
274273
MUST_CHANGE_PASSWORD = FALSE;
275-
274+
276275
-- 将 RSA 公钥绑定到该用户
277276
ALTER USER snowpipeuser SET RSA_PUBLIC_KEY = '
278277
<YOUR_PUBLIC_KEY_CONTENTS_LINE_1>
@@ -294,19 +293,19 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
294293
295294
```sql
296295
CREATE OR REPLACE ROLE snowpipe;
297-
296+
298297
-- 授权数据库和表的使用与读写权限
299298
GRANT USAGE ON DATABASE testdatabase TO ROLE snowpipe;
300299
GRANT USAGE ON SCHEMA testdatabase.public TO ROLE snowpipe;
301300
GRANT INSERT, SELECT ON testdatabase.public.emqx TO ROLE snowpipe;
302-
301+
303302
-- 聚合模式需要访问存储区和管道
304303
GRANT READ, WRITE ON STAGE testdatabase.public.emqx TO ROLE snowpipe;
305304
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqx TO ROLE snowpipe;
306-
305+
307306
-- 流式模式需要访问流式管道
308307
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqxstreaming TO ROLE snowpipe;
309-
308+
310309
-- 将角色授予用户,并设置为默认角色
311310
GRANT ROLE snowpipe TO USER snowpipeuser;
312311
ALTER USER snowpipeuser SET DEFAULT_ROLE = snowpipe;
@@ -341,7 +340,7 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
341340
- 或在系统的 `/etc/odbc.ini` 文件中配置;
342341
343342
- 如果使用密钥对认证(Key-pair authentication),则无需提供密码。
344-
343+
345344
::: tip
346345
347346
使用密码或私钥进行身份验证,而不是两者兼用。如果此处未配置这两种方式,请确保在 `/etc/odbc.ini` 中设置了适当的凭证。
@@ -356,7 +355,7 @@ openssl rsa -in snowflake_rsa_key.private.pem -pubout -out snowflake_rsa_key.pub
356355
357356
- **代理主机**:代理服务器的主机名或 IP 地址。
358357
- **代理端口**:代理服务器使用的端口号。
359-
358+
360359
6. 如果您想建立一个加密连接,单击**启用 TLS** 切换按钮。有关 TLS 连接的更多信息,请参见[启用 TLS 加密访问外部资源](../network/overview.md/#tls-for-external-resource-access)。流式模式必须启用 TLS,因为通信是通过 HTTPS 进行的。
361360
362361
7. 高级配置(可选),请参考[高级设置](#高级设置)。
@@ -546,4 +545,4 @@ mqttx pub -i emqx_c -t t/1 -m '{ "msg": "Hello Snowflake" }'
546545
| **缓存队列最大长度** | 指定每个缓冲工作进程在 Snowflake Sink 中可以缓冲的最大字节数。缓冲工作进程临时存储数据,以便更有效地处理数据流。在系统性能和数据传输要求下,可以调整此值。 | `256` MB |
547546
| **请求模式** | 允许您在 `同步``异步` 请求模式之间进行选择,以根据不同需求优化消息传输。在异步模式下,写入 Snowflake 不会阻塞 MQTT 消息的发布过程,但这可能会导致客户端在消息到达 Snowflake 之前就收到消息。 | `异步` |
548547
| **最大批量请求大小** | 指定从 EMQX 向 Snowflake 传输数据时的单次传输最大数据批大小。通过调整批处理大小,您可以微调 EMQX 和 Snowflake 之间数据传输的效率和性能。<br />如果将“批处理大小”设置为 "1",则数据记录将单独发送,而不会被分组为批处理。 | `100` |
549-
| **请求飞行队列窗口** | “在途请求队列”指已启动但尚未收到响应或确认的请求。此设置控制 Sink 与 Snowflake 通信期间同时存在的最大在途请求数量。<br/>**请求模式**设置为 `异步` 时,“在途请求队列窗口”参数尤其重要。如果需要严格按顺序处理来自同一 MQTT 客户端的消息,则应将此值设置为 `1`| `100` |
548+
| **请求飞行队列窗口** | “在途请求队列”指已启动但尚未收到响应或确认的请求。此设置控制 Sink 与 Snowflake 通信期间同时存在的最大在途请求数量。<br/>**请求模式**设置为 `异步` 时,“在途请求队列窗口”参数尤其重要。如果需要严格按顺序处理来自同一 MQTT 客户端的消息,则应将此值设置为 `1`| `100` |

0 commit comments

Comments
 (0)