You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: en_US/data-integration/snowflake.md
+24-26Lines changed: 24 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,7 +24,7 @@ EMQX utilizes the rule engine and Sink to forward device events and data to Snow
24
24
25
25
Snowpipe Streaming is currently a [preview feature](https://docs.snowflake.com/en/release-notes/preview-features) in Snowflake. It is available only for accounts hosted on AWS.
26
26
27
-
:::
27
+
:::
28
28
29
29
After events and message data are written to the Snowflake, they can be accessed for a variety of business and technical purposes, including:
30
30
@@ -101,7 +101,7 @@ The script automatically downloads the Snowflake ODBC `.deb` installation packag
101
101
Run the following command to view the configurations in the `/etc/odbc.ini` file:
-- Create a pipe for aggregated mode that copies from the stage
294
294
CREATE PIPE IF NOT EXISTS testdatabase.public.emqx AS
295
295
COPY INTO testdatabase.public.emqx
296
296
FROM @testdatabase.public.emqx
297
297
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
298
-
298
+
299
299
-- Create a pipe for streaming mode (direct ingestion)
300
300
CREATE PIPE IF NOT EXISTS testdatabase.public.emqxstreaming AS
301
301
COPY INTO testdatabase.public.emqx (
@@ -312,8 +312,7 @@ This includes:
312
312
$1:publish_received_at::TIMESTAMP_LTZ
313
313
FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING'))
314
314
);
315
-
MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE;
316
-
315
+
317
316
```
318
317
319
318
- The `COPY INTO` inside the pipe ensures Snowflake automatically loads staged or streamed data into your table.
@@ -326,7 +325,7 @@ This includes:
326
325
CREATE USER IF NOT EXISTS snowpipeuser
327
326
PASSWORD = 'Snowpipeuser99'
328
327
MUST_CHANGE_PASSWORD = FALSE;
329
-
328
+
330
329
-- Bind the RSA public key to the user
331
330
ALTER USER snowpipeuser SET RSA_PUBLIC_KEY = '
332
331
<YOUR_PUBLIC_KEY_CONTENTS_LINE_1>
@@ -348,19 +347,19 @@ This includes:
348
347
349
348
```sql
350
349
CREATE OR REPLACE ROLE snowpipe;
351
-
350
+
352
351
-- Grant usage and read/write permissions
353
352
GRANT USAGE ON DATABASE testdatabase TO ROLE snowpipe;
354
353
GRANT USAGE ON SCHEMA testdatabase.public TO ROLE snowpipe;
355
354
GRANT INSERT, SELECT ON testdatabase.public.emqx TO ROLE snowpipe;
356
-
355
+
357
356
-- Aggregated mode requires access to stage and pipe
358
357
GRANT READ, WRITE ON STAGE testdatabase.public.emqx TO ROLE snowpipe;
359
358
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqx TO ROLE snowpipe;
360
-
359
+
361
360
-- Streaming mode requires permissions on the streaming pipe
362
361
GRANT OPERATE, MONITOR ON PIPE testdatabase.public.emqxstreaming TO ROLE snowpipe;
363
-
362
+
364
363
-- Link role to the user and set it as default
365
364
GRANT ROLE snowpipe TO USER snowpipeuser;
366
365
ALTER USER snowpipeuser SET DEFAULT_ROLE = snowpipe;
@@ -395,7 +394,7 @@ If you plan to use the aggregated upload mode in your Snowflake Sink, you need t
395
394
- Or configure it in`/etc/odbc.ini`;
396
395
397
396
- If using key-pair authentication instead, leave this field blank.
398
-
397
+
399
398
::: tip
400
399
401
400
Use either Password or Private Key forauthentication, not both. If neither is configured here, ensure the appropriate credentials are setin`/etc/odbc.ini`.
@@ -404,14 +403,14 @@ If you plan to use the aggregated upload mode in your Snowflake Sink, you need t
404
403
405
404
- **Private Key Path**: The absolute file path to the private RSA key used for authenticating with Snowflake via ODBC. This path must be consistent across all nodes of the cluster. For example:
406
405
`/etc/emqx/certs/snowflake_rsa_key.private.pem`.
407
-
406
+
408
407
- **Private Key Password**: The password used to decrypt the private RSA key file, if the key is encrypted. Leave this field blank if the key was generated without encryption (i.e., with the `-nocrypt` option in OpenSSL).
409
-
408
+
410
409
- **Proxy**: Configuration settings for connecting to Snowflake through an HTTP proxy server. HTTPS proxies are **not** supported. By default, no proxy is used. To enable proxy support, selectthe`Enable Proxy` and provide the following:
411
410
412
411
- **Proxy Host**: The hostname or IP address of the proxy server.
413
412
- **Proxy Port**: The port number used by the proxy server.
414
-
413
+
415
414
6. If you want to establish an encrypted connection, click the **Enable TLS** toggle switch. For more information about TLS connection, see [TLS for External Resource Access](../network/overview.md/#tls-for-external-resource-access). TLS must be enabled for streaming mode, as communication is over HTTPS.
416
415
417
416
7. Advanced settings (optional): See [Advanced Settings](#advanced-settings).
@@ -475,9 +474,9 @@ This section demonstrates how to create a rule in EMQX to process messages (e.g.
475
474
476
475
:::
477
476
::: tip
478
-
479
-
For Snowflake integration, it is important that the selected fields exactly match the number of columns and their names of the table defined in Snowflake, so avoid adding extra fields or selecting from `*`.
480
-
477
+
478
+
For Snowflake integration, it is important that the selected fields exactly match the number of columns and their names of the table defined in Snowflake, so avoid adding extra fields or selecting from `*`.
479
+
481
480
:::
482
481
483
482
@@ -597,4 +596,3 @@ This section delves into the advanced configuration options available for the Sn
597
596
|**Query Mode**| Allows you to choose between `synchronous` or `asynchronous` request modes to optimize message transmission according to different requirements. In asynchronous mode, writing to Snowflake does not block the MQTT message publishing process. However, this may lead to clients receiving messages before they arrive at Snowflake. |`Asynchronous`|
598
597
|**Batch Size**| Specifies the maximum size of data batches transmitted from EMQX to Snowflake in a single transfer operation. By adjusting the size, you can fine-tune the efficiency and performance of data transfer between EMQX and Snowflake.<br />If the "Batch Size" is set to "1," data records are sent individually, without being grouped into batches. |`100`|
599
598
|**Inflight Window**|"In-flight queue requests" refer to requests that have been initiated but have not yet received a response or acknowledgment. This setting controls the maximum number of in-flight queue requests that can exist simultaneously during Sink communication with Snowflake. <br/>When **Request Mode** is set to `asynchronous`, the "Request In-flight Queue Window" parameter becomes particularly important. If strict sequential processing of messages from the same MQTT client is crucial, then this value should be set to `1`. |`100`|
0 commit comments