Skip to content

Commit 2bc2dc3

Browse files
amotlamotl
committed
Integrate/MQTT: Add entry point page and micro tutorial
The rig uses Eclipse Mosquitto and LorryStream to invoke the data transfer procedure, all based on using Docker.
1 parent 796f993 commit 2bc2dc3

File tree

4 files changed

+178
-0
lines changed

4 files changed

+178
-0
lines changed

‎docs/ingest/etl/index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,13 @@ Load data from database systems.
145145
application logs, website clickstreams, and IoT telemetry data, for machine
146146
learning (ML), analytics, and other applications.
147147

148+
- {ref}`mqtt`
149+
150+
MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT).
151+
It is designed as an extremely lightweight publish/subscribe messaging transport
152+
that is ideal for connecting remote devices with a small code footprint and minimal
153+
network bandwidth.
154+
148155
- {ref}`risingwave`
149156

150157
RisingWave is a stream processing and management platform that allows configuring

‎docs/integrate/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ marquez/index
4444
meltano/index
4545
metabase/index
4646
mongodb/index
47+
mqtt/index
4748
mysql/index
4849
n8n/index
4950
nifi/index

‎docs/integrate/mqtt/index.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
(mqtt)=
2+
# MQTT
3+
4+
```{div} .float-right
5+
[![MQTT logo](https://mqtt.org/assets/img/mqtt-logo.svg){width=180px loading=lazy}][MQTT]
6+
```
7+
```{div} .clearfix
8+
```
9+
10+
:::{rubric} About
11+
:::
12+
13+
[MQTT] is an OASIS standard messaging protocol for the Internet of Things (IoT).
14+
15+
It is designed as an extremely lightweight publish/subscribe messaging transport
16+
that is ideal for connecting remote devices with a small code footprint and minimal
17+
network bandwidth.
18+
19+
MQTT today is used in a wide variety of industries, such as automotive, manufacturing,
20+
telecommunications, and oil and gas. It enables efficient, reliable messaging between
21+
devices and backends over constrained networks.
22+
23+
:::{rubric} Synopsis
24+
:::
25+
26+
Use LorryStream to receive JSON data from an MQTT topic, continuously loading
27+
records into CrateDB.
28+
```shell
29+
uvx --from=lorrystream lorry relay \
30+
"mqtt://localhost/testdrive/%23?content-type=json" \
31+
"crate://localhost/?table=testdrive"
32+
```
33+
34+
:::{rubric} Learn
35+
:::
36+
37+
[LorryStream] is a lightweight and polyglot stream-processing library, used as a
38+
data backplane, message relay, or pipeline subsystem.
39+
[Node-RED] is a workflow automation tool that allows you to orchestrate message flows
40+
and transformations via a comfortable web interface.
41+
42+
::::{grid}
43+
44+
:::{grid-item-card} Tutorial: Use LorryStream
45+
:link: mqtt-tutorial
46+
:link-type: ref
47+
How to load data from an MQTT topic into CrateDB using LorryStream.
48+
:::
49+
50+
:::{grid-item-card} Tutorial: Use Node-RED
51+
:link: https://community.cratedb.com/t/ingesting-mqtt-messages-into-cratedb-using-node-red/803
52+
:link-type: url
53+
Ingesting MQTT messages into CrateDB using Node-RED.
54+
:::
55+
56+
::::
57+
58+
:::{toctree}
59+
:maxdepth: 1
60+
:hidden:
61+
Tutorial <tutorial>
62+
:::
63+
64+
65+
[LorryStream]: https://lorrystream.readthedocs.io/
66+
[MQTT]: https://mqtt.org/
67+
[Node-RED]: https://nodered.org/

‎docs/integrate/mqtt/tutorial.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
(mqtt-tutorial)=
2+
3+
# Load data from an MQTT topic into CrateDB
4+
5+
The tutorial will walk you through starting the [Eclipse Mosquitto] broker and CrateDB,
6+
publishing JSON data to an MQTT topic, subscribing to the topic to relay
7+
data into a CrateDB table continuously, and validating that the data has
8+
been stored successfully.
9+
The data transfer is supported by the [LorryStream MQTT source] data
10+
pipeline element.
11+
12+
## Prerequisites
13+
14+
Docker is used for running all components. This approach works consistently
15+
across Linux, macOS, and Windows.
16+
17+
Alternatively, you can use Podman. You can also use a different MQTT broker such as
18+
EMQX, HiveMQ, VerneMQ, or RabbitMQ. Azure IoT Hub speaks MQTT as well, but with
19+
protocol and authentication specifics; adjust settings accordingly.
20+
21+
Create a shared network.
22+
```shell
23+
docker network create cratedb-demo
24+
```
25+
26+
Start CrateDB.
27+
```shell
28+
docker run --name=cratedb --rm -it --network=cratedb-demo \
29+
--publish=4200:4200 --publish=5432:5432 \
30+
--env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node
31+
```
32+
33+
Start Mosquitto.
34+
```shell
35+
docker run --name=mosquitto --rm -it --network=cratedb-demo \
36+
--publish=1883:1883 docker.io/eclipse-mosquitto \
37+
mosquitto -c /mosquitto-no-auth.conf
38+
```
39+
> Note: This broker configuration allows anonymous access for demonstration purposes only.
40+
> Do not expose it to untrusted networks. For production, configure authentication/TLS.
41+
42+
Prepare shortcuts for the CrateDB shell, LorryStream, and the Mosquitto client
43+
programs.
44+
45+
::::{tab-set}
46+
47+
:::{tab-item} Linux and macOS
48+
To make the settings persistent, add them to your Shell profile (`~/.profile`).
49+
```shell
50+
alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash"
51+
alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry"
52+
alias mosquitto_pub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub"
53+
alias mosquitto_sub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub"
54+
```
55+
:::
56+
:::{tab-item} Windows PowerShell
57+
To make the settings persistent, add them to your PowerShell profile (`$PROFILE`).
58+
```powershell
59+
function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args }
60+
function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args }
61+
function mosquitto_pub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub @args }
62+
function mosquitto_sub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub @args }
63+
```
64+
:::
65+
:::{tab-item} Windows Command
66+
```shell
67+
doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $*
68+
doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $*
69+
doskey mosquitto_pub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub $*
70+
doskey mosquitto_sub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub $*
71+
```
72+
:::
73+
74+
::::
75+
76+
## Usage
77+
78+
Subscribe to all MQTT topics on the broker to monitor any traffic.
79+
```shell
80+
mosquitto_sub -h mosquitto -t "#" -v
81+
```
82+
83+
Invoke the data transfer pipeline.
84+
```shell
85+
lorry relay \
86+
"mqtt://mosquitto/testdrive/%23?content-type=json" \
87+
"crate://cratedb/?table=testdrive"
88+
```
89+
90+
Publish a JSON message to an MQTT topic.
91+
```shell
92+
echo '{"temperature": 42.84, "humidity": 83.1}' | \
93+
mosquitto_pub -h mosquitto -t testdrive/channel1 -s
94+
```
95+
96+
Inspect data stored in CrateDB.
97+
```shell
98+
crash --hosts cratedb -c "SELECT * FROM testdrive"
99+
```
100+
101+
102+
[Eclipse Mosquitto]: https://mosquitto.org/
103+
[LorryStream MQTT source]: https://lorrystream.readthedocs.io/source/mqtt.html

0 commit comments

Comments
 (0)