diff --git a/docs/ingest/etl/index.md b/docs/ingest/etl/index.md index 08fd4486..987a380f 100644 --- a/docs/ingest/etl/index.md +++ b/docs/ingest/etl/index.md @@ -145,6 +145,13 @@ Load data from database systems. application logs, website clickstreams, and IoT telemetry data, for machine learning (ML), analytics, and other applications. +- {ref}`mqtt` + + MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT). + It is designed as an extremely lightweight publish/subscribe messaging transport + that is ideal for connecting remote devices with a small code footprint and minimal + network bandwidth. + - {ref}`risingwave` RisingWave is a stream processing and management platform that allows configuring diff --git a/docs/integrate/index.md b/docs/integrate/index.md index 4603241c..12b34a32 100644 --- a/docs/integrate/index.md +++ b/docs/integrate/index.md @@ -44,6 +44,7 @@ marquez/index meltano/index metabase/index mongodb/index +mqtt/index mysql/index n8n/index nifi/index diff --git a/docs/integrate/mongodb/tutorial.md b/docs/integrate/mongodb/tutorial.md index 2c31bc2d..d298ad2c 100644 --- a/docs/integrate/mongodb/tutorial.md +++ b/docs/integrate/mongodb/tutorial.md @@ -1,4 +1,5 @@ (mongodb-tutorial)= +(migrating-mongodb)= # Import data from MongoDB In this quick tutorial, you'll use the [CrateDB Toolkit MongoDB I/O subsystem] diff --git a/docs/integrate/mqtt/index.md b/docs/integrate/mqtt/index.md new file mode 100644 index 00000000..4a451aca --- /dev/null +++ b/docs/integrate/mqtt/index.md @@ -0,0 +1,67 @@ +(mqtt)= +# MQTT + +```{div} .float-right +[![MQTT logo](https://mqtt.org/assets/img/mqtt-logo.svg){width=180px loading=lazy}][MQTT] +``` +```{div} .clearfix +``` + +:::{rubric} About +::: + +[MQTT] is an OASIS standard messaging protocol for the Internet of Things (IoT). + +It is designed as an extremely lightweight publish/subscribe messaging transport +that is ideal for connecting remote devices with a small code footprint and minimal +network bandwidth. + +MQTT today is used in a wide variety of industries, such as automotive, manufacturing, +telecommunications, and oil and gas. It enables efficient, reliable messaging between +devices and backends over constrained networks. + +:::{rubric} Synopsis +::: + +Use LorryStream to receive JSON data from an MQTT topic, continuously loading +records into CrateDB. +```shell +uvx --from=lorrystream lorry relay \ + "mqtt://localhost/testdrive/%23?content-type=json" \ + "crate://localhost/?table=testdrive" +``` + +:::{rubric} Learn +::: + +[LorryStream] is a lightweight and polyglot stream-processing library, used as a +data backplane, message relay, or pipeline subsystem. +[Node-RED] is a workflow automation tool that allows you to orchestrate message flows +and transformations via a comfortable web interface. + +::::{grid} + +:::{grid-item-card} Tutorial: Use LorryStream +:link: mqtt-tutorial +:link-type: ref +How to load data from an MQTT topic into CrateDB using LorryStream. +::: + +:::{grid-item-card} Tutorial: Use Node-RED +:link: https://community.cratedb.com/t/ingesting-mqtt-messages-into-cratedb-using-node-red/803 +:link-type: url +Ingesting MQTT messages into CrateDB using Node-RED. +::: + +:::: + +:::{toctree} +:maxdepth: 1 +:hidden: +Tutorial +::: + + +[LorryStream]: https://lorrystream.readthedocs.io/ +[MQTT]: https://mqtt.org/ +[Node-RED]: https://nodered.org/ diff --git a/docs/integrate/mqtt/tutorial.md b/docs/integrate/mqtt/tutorial.md new file mode 100644 index 00000000..0af449c4 --- /dev/null +++ b/docs/integrate/mqtt/tutorial.md @@ -0,0 +1,102 @@ +(mqtt-tutorial)= +# Load data from an MQTT topic into CrateDB + +The tutorial will walk you through starting the [Eclipse Mosquitto] broker and CrateDB, +publishing JSON data to an MQTT topic, subscribing to the topic to relay +data into a CrateDB table continuously, and validating that the data has +been stored successfully. +The data transfer is supported by the [LorryStream MQTT source] data +pipeline element. + +## Prerequisites + +Docker is used for running all components. This approach works consistently +across Linux, macOS, and Windows. + +Alternatively, you can use Podman. You can also use a different MQTT broker such as +EMQX, HiveMQ, VerneMQ, or RabbitMQ. Azure IoT Hub speaks MQTT as well, but with +protocol and authentication specifics; adjust settings accordingly. + +Create a shared network. +```shell +docker network create cratedb-demo +``` + +Start CrateDB. +```shell +docker run --name=cratedb --rm -it --network=cratedb-demo \ + --publish=4200:4200 --publish=5432:5432 \ + --env=CRATE_HEAP_SIZE=2g docker.io/crate -Cdiscovery.type=single-node +``` + +Start Mosquitto. +```shell +docker run --name=mosquitto --rm -it --network=cratedb-demo \ + --publish=1883:1883 docker.io/eclipse-mosquitto \ + mosquitto -c /mosquitto-no-auth.conf +``` +> Note: This broker configuration allows anonymous access for demonstration purposes only. +> Do not expose it to untrusted networks. For production, configure authentication/TLS. + +Prepare shortcuts for the CrateDB shell, LorryStream, and the Mosquitto client +programs. + +::::{tab-set} + +:::{tab-item} Linux and macOS +To make the settings persistent, add them to your shell profile (`~/.profile`). +```shell +alias crash="docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash" +alias lorry="docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry" +alias mosquitto_pub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_pub" +alias mosquitto_sub="docker run --rm -i --network=cratedb-demo docker.io/eclipse-mosquitto mosquitto_sub" +``` +::: +:::{tab-item} Windows PowerShell +To make the settings persistent, add them to your PowerShell profile (`$PROFILE`). +```powershell +function crash { docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash @args } +function lorry { docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry @args } +function mosquitto_pub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub @args } +function mosquitto_sub { docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub @args } +``` +::: +:::{tab-item} Windows Command +```shell +doskey crash=docker run --rm -it --network=cratedb-demo ghcr.io/crate/cratedb-toolkit crash $* +doskey lorry=docker run --rm -i --network=cratedb-demo ghcr.io/daq-tools/lorrystream lorry $* +doskey mosquitto_pub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_pub $* +doskey mosquitto_sub=docker run --rm -i --network=cratedb-demo eclipse-mosquitto mosquitto_sub $* +``` +::: + +:::: + +## Usage + +Subscribe to all MQTT topics on the broker to monitor any traffic. +```shell +mosquitto_sub -h mosquitto -t "#" -v +``` + +Invoke the data transfer pipeline. +```shell +lorry relay \ + "mqtt://mosquitto/testdrive/%23?content-type=json" \ + "crate://cratedb/?table=testdrive" +``` + +Publish a JSON message to an MQTT topic. +```shell +echo '{"temperature": 42.84, "humidity": 83.1}' | \ + mosquitto_pub -h mosquitto -t testdrive/channel1 -s +``` + +Inspect data stored in CrateDB. +```shell +crash --hosts cratedb -c "SELECT * FROM testdrive" +``` + + +[Eclipse Mosquitto]: https://mosquitto.org/ +[LorryStream MQTT source]: https://lorrystream.readthedocs.io/source/mqtt.html