Merge pull request #3209 from emqx/r601-message-queue-improvements

update(MQ): Message Queue improvements

Author: Meggielqk

en_US/cluster-linking/configuration.md

@@ -1,4 +1,4 @@
-# Configure Cluster Linking
+# Cluster Linking User Guide
 
 This page provides guidelines for configuring and managing the Cluster Linking feature through the EMQX Dashboard, configuration file, and REST API.
 

en_US/cluster-linking/quick-start.md

@@ -1,4 +1,4 @@
-# Quick Start with Cluster Linking
+# Cluster Linking Quick Start
 
 This page provides a quick start guide to setting up cluster linking between two distant EMQX clusters.
 

en_US/message-queue/message-queue-task.md

@@ -2,17 +2,17 @@
 
 This page walks you through the practical usage of the Message Queue feature in EMQX, from creating queues to configuring their behavior and managing them using the Dashboard, REST API, or configuration files.
 
-## Create Message Queue via Dashboard
+## Manually Create Message Queue via Dashboard
 
-Message Queues must be explicitly declared/created before they can store or dispatch messages.
+Message Queues must be explicitly declared/created before they can store or dispatch messages. You can create message queues either manually or automatically. For details about automatic creation, see [Automatically Create Message Queue via Dashboard](#automatically-create-message-queue-via-dashboard).
 
-To create a new Message Queue using the EMQX Dashboard:
+To create a new Message Queue manually using the EMQX Dashboard:
 
 1. Navigate to **Message Queue** in the left menu.
 
 2. Click the **Create** button on the page.
 
-3. In the **Create Message Queue** dialog:
+3. In the **Create Message Queue** dialog, configure the following options:
 
    - **Topic Filter**: Enter the topic or topic filter (e.g., `t/1`).  It defines which published messages are enqueued based on topic matching. A queue will collect all messages that match this topic filter.
 
@@ -26,13 +26,23 @@ To create a new Message Queue using the EMQX Dashboard:
 
    - **Data Retention Period**: Specify how long messages should be retained in the queue. You can set the time unit (e.g., days).
 
-   - **Last Value Semantics**: Toggle this switch on if you want new messages with the same queue key to overwrite older messages in the same queue. When enabled, a new message with the same queue key will overwrite any previous, unconsumed message with that key in the queue.
+   - **Last Value Semantics**: This option is enabled by default. When enabled, a new message with the same queue key will overwrite any previous, unconsumed message with that key in the same queue. This ensures only the most recent message per key is retained. The default key is the client ID of the message publisher.
 
      - **[Queue Key Expression](#queue-key-expression)**: When Last-Value Semantics is enabled, this field defines the expression used to extract the key from each message. The default value is `message.from`, which means the client ID of the message publisher. This field supports configuration using [Variform expressions](../configuration/configuration.md#variform-expressions).
 
+   - **Max Shard Message Count**: (Optional) Sets the maximum number of messages allowed in each shard of the queue. You can toggle this setting on and enter a custom value, or leave it disabled to allow unlimited messages (`infinity`). This setting is persisted to durable storage.
+
+   - **Max Shard Message Bytes**: (Optional) Sets the maximum total size (in bytes) of messages in each shard of the queue. You can toggle this setting on and enter a value (e.g., `200MB`), or leave it disabled for unlimited size (`infinity`). This setting is persisted to durable storage.
+
+     ::: tip Performance Note
+
+     Queues with size limits may have slower write performance, especially under high throughput conditions.
+
+     :::
+
 4. Click **Create** to save the queue.
 
-The new queue will appear in the Message Queue list, showing its topic filter, dispatch strategy, last-value semantics status, and data retention period. You can edit or delete queues using the buttons in the **Actions** column.
+The new queue will appear in the Message Queue list, showing its topic filter, dispatch strategy, last-value semantics status, and data retention period. You can edit queue settings or delete queues using the buttons in the **Actions** column.
 
 ### Queue Key Expression
 
@@ -110,23 +120,85 @@ Queue Key Expressions are evaluated against the following message structure:
 
 </details>
 
+## Automatically Create Message Queue via Dashboard
+
+Starting from EMQX 6.0.1, Message Queues can be automatically created when clients subscribe to a `$q/`-prefixed topic. This allows queues to be provisioned dynamically without manual setup.
+
+The queues may be auto-created either as regular queues or last-value semantics queues. 
+
+::: tip Note
+
+To ensure proper queue behavior, you can enable either **Auto Create Regular Message Queue** or **Auto Create Last Value Semantics Queue**, but not both at the same time.
+
+:::
+
+### Auto Create Last Value Semantics Queue
+
+This option is turned on by default in the **Message Queue** tab under **MQTT Settings**. It allows EMQX to automatically create queues that support Last-Value Semantics, where only the most recent message with a given key is retained.
+
+1. Navigate to **Management** -> **MQTT Settings** -> **Message Queue** tab.
+
+2. By default, **Enable Auto Create Last Value Semantics Queue** is enabled.
+
+   Configure the following:
+
+   - **Queue Key Expression**: Required. Defines how to extract a unique key from each message (default: `message.from`).
+   - **Dispatch Strategy**: Determines how messages are distributed to subscribers (default: `Random`).
+   - **Data Retention Period**: Specifies how long messages should be retained in the queue.
+
+3. Click **Save Changes**.
+
+When a client subscribes to a topic such as `$q/test`, EMQX will automatically create a last-value semantics queue, which will appear in the **Message Queue** list.
+
+### Auto Create Regular Message Queue
+
+This option can be enabled manually if you prefer regular queues where messages are stored independently and not overwritten.
+
+1. Go to **Management** -> **MQTT Settings** -> **Message Queue** tab.
+2. Turn on **Enable Auto Create Regular Message Queue**.
+3. Configure the following:
+   - **Dispatch Strategy**: Determines how messages are distributed to subscribers (default: `Random`).
+   - **Data Retention Period**: Specifies how long messages should be retained in the queue.
+4. Click **Save Changes**.
+
 ## Configure Message Queue Settings
 
-This section explains how to configure global settings that apply to all Message Queues in EMQX. These settings control message retention, cleanup intervals, and internal queue behavior. You can configure them via the Dashboard, REST API, or configuration file.
+This section explains how to configure global settings that apply to all Message Queues in EMQX. These settings control message retention, cleanup intervals, internal queue behavior, and queue auto-creation behavior. You can configure them via the Dashboard, REST API, or configuration file.
 
 ### Dashboard
 
 You can update Message Queue settings directly from the EMQX Dashboard without restarting the broker. This is useful for making changes to system-wide behavior at runtime.
 
 To configure global settings for Message Queues via the Dashboard:
 
-1. Navigate to the **Message Queue** page from the left menu.
-2. Click the **Settings** button in the top-right corner of the page.
-3. You will be redirected to the **MQTT Settings** -> **Message Queue** tab. In this panel, you can configure the following parameters:
+1. Go to **Management** -> **MQTT Settings** -> **Message Queue** tab.
+
+   Alternatively, you can click the **Settings** button in the top-right corner of the **Message Queue** page.
+
+2. In the **Message Queue** panel, the following configuration options are available:
+   - **Enable Message Queue**: The message queue system is enabled by default and cannot be disabled via the Dashboard.
+
+     > To disable it, you must modify the configuration file directly.
+
+   - **Max Queue Count**: Sets the maximum number of queues that can be created.
+
    - **GC Interval**: The interval at which expired messages are cleaned up from queues. Default is `1` hour.
+
    - **Regular Queue Retention Period**: The maximum duration for which messages are retained in regular queues. Default is `7` days.
+
    - **Find Queue Retry Interval**: When a client subscribes to a `$q/`-prefixed queue topic and the corresponding queue does not yet exist, this setting controls how often the client retries to find the queue. Default is `10` seconds.
-4. After making changes, click **Save Changes** to apply the new settings.
+
+   - **Auto-Creation Options**: EMQX supports dynamic queue provisioning through auto-creation features.
+
+     - **Auto Create Last Value Semantics Queue** (enabled by default): When a client subscribes to a `$q/` topic and no matching queue exists, EMQX will automatically create a queue with Last-Value Semantics enabled. 
+
+       For details of the settings, see [Auto Create Last Value Semantics Queue](#auto-create-last-value-semantics-queue).
+
+     - **Auto Create Regular Message Queue**: Can be enabled as an alternative to the above. When enabled, EMQX will create regular (non-overwriting) queues automatically for `$q/` subscriptions.
+
+       For details of the settings, see [Auto Create Regular Message Queue](#auto-create-regular-message-queue).
+
+3. After making changes, click **Save Changes** to apply the new settings.
 
 ### REST API
 
@@ -145,6 +217,8 @@ mq {
     gc_interval = 1h
     regular_queue_retention_period = 1d
     find_queue_retry_interval = 10s
+    max_queue_count = 100
+    }
 }
 ```
 
@@ -156,6 +230,7 @@ mq {
   Sets the maximum time that messages are retained in a regular queue. After this period, messages will be purged.
 - **`find_queue_retry_interval`**:
   Determines how frequently a subscriber retries to locate a queue when subscribing to a `$q/` topic that does not yet exist.
+- **`max_queue_count`**: (Optional) Sets the maximum number of queues that can be created.
 
 ## Manage Message Queue via REST API
 
@@ -168,7 +243,7 @@ Create a new message queue by specifying the topic filter and queue properties s
 ```bash
 curl -s -u key:secret -X POST -H "Content-Type: application/json" \
 http://localhost:18083/api/v5/message_queues \
--d '{"topic_filter": "t1/#", "is_lastvalue": false}' | jq
+-d '{"topic_filter": "t1/#", "is_lastvalue": false, "limits": {"max_shard_message_count": 10000, "max_shard_message_bytes": "200MB"}}' | jq
 ```
 
 ### List All Message Queues
@@ -187,7 +262,7 @@ Update the properties of an existing queue, such as its dispatch strategy:
 ```bash
 curl -s -u key:secret -X PUT -H "Content-Type: application/json" \
 http://localhost:18083/api/v5/message_queues/t1%2F%23 \
--d '{"dispatch_strategy": "least_inflight"}' | jq
+-d '{"dispatch_strategy": "least_inflight", "limits": {"max_shard_message_count": 5000, "max_shard_message_bytes": "100MB"}}' | jq
 ```
 
 ### Delete a Message Queue
@@ -214,6 +289,17 @@ http://localhost:18083/api/v5/message_queues/t1%2F%23
 
 ### What happens when queues exceed capacity?
 
-- Currently, Message Queues are not limited by size (number of messages or total bytes), but they are time-limited via the configured *retention period*.
-- Once messages expire (i.e., exceed the retention period), they are no longer eligible for delivery and will be automatically purged by EMQX during regular garbage collection cycles.
+Message Queues in EMQX now support multiple types of capacity limits. If any of these limits are reached, EMQX will remove the oldest messages during garbage collection (GC) until the queue size returns within the configured bounds.
+
+- **Time-based limit**: All queues are still subject to the configured *retention period*. Once messages exceed the retention period, they are no longer eligible for delivery and will be automatically purged during GC.
+
+- **Size-based limits**: You can optionally configure per-shard limits on:
+
+  - **Max number of messages** (`max_shard_message_count`)
+  - **Max total size of messages in bytes** (`max_shard_message_bytes`)
+
+  These limits are soft and applied during GC, not in real time. Queues may temporarily exceed the configured thresholds between GC cycles.
+  
+  Note that these limits apply per shard in durable storage. For information on how to configure the number of shards, see [Number of Shards](../durability/managing-replication.md#number-of-shards). In addition, size limits do not account for the [replication factor](../durability/managing-replication.md#replication-factor); the actual physical storage used by a queue will be multiplied by the replication factor.
+