Seems to be a limit on lenght of messages and/or topic names #170
Comments
|
There is no absolute limit, but in practice the amount of available RAM imposes a limit. |
|
Hi, I also noticed that when the publishing of larger payloads works, on the broker the last few chars of the JSON payloads are missing. Often only the I'm confident that this is not a RAM issue, because my ESP32-S3 has 8MB of RAM and only <2MB are used. I use MQTT V5 with the following properties: |
|
tracing the communication with wireshark i can see that even a message with a total lenght of 216 makes it fail |
|
here are some captures with wireshark, my observations: even adding 40 caracters makes it stall and reconnect etc, both with Qos=0 and 1, with the original message its better but not perfect |
|
i let it run for 8 hours with the short message and there are no reconnects and no seeming hiccups, wireshark says that the packages are malformed
i dont expect the mosquitto broker to create the problem, it has been in use for a long time and is not under any load
|
|
@bobveringa Given your experience with large payloads and the fact that the user is using V5, do you have any views on this? |
|
I don't think I can offer much insight beside the fact that the payloads we send and receive are much lager than this example here. About 4500 byes per message when sending, and about 600-900 bytes when receiving. To topic that are much lager than the example here. Maybe this is some kind of broker limitation? Or some undefined behaviour for the pico W. |
|
i was thinking that it could be my interrupt based handling of an input pin that was chopping up the mqtt transmission Function to handle interruptsdef blink(pin): Set up the interrupt on GP15 for rising edgegp15.irq(trigger=machine.Pin.IRQ_RISING, handler=blink) but it gives the same result even after disabling it and just outputting a fixed string looking at wireshark i can see that the first publish is transmitted in 3 packages marked with yellow and thereafter it stalls and never publishes again RAM free 147104 alloc 42848
but mqtt explorer does not show it, so i presume that mosquitto drops the package i will try the https://github.com/bobveringa/micropython-mqtt version and see if i get same results |
# Function to handle interrupts
...
def blink(pin):
...
# Increment interrupt count
interrupt_count += 1
# Add interrupt data to the ring buffer
timestamp = get_ntp_timestamp()
...Is And is your My thought is, what happens if your program is in the proces of a publish, and gets a interrupt, that calls another network functions. Ys |
|
Now i have started over with the version @bobveringa has on his github with no code modifications other than this and the mqtt_local credentials i have found that it can publish a message lenght of 237 just fine but not 254,
at the longer message it transmits the mqtt package but before recieving a [ACK] it starts retransmitting
is it a problem in the TCP/IP stack on the Pico W? or i what direction should i look? I have tried to publish the long message with 86 value-pairs from MQTT explorer on my pc and it works just fine, so i think we can rule out that its mosquitto on a raspberry pi that is causing the problem |
|
analyzing a transmission from my pc to the broker i see ->publish message
anyone has an idea why the PicoW never gets its ACK on the longer message? |
|
trying the newest micropython release as there is a mention of some error with two cores accessing wifi, micropython/micropython#16915 i can see that the mqtt does not improve testing general wifi connectivity with some examples using usocket i see something strange: the first homebrew code example fails when it reaches ~>150-200 bytes, the second can transmit 20000 bytes with a little retransmission but no stall, why? from utime import sleep
import network
import socket
import json
import time
# Function to connect to Wi-Fi
def lan_connect():
ssid = 'xxx'
wpa2 = 'xxx'
wlan = network.WLAN(network.STA_IF)
while not wlan.isconnected():
wlan.disconnect()
wlan.active(False)
time.sleep(1)
wlan.active(True)
wlan.connect(ssid, wpa2)
time.sleep(10)
print("Connected to Wi-Fi")
print(wlan.ifconfig())
# Function to send message to specified host and port
def send_message(s, message):
host = '192.168.10.120'
port = 4000
try:
# Convert string to bytes if it's not already bytes
if isinstance(message, str):
message = message.encode('utf-8')
s.sendall(message)
print(f"Message sent to {host}:{port}")
print(f"Sent {len(message)} bytes")
except Exception as e:
print(f"Failed to send message: {e}")
# Function to generate the message
def generate_message(iteration):
message = ""
for i in range(iteration + 1):
base_char = chr(65 + (i // 10)) # Start with 'A', 'B', 'C', etc.
repeat_count = i * 10
message += base_char * repeat_count
return message
# Connect to Wi-Fi
lan_connect()
iteration = 0
# Create socket outside the loop with improved buffer settings
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = '192.168.10.120'
port = 4000
s.connect((host, port))
try:
while True:
pin.toggle()
sleep(5) # sleep 5 seconds
message = generate_message(iteration)
send_message(s, message) # send message
iteration += 1
except KeyboardInterrupt:
pass
finally:
s.close()
pin.off()
print("Finished.")
Example 2
import network
import utime as time
import usocket as socket
# Set your wifi ssid and password here
WIFI_SSID = const('xxx')
WIFI_PASSWORD = const('xxx')
# Set the server address here like 1.2.3.4
SERVER_ADDR = const('192.168.10.120')
# These constants should match the server
BUF_SIZE = const(20000)
SERVER_PORT = const(4000)
TEST_ITERATIONS = const(10)
# Check if wifi details have been set
if len(WIFI_SSID) == 0 or len(WIFI_PASSWORD) == 0:
raise RuntimeError('set wifi ssid and password in this script')
# Check server ip address set
if len(SERVER_ADDR) == 0:
raise RuntimeError('set the IP address of the server')
# Start connection
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(WIFI_SSID, WIFI_PASSWORD)
# Wait for connect success or failure
max_wait = 20
while max_wait > 0:
if wlan.status() < 0 or wlan.status() >= 3:
break
max_wait -= 1
print('waiting for connection...')
time.sleep(1)
# Handle connection error
if wlan.status() != 3:
raise RuntimeError('wifi connection failed %d' % wlan.status())
else:
print('connected')
status = wlan.ifconfig()
print( 'ip = ' + status[0] )
# Open socket to the server
sock = socket.socket()
addr = (SERVER_ADDR, SERVER_PORT)
sock.connect(addr)
# repeat test for a number of iterations
for test_iteration in range(TEST_ITERATIONS):
# Read BUF_SIZE bytes from the server
read_buf = sock.read(BUF_SIZE)
print('read %d bytes from server' % len(read_buf))
# Check size of data received
if len(read_buf) != BUF_SIZE:
raise RuntimeError('wrong amount of data read')
# Send the data back to the server
write_len = sock.write(read_buf)
print('written %d bytes to server' % write_len)
if write_len != BUF_SIZE:
raise RuntimeError('wrong amount of data written')
# All done
sock.close()
print("test completed") |
|
You should use GitHub Flavoured Markdown to format your posts. In particular use |
|
I cannot reproduce your issue in the micropython-lib mqtt. I tried a growing range from a few bytes up to 138272 where the code fails on memory fragmentation due to no/sloppy memory design. So the issue is not the mqtt library (nor the mqtt_as library). How to find the cause , and then the solution |
Didn't you just demonstrate, that it has nothing to do with mqtt ? With these two examples, you tell us, that the first example misbehaves, as you initially reported, without any mqtt-code. And what is at And then you tell us, that the second example, works (far) better; still with no mqtt-code. So maybe you should search, within the differences in your two code-snips. First thing I notice, is that in your first example, you only send, and do not read from your socket. Secondly your initial connecting method to the wifi differs. Have you tried on another AP/wifi, maybe it's some (weird) incompatibilty between your AP and your device .. ? And again, I do think you just demonstrated, that you have your described error, even when not using/including mqtt-code. Hence the issue is misplaced in this forum. |
|
Hello everybody. I have the same error: https://github.com/orgs/micropython/discussions/17065 All works great with just some of bytes, but if the data is like 10kB starts to freeze. Thank you very much. |
yes, i belive so thats why i raised the question over at the micropython forum after discovering this lets close the thread here and point people elseware to find the solution |
Please (re)read this message, in this thread: #170 (comment) |
Hello. I'll try an older micropython firmware on my ESP32 device, but my error is just a freeze in the execution. If I put a JSON with for example 50Bytes all works great, but If a publish one with 10kB when the executions connects with the broker just freezes. Thank you very much. Best regards. |
In my testing on a Pico W i can publish a short message to
a/b/c/d/e/mqtt just fine
but not
sensors/blinktaeller/28cdc101dd54/mqtt_as
it publishes
1 repubs: 0 outages: 0 free: 149600 just fine
but not
{"msg": "0 repubs: 0 outages: 0 free: 153392", "ring_buffer": [[1, 1609459449470], [2, 1609459450652], [3, 1609459451452], [4, 1609459452058], [5, 1609459452646], [6, 1609459454092], [7, 1609459454721], [8, 1609459455818], [9, 1609459456859], [10, 1609459456892], [11, 1609459456996], [12, 1609459457071], [13, 1609459457120], [14, 1609459457196], [15, 1609459457243], [16, 1609459457306], [17, 1609459457336], [18, 1609459457366], [19, 1609459457475], [20, 1609459457576], [21, 1609459457612], [22, 1609459457694], [23, 1609459457801], [24, 1609459457863], [25, 1609459457897], [26, 1609459457930], [27, 1609459457963], [28, 1609459458058], [29, 1609459458117], [30, 1609459458157], [31, 1609459458247], [32, 1609459458357], [33, 1609459458464], [34, 1609459458538], [35, 1609459458567], [36, 1609459458639], [37, 1609459458700], [38, 1609459458769], [39, 1609459458832], [40, 1609459458918], [41, 1609459459001], [42, 1609459459104], [43, 1609459459141], [44, 1609459459222], [45, 1609459459293], [46, 1609459459338], [47, 1609459459374], [48, 1609459459457], [49, 1609459459527], [50, 1609459459628], [51, 1609459459665], [52, 1609459459697], [53, 1609459459735], [54, 1609459459786], [55, 1609459459864], [56, 1609459459891], [57, 1609459459992], [58, 1609459460051], [59, 1609459460105], [60, 1609459460132], [61, 1609459460236], [62, 1609459460292], [63, 1609459460329], [64, 1609459460418], [65, 1609459460498], [66, 1609459460556], [67, 1609459460651], [68, 1609459460741], [69, 1609459460809], [70, 1609459460878], [71, 1609459460979], [72, 1609459461036], [73, 1609459461126], [74, 1609459461149], [75, 1609459461235], [76, 1609459461304], [77, 1609459461409], [78, 1609459461444], [79, 1609459461497], [80, 1609459461574], [81, 1609459461608], [82, 1609459461708], [83, 1609459461792], [84, 1609459461856], [85, 1609459461937], [86, 1609459461972], [87, 1609459462027], [88, 1609459462097]]}
it publises the first, then tries no 2, but after a while it says wifi down and reconnects without continuing the publish
The text was updated successfully, but these errors were encountered: