There is a suspected premature free during the reconnection process

[x152152]

Describe the bug

• Version: v1.3.8
• Client: async client

I found that my program crashed and the stack information was roughly as follows：

==3219397==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060010231a8 at pc 0x7ffae61b5cad bp 0x7ffa80d4ed80 sp 0x7ffa80d4e528
READ of size 41 at 0x6060010231a8 thread T87
#0 0x7ffae61b5cac in __interceptor_fopen64 ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:5757
#1 0x7ffae089f5d9 in BIO_new_file (/lib/x86_64-linux-gnu/libcrypto.so.1.1+0xad5d9)
#2 0x7ffae09e064b in X509_load_cert_crl_file (/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1ee64b)
#3 0x7ffae09e07b9 (/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1ee7b9)
#4 0x7ffae09e3612 in X509_STORE_load_locations (/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1f1612)
#5 0x7ffa8c421974 in SSLSocket_createContext SSLSocket.c:632
#6 0x7ffa8c421d23 in SSLSocket_setSocketForSSL SSLSocket.c:711
#7 0x7ffa8c41f8e1 in MQTTAsync_connecting MQTTAsyncUtils.c:2708
#8 0x7ffa8c420204 in MQTTAsync_cycle MQTTAsyncUtils.c:2871
#9 0x7ffa8c41d418 in MQTTAsync_receiveThread MQTTAsyncUtils.c:1955
#10 0x7ffae1289608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608)
#11 0x7ffae11ae132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

0x6060010231a8 is located 8 bytes inside of 64-byte region [0x6060010231a0,0x6060010231e0)
freed by thread T94 here:
#0 0x7ffae626d40f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:122
#1 0x7ffa8c43a290 in myfree Heap.c:283
#2 0x7ffa8c413b95 in MQTTAsync_connect MQTTAsync.c:728
#3 0x7ffa8c595518 in mqtt::async_client::connect(mqtt::connect_options) async_client.cpp:514
#4 0x7ffa8c5ae57c in mqtt::client::connect(mqtt::connect_options) client.cpp:91
#5 0x7ffa901f7f98 in MqttSyncClient::SubReconnect() mqtt_syncclient.cpp:161

previously allocated by thread T94 here:
#0 0x7ffae626d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
#1 0x7ffa8c439e9a in mymalloc Heap.c:199
#2 0x7ffa8c425c52 in MQTTStrdup MQTTProtocolClient.c:940
#3 0x7ffa8c413dec in MQTTAsync_connect MQTTAsync.c:756
#4 0x7ffa8c595518 in mqtt::async_client::connect(mqtt::connect_options) async_client.cpp:514
#5 0x7ffa8c5ae57c in mqtt::client::connect(mqtt::connect_options) client.cpp:91
#6 0x7ffa901f7f98 in MqttSyncClient::SubReconnect() mqtt_syncclient.cpp:161

Thread T87 created by T0 here:
#0 0x7ffae619a815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
#1 0x7ffa8c430d60 in Thread_start Thread.c:75
#2 0x7ffa8c4136d5 in MQTTAsync_connect MQTTAsync.c:647
#3 0x7ffa8c595518 in mqtt::async_client::connect(mqtt::connect_options) async_client.cpp:514
#4 0x7ffa8c5ae57c in mqtt::client::connect(mqtt::connect_options) client.cpp:91
#5 0x7ffa901f138d in MqttSyncClient::SubConnect() mqtt_syncclient.cpp:124

Thread T94 created by T0 here:
#0 0x7ffae619a815 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cc:208
#1 0x7ffae14e70c9 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_deletestd::thread::_State >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd70c9)

It seems that the client's SSL resources are operated by two threads, causing a memory leak.
2 threads performing MQTTAsync_connect operations at the same time may cause the case, which
is my usage fault maybe(When the connection is disconnected I will use connect instead of reconnect).

Looking forward to your reply, thank you.

[icraggs]

I would try the latest release, 1.3.8 is over 3 years old now.

It seems you have three different connects running simultaneously on separate threads?

It looks more like a premature free rather than memory leak to me.

Is this reproducible? Can you take a client library trace if so (as described in the readme).

[x152152]

I would try the latest release, 1.3.8 is over 3 years old now.

Thanks, I have reproduced this in both 1.3.8 and 1.3.13 versions

It seems you have three different connects running simultaneously on separate threads?

I think it is two connecting threads running at the same time, From the stack I provided, it should be T94 and T87.
Thread 97 is from my application's active monitoring. When I find that is_connected() is false, I will connect().
When the program is initialized, thread 87 performs connect and sub，and then rans to MQTTAsync_cycle.In some weak network conditions, SSL and TCP handshake connections will be performed, during this connection process, the SSL resources released during the connection process of thread 97 may be used.My analysis may be wrong, please correct me.

if (m->c->connect_state == TCP_IN_PROGRESS || m->c->connect_state == SSL_IN_PROGRESS || m->c->connect_state == WEBSOCKET_IN_PROGRESS)
*rc = MQTTAsync_connecting(m);

It looks more like a premature free rather than memory leak to me.

Yes, I think so too, I didn't express it clearly enough.

Is this reproducible? Can you take a client library trace if so (as described in the readme).

I ran my program in a weak network environment simulated by my script , I found that it can be reproduced(it may appear after running for 1-2 days).I didn't set the trace environment variable before, I will try to turn on later. Do you need more information? Or is the stack information scanned by the asan tool enough?
Thanks. @icraggs

[icraggs]

Any reason why you're not using auto reconnect?

If you really want to do it yourself, then it's better to use the connectionLost callback to determine when to reconnect, that's what it's intended for.

I see you're using a C++ client - the Paho one?

Here you've got one thread connecting and two re-connecting. If you are connecting in more than one thread at a time, you should be using different client objects. In fact it looks like you've got two reconnects going before the first connect has completed (going by your method names).

It looks like there are three threads interfering with each other when I think they should be locked out. Not sure how that can happen.

If you can get the full stack trace on version 1.3.13, or a client library trace or perhaps your application program or snippets those could help.