MQTT is still broken. Are there plans to fix it?

[DouglasWWolf]

Gang,

The version of MQTT in ESP-IDF v3.3.1 is fundamentally broken... it drops messages all over the place. (This is a known bug in Espressif's implementation)

Does anyone know if and/or when this might get fixed? This is a bad enough bug that I may end up having to switch to a different microcontroller for a commercial project.

--- Doug

[ESP_cermak]

Hi Doug,

Which bug are you referring to? AFAIK there are/were currently 3 limitations of the library:

1) Wrong reception of retained messages (e.g. when one TCP packet contain multiple MQTT messages)
2) Library is not thread safe
3) Messages (with qos>0) sent while client went off-line are not being resend on reconnection


Item 1) is already in master (please check 3.3-dev), the other two in progress.

If you know about anything else, please file an issue in ESP-IDF.
Thank you!

David

[DouglasWWolf]

ESP Cermak,

I was referring to the bug where it drops messages when multiple messages are received in rapid succession. (This is a well known bug in tuan's code, which is the the original source of the mqtt library in the esp-idf)

I'm using esp-idf v3.3-dev-278. Is there an update since then that solves the mqtt dropped messages issue?

Thanks for the help. This is a product breaker for us.

--- Doug

[DouglasWWolf]

Cermak,

I grabbed the latest beta release (the current master) and did a brief MQTT test with it, and it appears to resolve the immediate problem of incoming messages being dropped.


Thanks for the help!

Any idea when we'll get a new stable release?

--- Doug

[DouglasWWolf]

Cermak,

I spoke too soon.

While the MQTT situation is much better, its still broken.
I'm sending 58 messages in rapid succession, each message about 70 bytes in length. *Most* of the messages are delivered to the app in one piece... but it looks like others are corrupted (i.e., they have a nul-byte inserted into them at random places)

--- Doug

[DouglasWWolf]

Gang,

The MQTT bug that caused many dropped messages when messages arrive in quick succession has been fixed, but in the process, a new MQTT bug has been introduced.

Every now and then and incoming message gets broken into two separate messages. It's not duplicated, it's simply broken in half and delivered in two parts.

I test this by sending the same 52 character message many times in rapid succession. At least once every 60 messages or so, the "break the message in half and deliver it in two pieces" bug crops up.

For the record, the string that I'm sending as a message over and over is:
"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

I'm using mosquitto_sub to subscribe to the topic for that message and can confirm that mosquitto broker is sending the message properly. (I've also confirmed this by examining the packets via Wireshark)

Guys, I know you're working hard on delivering a good product... but I'm sure I can't be the only engineer looking to use the ESP32 in a commercial IoT product, and MQTT has *got* to work reliably in order for the ESP to be usable.

Thanks for your help.

--- Doug

[ESP_igrr]

Moderator's note: have merged new topic with the previous one, since it is related to the same issue.

[ESP_cermak]

Hi Doug,

I believe that what you describe is an intended behaviour. Message can be broken down to more 'data-received' events. Can you please check that the message id is the same for both parts? There's also some more info such as total/remaining message length within the data event to help you track the message. Some messages must be broken down to more data events, e.g. when received message is longer than incoming buffer size.

[DouglasWWolf]

Cermak,

I've dumped the data_len, current_data_offset and total_data_len from a bunch of MQTT_EVENT_DATA events. What I've found makes little sense. What I see:

data_len is always sensible: it's the number of data bytes in this particular messsage
data_offset is always sensible: it's the offset of this chunk of data within the overall message
total_data_len is usually garbage... it's negative numbers more than half the time.


Am I missing something?

--- Doug

[DouglasWWolf]

Cermak,

For what it's worth: after a little testing, I see that the issue of event->total_data_len containing garbage (i.e., negative numbers) only occurs when I'm the ESP32 is receiving messages that were sent in rapid succession. When messages are sent to the ESP32 more slowly (such that there is only ever one message in the ESP32's incoming message buffer) event->total_data_len always looks reasonable.

--- Doug