Sending large data over MQTT

[davdav]

Hi everybody,

I have to send multiple files (about 1000) content (for a total of about 1 MB or more) via MQTT to a remote broker.

Since ESP32-WROOM-32 has limited RAM I can't read all files and put in a buffer before publish it.

Is there any way to perform a "chuncked/segmented" publishing with esp-mqtt library?
It seems that

Code: Select all

esp_mqtt_client_publish

need to know the length of the message and the pointer to content, but in my case I can't get the total length and I would like to open the file one by one and send the data during the publish connection.


Thanks

[agatrost1620]

@davdav were you able to figure out a solution?

I am trying to send 1 file that is in the flash using mqtt. I do not have enough memory to allocate for the size of the file, nor can a create an array large enough either. the file at this time is 48kB, and I would like to be able to not worry about the size, because it could be as big as 256kB in the future.

Is there a way to use a FILE * to send using mqtt?

Thanks in advance,

[PeterR]

Assuming qos is 0 then it should be possible.
Not entirely clear on your system but if you store your 'file' as raw data in FLASH then you could use the existing API. Of course you would not be able to retain data.
If you want to use a file within a file system then you would have to dig into the IDF library. MQTT is built on a websocket and so you can stream (assuming qos 0) by which there is no need to complete within a single API call. I only saw block transfers in the IDF but there is no reason why not from a file (using qos=0)
Maybe worth a feature request or a little project for you.

Another approach might be to create many segment endpoints e.g. myfile/seg0 ... myfile/seg100 etc.

[davdav]

@davdav were you able to figure out a solution?

I am trying to send 1 file that is in the flash using mqtt. I do not have enough memory to allocate for the size of the file, nor can a create an array large enough either. the file at this time is 48kB, and I would like to be able to not worry about the size, because it could be as big as 256kB in the future.

Is there a way to use a FILE * to send using mqtt?

Thanks in advance,

Hello @agatrost1620,
I managed on "application level", i.e. I implemented a "protocol" between esp32 and the backend of the mqtt broker on the server. In practice it is a sort of segmentation as @PeterR suggested.

I have also raised a question in esp-mqtt github repo. See here:

https://github.com/espressif/esp-mqtt/issues/123

maybe it can help to find a good solution for you.


Regards