Protocol for handling large files over MQTT

[rtborg]

I need to be able to send and receive relatively large files over MQTT (~50 KB). The files should be read from and written to SD card. Could someone point me to an example of an application protocol which handles the segmentation of the files during sending/receiving? I'd rather take a look at a solution before start implementing it myself.

Currently what I am trying is:
In MQTT event handler:
- If message has topic and topic len is not 0, save them to static buffers
- Pass message data to client function using topic and topic len
- If message has no topic and len is 0, pass message data to client function using saved topic nad topic len. Message is assumed to be part of previous one.

Thanks

[vanBassum]

My knoledge of MQTT is quite limited, but it doens't seem to be the best choice for large files. What do these files contain?
What you could do is, start a http server and provide the file that wat. Use MQTT to report the URL to use to download the file.

If you better describe the usecase that way other people can help you better towards a suitable solution.

[rtborg]

Thanks for the input.

The device I am working on receives configuration file(s) from a server which runs a MQTT broker. Upon boot, the device subscribes for a certain topic ("config"), and at some point the server publishes the contents of the configuration file as JSON.

The device can also alter the configuration file(s), and when that happens the complete file should be published to a certain MQTT topic.

I don't have control over that end of the application, so what I have to do is make the device conform to the already established protocol.

[chegewara]

50kB is huge size for config file. I didnt try that big files, but i believe esp32 mqtt client should receive it in chunks, at least it was the case when my broker sent json config bigger than RX buffer size. The problem with mqtt is it is not guarantee to receive messages in the same order you are expecting, so you have to implement protocol with message ID and reconstruction options.
Something like that is implemented in AWS jobs for example when you are performing OTA over mqtt.

[rtborg]

I agree it's a huge size, but that's what I have to work with. It's good to be aware of the possible out-of-order arrival of the messages, I'll take a look at how that's done in AWS OTA.

[vanBassum]

You could use Brotli compression, its very effective when using JSON.