KAFKA producer on ESP-32 running esp-idf

[Dauggie]

For my project, an ESP-32 device must send raw data from high sample rate sensors (accelerometer - 4Khz, EMI - 500Khz, Microphone - 17Khz) to an AWS server. I am using this data to feed a model which performs human activity recognition via Deep Learning.

Code: Select all

    <code>4Khz=4000 samples/sec
    17Khz=17,000 samples/sec
    500Khz=500,000 samples/sec

Based on my research MQTT does not seem to have the ability to handle such high throughput from sensors so I turned to Kafka which can handle 1m/sec to resolve my issue.
I been unable to find any Kafka libraries that will work with the ESP-32 natively. I have seen librdkafka which is built in C.
I have two questions:

• Is there any library currently available for running a Kafka producer on an ESP-32.
  If there is no existing library, is there a way for me to write a wrapper around librdkafka to leverage the C APIs available on the ESP-32 platform. If anyone has had experience writing wrappers I would love a detailed description or direction towards someone who has put together a nice writeup.

[ESP_Angus]

(Moderator's note: I've split this question to a new topic and moved it out of the News forum.)

I'm not familiar with Kafka, but you're correct that sending 21,500 MQTT messages/sec will not be possible on an ESP32.

Do you need to send one sample per MQTT message? If you can combine multiple samples into a single payload then you should be able to use MQTT as well.