Is there a way to have ESP32 send its logs to a web server?

[statusquo]

I have a bunch of ESP32s at remote locations, they are all flashed with OTA.

I'd like to be able to see their logs without having to be at their location in case I have some issues I'd like to troubleshoot.

Besides them at the same location are PCs that I have access to via Anydesk, so I'd either like ESP32s to send their logs
to my server or store them locally on the PCs in case there is no internet.

Is this easily doable?

[alanesq]

Hi,

The easiest way I have found is this: https://randomnerdtutorials.com/esp32-c ... to-server/
although I don't know how secure it would be

Another option would be to have it FTP the files (there are libraries for FTP)

or failing that it could email them to you.

[gtjoseph]

esp_log_set_vprintf() allows you to provide an alternate backend for the log subsystem. You could create a new backend that uses the http client to send the messages anywhere you want.
https://docs.espressif.com/projects/esp ... ntf_like_t

[nwolcott]

I used the MQTT client to send log prints to a broker on the network. It works very well and can handle a ridiculous number of prints per second.
I used the mqtt_event_handler callback event MQTT_EVENT_CONNECTED to register a new log vprintf function.
Here's how I did it.
To get the log output, just subscribe to the log's topic on the MQTT broker.

Code: Select all

// A simple helper function to publish a string log message.
// The return code is passed through from the underlying call.
int mqttSendLog(char *msg, int length)
{
   // Use whatever MQTT topic you like here.
   // I usually use no QOS here to keep things fast. (Also it's a tcp connection to the broken on my local network).
    return esp_mqtt_client_publish(mqttClient, "test/Nate/log", msg, length, MQTT_QOS_NONE, 0);
}

// This function can be registered with the ESP_LOG subsystem to replace it's default
// output function. This lets us get system print logs over MQTT.
// The function signature must be the same as vprintf.
int systemLogApiVprintf(const char* fmt, va_list args)
{
   // This also sends the log print to the normal serial output.
    vprintf(fmt, args); // If you don't want this behavior, just comment this out.

    int status = 0;
    // In my system, I keep my prints pretty short so:
    #define LOG_BUFFER_SIZE (128)
    
    // Allocate some memory.
    char *logBuff = malloc(LOG_BUFFER_SIZE);
    if(logBuff != NULL)
    {
        // Print the log entry into a buffer.
        status = vsnprintf(logBuff, LOG_BUFFER_SIZE, fmt, args);
        // Send it to the MQTT client.
        int mqttStatus = mqttSendLog(logBuff, status);
        if(mqttStatus < 0)
        {
           // This is a global in my system to keep track of printing errors.
            droppedPrintCount++;
        }
        // Free the buffer.
        free(logBuff);
    }
    
    return status;
}

// Save the original registered log handler function pointer so we can restore it if
// we need to.
vprintf_like_t originalLogPrinter = NULL;

static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
{
    esp_mqtt_event_handle_t event = event_data;
    esp_mqtt_client_handle_t client = event->client;

    switch ((esp_mqtt_event_id_t)event_id) {
    case MQTT_EVENT_CONNECTED:
        ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");

        // Register the new log handler.
        originalLogPrinter = esp_log_set_vprintf(systemLogApiVprintf);

        // Try printing on the new handler.
        ESP_LOGI(TAG, "Switched to MQTT logging");
        break;
        
        // The rest of the mqtt event handler code, etc...
        }
}

[mbratch]

If your server supports the syslog service (eg., Linux), then you can configure and turn on that service on the server, and send log messages to it via UDP or TCP/IP packets. You just format the log message properly and send it to the IP address and syslog port number of the server. The default syslog port number is 514. There are a number of cloud based syslog servers as well. I use syslog and it works great.

The syslog service supports log message priorities, timestamps, etc. You can find lots of information online if you search.

[sukeshak]

I have a bunch of ESP32s at remote locations, they are all flashed with OTA.

I'd like to be able to see their logs without having to be at their location in case I have some issues I'd like to troubleshoot.

Besides them at the same location are PCs that I have access to via Anydesk, so I'd either like ESP32s to send their logs
to my server or store them locally on the PCs in case there is no internet.

Is this easily doable?

Check if this component works for your scenario. It redirects Serial output (IDF logging) to network
https://github.com/nopnop2002/esp-idf-net-logging

The following protocols are available for this project.
UDP
TCP
MQTT
HTTP(POST)

[Cacomixtle]

I used the MQTT client to send log prints to a broker on the network. It works very well and can handle a ridiculous number of prints per second.
I used the mqtt_event_handler callback event MQTT_EVENT_CONNECTED to register a new log vprintf function.
Here's how I did it.
To get the log output, just subscribe to the log's topic on the MQTT broker.

Code: Select all

// A simple helper function to publish a string log message.
// The return code is passed through from the underlying call.
int mqttSendLog(char *msg, int length)
{
   // Use whatever MQTT topic you like here.
   // I usually use no QOS here to keep things fast. (Also it's a tcp connection to the broken on my local network).
    return esp_mqtt_client_publish(mqttClient, "test/Nate/log", msg, length, MQTT_QOS_NONE, 0);
}

// This function can be registered with the ESP_LOG subsystem to replace it's default
// output function. This lets us get system print logs over MQTT.
// The function signature must be the same as vprintf.
int systemLogApiVprintf(const char* fmt, va_list args)
{
   // This also sends the log print to the normal serial output.
    vprintf(fmt, args); // If you don't want this behavior, just comment this out.

    int status = 0;
    // In my system, I keep my prints pretty short so:
    #define LOG_BUFFER_SIZE (128)
    
    // Allocate some memory.
    char *logBuff = malloc(LOG_BUFFER_SIZE);
    if(logBuff != NULL)
    {
        // Print the log entry into a buffer.
        status = vsnprintf(logBuff, LOG_BUFFER_SIZE, fmt, args);
        // Send it to the MQTT client.
        int mqttStatus = mqttSendLog(logBuff, status);
        if(mqttStatus < 0)
        {
           // This is a global in my system to keep track of printing errors.
            droppedPrintCount++;
        }
        // Free the buffer.
        free(logBuff);
    }
    
    return status;
}

// Save the original registered log handler function pointer so we can restore it if
// we need to.
vprintf_like_t originalLogPrinter = NULL;

static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
{
    esp_mqtt_event_handle_t event = event_data;
    esp_mqtt_client_handle_t client = event->client;

    switch ((esp_mqtt_event_id_t)event_id) {
    case MQTT_EVENT_CONNECTED:
        ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");

        // Register the new log handler.
        originalLogPrinter = esp_log_set_vprintf(systemLogApiVprintf);

        // Try printing on the new handler.
        ESP_LOGI(TAG, "Switched to MQTT logging");
        break;
        
        // The rest of the mqtt event handler code, etc...
        }
}

Hi, Have you experienced a Stack overflow in task mqtt_task? if you have, how did you resolve it?