[Solved] esp_partition_write for OTA

[HT_espressif]

Hello,

I want to implement OTA programming for the ESP32. But I dont want to use a webserver but through direct file transfer on an html page. So I wrote this handler which triggers on a HTTP post request. The function works and I can select a binary file and program the esp32. However the esp_partition_write function keeps adding whitespace to my partition if I dont use vTaskDelay(). (checked it by reading the flash and comparing it to my orignal binary)

With vTaskDelay(8) it works but the only way I got it to work is by experimenting a bit with the delay value. I would like to know why this happens. My guess is that the esp_partition_write function is still doing something in the background and I need to first wait for it to finish. The uploading is also very slow due to the delay and I would like to improve that to similar speeds as the regular esp_https_ota function. Is there maybe some kind of flag I can read out before calling the function again?

Code: [Select all] [Expand/Collapse]

1. // Handler for esp32 ota programming
2. esp_err_t program_esp32_handler(httpd_req_t* req)
3. {
4.     esp_err_t error = 0;
5.     printf("esp32_handler\n\r");
6.     extern const unsigned char cacert_pem_start [] asm("_binary_cacert_pem_start");
7.     extern const unsigned char cacert_pem_end   [] asm("_binary_cacert_pem_end");
8.     ESP_LOGI(TAG, "ESP32 programming starting");
9.     char url[1024];
10.     char buf[512] = {0};
11.     int received;
12.     int remaining = req->content_len;
13.     int iterator  = 0;
14.  
15.     // Find the partition map in the partition table
16.     const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "ota_1");
17.     assert(partition != NULL);
18.  
19.     ESP_ERROR_CHECK(esp_partition_erase_range(partition, 0, partition->size));
20.     printf("content length %d\n\r", req->content_len);
21.  
22.     while (remaining > 0)
23.     {
24.         //ESP_LOGI(TAG, "Remaining size : %d", remaining);
25.         /* Receive the file part by part into a buffer */
26.         memset(buf, 0, 512);
27.         if ((received = httpd_req_recv(req, buf, MIN(remaining, 512))) <= 0)
28.         {
29.             if (received == HTTPD_SOCK_ERR_TIMEOUT) continue;
30.  
31.             ESP_LOGE(TAG, "File reception failed!");
32.             httpd_resp_send_err(req, HTTPD_500_INTERNAL_SERVER_ERROR, "Failed to receive file");
33.             return ESP_FAIL;
34.         }
35.         //ESP_LOGI(TAG, "Received size : %d", received);
36.         ESP_ERROR_CHECK(esp_partition_write(partition, 0 + (iterator * 512), buf, MIN(remaining, 512)));
37.         vTaskDelay(8); // Note required otherwise esp_partition_write adds whitespace
38.         remaining -= received;
39.         iterator++;
40.     }
41.     ESP_LOGI(TAG, "File reception complete");
42.  
43.     // Redirect onto root to see the updated file list
44.     httpd_resp_set_status(req, "303 See Other");
45.     httpd_resp_set_hdr   (req, "Location", "/");
46. #ifdef CONFIG_EXAMPLE_HTTPD_CONN_CLOSE_HEADER
47.     httpd_resp_set_hdr(req, "Connection", "close");
48. #endif
49.     httpd_resp_sendstr(req, "File uploaded successfully");
50.  
51.     partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "ota_1");
52.     esp_ota_set_boot_partition(partition);
53.  
54.     vTaskDelay(200);
55.     esp_restart();
56.     return error;
57. }

GeSHi © Codebox Plus Extension

[ESP_Sprite]

What do you mean by 'whitespace'? Can you give us a hexdump of the partition with and without the vTaskDelay?

[HT_espressif]

Hello Sprite,

I attached a picture from this free text compare website you can upload text there and it will show where it is different. https://www.diffchecker.com/diff
The picture I attached is a screenshot of that webpage.

As you can see sometimes there is whitespace added in random posititions if I dont add the vTaskDelay(8) in the loop. The positions are random and not in fixed locations if I run the same code twice the whitespace is in other spots. The whole file looks like a carbon copy otherwise so my guess is the whitespace is causing it to not be read properly.

[boarchuz]

if ((received = httpd_req_recv(req, buf, MIN(remaining, 512))) <= 0)
...
ESP_ERROR_CHECK(esp_partition_write(partition, 0 + (iterator * 512), buf, MIN(remaining, 512)));

You're unconditionally writing 512 bytes of the buffer, despite perhaps having less available. Everything after the actual number of bytes received is zeroed.

The delay happens to make things work because the internal buffer has time to queue up 512+ bytes.

[HT_espressif]

if ((received = httpd_req_recv(req, buf, MIN(remaining, 512))) <= 0)
...
ESP_ERROR_CHECK(esp_partition_write(partition, 0 + (iterator * 512), buf, MIN(remaining, 512)));

You're unconditionally writing 512 bytes of the buffer, despite perhaps having less available. Everything after the actual number of bytes received is zeroed.

The delay happens to make things work because the internal buffer has time to queue up 512+ bytes.

Ah yes thank you so much I found the issue httpd_req_recv does not always return 512 bytes but returns up to 512 bytes.
Therefore the offset is not correct because part of the array contains zeroes and therefore whitespace.
I changed the code and now it is running very well now I also included a check to see which partition is the current boot partition.
This topic can be marked as solved if anyone is still following along.

Code: Select all

// Handler for esp32 ota programming
esp_err_t program_esp32_handler(httpd_req_t* req)
{
	esp_err_t error = 0;
    printf("esp32_handler\n\r");
    extern const unsigned char cacert_pem_start [] asm("_binary_cacert_pem_start");
    extern const unsigned char cacert_pem_end   [] asm("_binary_cacert_pem_end");
    ESP_LOGI(TAG, "ESP32 programming starting");
    char url[1024];
    char buf[512] = {0};
    int received;
    int offset = 0;
    int remaining = req->content_len;

    // Get first ota partition pointer
    const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "ota_0");
    assert(partition != NULL);
    // If ota_0 is the boot partition use ota_1 to write to otherwise keep ota_0
    if (esp_ota_get_boot_partition() == partition) partition = esp_partition_find_first(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, "ota_1");
    assert(partition != NULL);

    ESP_ERROR_CHECK(esp_partition_erase_range(partition, 0, partition->size));
    printf("content length %d\n\r", req->content_len);

    while (remaining > 0)
    {
        //ESP_LOGI(TAG, "Remaining size : %d", remaining);
        /* Receive the file part by part into a buffer */
    	memset(buf, 0, 512);
        if ((received = httpd_req_recv(req, buf, MIN(remaining, 512))) <= 0)
        {
            if (received == HTTPD_SOCK_ERR_TIMEOUT) continue;

            ESP_LOGE(TAG, "File reception failed!");
            httpd_resp_send_err(req, HTTPD_500_INTERNAL_SERVER_ERROR, "Failed to receive file");
            return ESP_FAIL;
        }
        //ESP_LOGI(TAG, "Received size : %d", received);
        ESP_ERROR_CHECK(esp_partition_write(partition, 0 + offset, buf, received));
        offset     += received;
        remaining -= received;
    }
    ESP_LOGI(TAG, "File reception complete");

    // Redirect onto root to see the updated file list
    httpd_resp_set_status(req, "303 See Other");
    httpd_resp_set_hdr   (req, "Location", "/");
#ifdef CONFIG_EXAMPLE_HTTPD_CONN_CLOSE_HEADER
    httpd_resp_set_hdr(req, "Connection", "close");
#endif
    httpd_resp_sendstr(req, "File uploaded successfully");

    esp_ota_set_boot_partition(partition);

    vTaskDelay(200);
    esp_restart();
	return error;
}