Memory leakage: esp_partition_write

pablopabota
Posts: 4
Joined: Fri Jan 06, 2023 6:54 pm

Memory leakage: esp_partition_write

Postby pablopabota » Thu Jan 19, 2023 3:05 am

Hi,

I'm sending data over MQTT to my server and I found a memory leakage after saving data in flash memory. I don't know if I'm missing something when writting data in flash.

Here is what I get in console:

Code: Select all

I (19:23:47.354) mqtt: mqtt_publish on topic /telemetry, payload {"ts":1673033027000,"values":{"anguloX":0,"anguloY":0,"anguloZ":0,"VelAnguloX":0.0079987,"VelAnguloY":0.00451767,"VelAnguloZ":0.00185478}}
I (19:23:47.410) mqtt: MQTT_EVENT_PUBLISHED, msg_id=40740
I (19:23:47.621) main: 
Free heap: 38448
Minimum heap: 26456

...
(This happens multiple times)
...

I (19:24:02.069) mqtt: mqtt_publish on topic /telemetry, payload {"ts":1673033041000,"values":{"anguloX":0,"anguloY":0,"anguloZ":0,"VelAnguloX":0.023266,"VelAnguloY":-0.0164746,"VelAnguloZ":-0.00577879}}
I (19:24:02.123) mqtt: MQTT_EVENT_PUBLISHED, msg_id=39102
I (19:24:02.671) main: 
Free heap: 30812
Minimum heap: 26456

I (19:24:03.132) mqtt: mqtt_publish on topic /telemetry, payload {"ts":1673033043000,"values":{"anguloX":0,"anguloY":0,"anguloZ":0,"VelAnguloX":0.0289911,"VelAnguloY":0.000700951,"VelAnguloZ":0.0247555}}
I (19:24:03.193) mqtt: MQTT_EVENT_PUBLISHED, msg_id=60714
I (19:24:03.671) main: 
Free heap: 30272
Minimum heap: 25956

...
(Minimum heap keeps going down until..)
...

I (19:24:56.166) mqtt: mqtt_publish on topic /telemetry, payload {"ts":1673033096000,"values":{"anguloX":0,"anguloY":0,"anguloZ":0,"VelAnguloX":0.000365198,"VelAnguloY":0.0102429,"VelAnguloZ":0.00948834}}
I (19:24:56.761) main: 
Free heap: 1956
Minimum heap: 712

I (19:24:57.762) main: 
Free heap: 2064
Minimum heap: 712

I (19:24:58.762) main: 
Free heap: 2064
Minimum heap: 712

I (19:24:59.762) main: 
Free heap: 2064
Minimum heap: 712

I (19:25:00.761) main: 
Free heap: 2064
Minimum heap: 712

I (19:25:01.762) main: 
Free heap: 2064
Minimum heap: 712

I (88169) wifi:bcn_timout,ap_probe_send_start
I (19:25:02.119) wifi: wifi_tcp_event_handler
I (19:25:02.120) wifi: BASE: WIFI_EVENT | ID: 21
W (19:25:02.120) wifi: WIFI_EVENT_STA_BEACON_TIMEOUT
I (19:25:02.126) wifi: wifi_update_status: status 1
I (88179) wifi:state: run -> init (0)
I (88189) wifi:pm stop, total sleep time: 41487982 us / 83411741 us
W (88199) wifi:<ba-del>idx
W (88199) wifi:<ba-del>idx
I (88199) wifi:new:<5,0>, old:<5,1>, ap:<5,1>, sta:<5,1>, prof:1
I (19:25:02.151) wifi: ESP_OK [0]: succeed
I (19:25:02.762) main: 
Free heap: 2780
Minimum heap: 712

WiFi crashes (Beacon timeout) when heap memory goes to low. So I started checking every time I'm sending data and realized that right before sending data I save it as backup in flash and if I comment that piece of code, minimum heap memory never goes down.

Here is how I save data in flash memory:

Code: Select all

void write_value_to_flash(uint32_t index, value_t* to_save_value)
{	
	uint32_t address_to_write = index * sizeof(value_t);
		
	// semaphore take to avoid writting when other task is writting 
	
	if (address_to_write % SPI_FLASH_SEC_SIZE == 0)
		esp_partition_erase_range(_log_partition, address_to_write, SPI_FLASH_SEC_SIZE);
	esp_partition_write(_log_partition, address_to_write, to_save_value, sizeof(value_t)); //if this line is commented, no memory leakage happens
	
	// semaphore give so writting is back available to other tasks 

}
Any piece of advise is welcome.

Best regards,
P

User avatar
mbratch
Posts: 303
Joined: Fri Jun 11, 2021 1:51 pm

Re: Memory leakage: esp_partition_write

Postby mbratch » Mon Jan 23, 2023 7:57 pm

You have comments that give/take a semaphore but no code that does anything with semaphores. Did you just omit that code in your post, or is there really no code there that is being executed in your memory leak scenario? Is `write_value_to_flash` the only thing you are commenting out when you have the non-leaky version?

You didn't mention what version of ESP-IDF you are using, but in the latest master branch, I do see what looks like a memory leak in an error return for `esp_partition_write` in `esp_partition/partition_linux.c`. I'm not sure if this is the version of `esp_partition_write` being called, but the leak occurs if it returns an error before the buffer is freed. You should check the return code of your `esp_partition_write` for error.

Code: Select all

esp_err_t esp_partition_write(const esp_partition_t *partition, size_t dst_offset, const void *src, size_t size)
{
    ...
    uint8_t *write_buf = malloc(size);
    if (write_buf == NULL) {
        return ESP_ERR_NO_MEM;
    }

    void *dst_addr = s_spiflash_mem_file_buf + partition->address + dst_offset;
    ESP_LOGV(TAG, "esp_partition_write(): partition=%s dst_offset=%zu src=%p size=%zu (real dst address: %p)", partition->label, dst_offset, src, size, dst_addr);

    // hook gathers statistics and can emulate limited number of write cycles
    if (!ESP_PARTITION_HOOK_WRITE(dst_addr, size)) {
        return ESP_FAIL;   // <---- write_buf was successfully allocated above, but not freed
    }
    ...

pablopabota
Posts: 4
Joined: Fri Jan 06, 2023 6:54 pm

Re: Memory leakage: esp_partition_write

Postby pablopabota » Mon Jan 23, 2023 9:19 pm

Hi mbratch,

I'm using 4.4.3 IDF version.

The semaphores are implemented an working, I just omit those pieces of code just to keep it clean.

I have been comenting different lines through my main code until I found that 'esp_partition_write' (inside my custom 'write_value_to_flash' function) is the function that causes the memory leakage.

The solution you say is to add a 'free(write_buf);' in the partition_linux.c function?

Checking on VSCode, I'm using the 'esp_write_partition' from the '/spi_flash/partition.c' file.

Hope that helps to find a solution to it, or any idea would be welcome.

Regards,
P

User avatar
mbratch
Posts: 303
Joined: Fri Jun 11, 2021 1:51 pm

Re: Memory leakage: esp_partition_write

Postby mbratch » Tue Jan 24, 2023 3:09 am

pablopabota wrote:
Mon Jan 23, 2023 9:19 pm
The semaphores are implemented an working, I just omit those pieces of code just to keep it clean.
So they don't allocate any dynamic resources?
The solution you say is to add a 'free(write_buf);' in the partition_linux.c function?
I'm not quite saying that. What I noticed is that there is one error condition in that function that returns without freeing a buffer, and I didn't know if that's the instance of that function you are calling. If it were the function and it is returning ESP_OK to you, then that leak does not apply to you. So my suggestion was to check the return value to see if it is actually succeeding. Currently, you are ignoring the return value.
Checking on VSCode, I'm using the 'esp_write_partition' from the '/spi_flash/partition.c' file.
That instance of the function doesn't seem to be performing any dynamic resource allocation. So I don't see how it would be leaking memory, unless I missed it reading the source.

Bear in mind that I was looking at the source in the master branch as I didn't know what version of ESP-IDF you were using. If you're using 4.4.3, it's a little older than that so the functions I looked at may be different. Worth checking.

pablopabota
Posts: 4
Joined: Fri Jan 06, 2023 6:54 pm

Re: Memory leakage: esp_partition_write

Postby pablopabota » Tue Jan 24, 2023 4:24 pm

So they don't allocate any dynamic resources?
They don't, 'xSemaphoreTakeRecursive' and 'xSemaphoreGiveRecursive' is used. As native freeRTOS functions I guess there is no problem with them, also only when 'esp_partition_write' is commented is when memory leakage ceases.
I'm not quite saying that. What I noticed is that there is one error condition in that function that returns without freeing a buffer, and I didn't know if that's the instance of that function you are calling. If it were the function and it is returning ESP_OK to you, then that leak does not apply to you. So my suggestion was to check the return value to see if it is actually succeeding. Currently, you are ignoring the return value.
The return value of the 'esp_partition_write' is ESP_OK (checked with esp_err_to_name function)

On the other hand I cannot find any allocation inside the IDF functions, here is what I got:

Code: Select all

esp_err_t esp_partition_write(const esp_partition_t* partition,
                             size_t dst_offset, const void* src, size_t size)
{
    assert(partition != NULL);
    if (dst_offset > partition->size) {
        return ESP_ERR_INVALID_ARG;
    }
    if (dst_offset + size > partition->size) {
        return ESP_ERR_INVALID_SIZE;
    }
    dst_offset = partition->address + dst_offset;
    if (!partition->encrypted) {
#ifndef CONFIG_SPI_FLASH_USE_LEGACY_IMPL
        return esp_flash_write(partition->flash_chip, src, dst_offset, size);
#else
        return spi_flash_write(dst_offset, src, size);
#endif // CONFIG_SPI_FLASH_USE_LEGACY_IMPL
    } else {
#if CONFIG_SPI_FLASH_ENABLE_ENCRYPTED_READ_WRITE
        if (partition->flash_chip != esp_flash_default_chip) {
            return ESP_ERR_NOT_SUPPORTED;
        }
#ifndef CONFIG_SPI_FLASH_USE_LEGACY_IMPL
        return esp_flash_write_encrypted(partition->flash_chip, dst_offset, src, size);
#else
        return spi_flash_write_encrypted(dst_offset, src, size);
#endif // CONFIG_SPI_FLASH_USE_LEGACY_IMPL
#else
        return ESP_ERR_NOT_SUPPORTED;
#endif // CONFIG_SPI_FLASH_ENABLE_ENCRYPTED_READ_WRITE
    }
}

esp_err_t IRAM_ATTR esp_flash_write(esp_flash_t *chip, const void *buffer, uint32_t address, uint32_t length)
{
    esp_err_t err = rom_spiflash_api_funcs->chip_check(&chip);
    VERIFY_CHIP_OP(write);
    CHECK_WRITE_ADDRESS(chip, address, length);
    if (buffer == NULL || address > chip->size || address+length > chip->size) {
        return ESP_ERR_INVALID_ARG;
    }
    if (length == 0) {
        return ESP_OK;
    }

    //when the cache is disabled, only the DRAM can be read, check whether we need to copy the data first
    bool direct_write = chip->host->driver->supports_direct_write(chip->host, buffer);

    // Indicate whether the bus is acquired by the driver, needs to be released before return
    bool bus_acquired = false;
    err = ESP_OK;
    /* Write output in chunks, either by buffering on stack or
       by artificially cutting into MAX_WRITE_CHUNK parts (in an OS
       environment, this prevents writing from causing interrupt or higher priority task
       starvation.) */
    uint32_t write_addr = address;
    uint32_t len_remain = length;
    while (1) {
        uint32_t write_len;
        const void *write_buf;
        uint32_t temp_buf[8];
        if (direct_write) {
            write_len = MIN(len_remain, MAX_WRITE_CHUNK);
            write_buf = buffer;
        } else {
            write_len = MIN(len_remain, sizeof(temp_buf));
            memcpy(temp_buf, buffer, write_len);
            write_buf = temp_buf;
        }

        //check before the operation, in case this is called too close to the last operation
        if (chip->chip_drv->yield) {
            err = chip->chip_drv->yield(chip, 0);
            if (err != ESP_OK) {
                return err;
            }
        }

        err = rom_spiflash_api_funcs->start(chip);
        if (err != ESP_OK) {
            break;
        }
        bus_acquired = true;

        err = chip->chip_drv->write(chip, write_buf, write_addr, write_len);
        len_remain -= write_len;
        assert(len_remain < length);

        if (err != ESP_OK || len_remain == 0) {
            // On ESP32, the cache re-enable is in the end() function, while flush_cache should
            // happen when the cache is still disabled on ESP32. Break before the end() function and
            // do end() later
            assert(bus_acquired);
            break;
        }

        err = rom_spiflash_api_funcs->end(chip, err);
        if (err != ESP_OK) {
            break;
        }
        bus_acquired = false;

        write_addr += write_len;
        buffer = (void *)((intptr_t)buffer + write_len);
    }

    return rom_spiflash_api_funcs->flash_end_flush_cache(chip, err, bus_acquired, address, length);
}
I'll try legacy 'spi_flash_write', yet this functions was removed in newer versions, if I'm not wrong.

Code: Select all

esp_err_t IRAM_ATTR spi_flash_write(size_t dst, const void *srcv, size_t size)
{
    const spi_flash_guard_funcs_t *guard =  spi_flash_guard_get();
    CHECK_WRITE_ADDRESS(dst, size);
    // Out of bound writes are checked in ROM code, but we can give better
    // error code here
    if (dst + size > g_rom_flashchip.chip_size) {
        return ESP_ERR_INVALID_SIZE;
    }
    if (size == 0) {
        return ESP_OK;
    }

    esp_rom_spiflash_result_t rc = ESP_ROM_SPIFLASH_RESULT_OK;
    COUNTER_START();
    const uint8_t *srcc = (const uint8_t *) srcv;
    /*
     * Large operations are split into (up to) 3 parts:
     * - Left padding: 4 bytes up to the first 4-byte aligned destination offset.
     * - Middle part
     * - Right padding: 4 bytes from the last 4-byte aligned offset covered.
     */
    size_t left_off = dst & ~3U;
    size_t left_size = MIN(((dst + 3) & ~3U) - dst, size);
    size_t mid_off = left_size;
    size_t mid_size = (size - left_size) & ~3U;
    size_t right_off = left_size + mid_size;
    size_t right_size = size - mid_size - left_size;

    rc = spi_flash_unlock();
    if (rc != ESP_ROM_SPIFLASH_RESULT_OK) {
        goto out;
    }
    if (left_size > 0) {
        uint32_t t = 0xffffffff;
        memcpy(((uint8_t *) &t) + (dst - left_off), srcc, left_size);
        spi_flash_guard_start();
        rc = spi_flash_write_inner(left_off, &t, 4);
        spi_flash_guard_end();
        if (rc != ESP_ROM_SPIFLASH_RESULT_OK) {
            goto out;
        }
        COUNTER_ADD_BYTES(write, 4);
    }
    if (mid_size > 0) {
        /* If src buffer is 4-byte aligned as well and is not in a region that requires cache access to be enabled, we
         * can write directly without buffering in RAM. */
#ifdef ESP_PLATFORM
        bool direct_write = esp_ptr_internal(srcc)
                && esp_ptr_byte_accessible(srcc)
                && ((uintptr_t) srcc + mid_off) % 4 == 0;
#else
        bool direct_write = true;
#endif
        while(mid_size > 0 && rc == ESP_ROM_SPIFLASH_RESULT_OK) {
            uint32_t write_buf[8];
            uint32_t write_size = MIN(mid_size, MAX_WRITE_CHUNK);
            const uint8_t *write_src = srcc + mid_off;
            if (!direct_write) {
                write_size = MIN(write_size, sizeof(write_buf));
                memcpy(write_buf, write_src, write_size);
                write_src = (const uint8_t *)write_buf;
            }
            spi_flash_guard_start();
            rc = spi_flash_write_inner(dst + mid_off, (const uint32_t *) write_src, write_size);
            spi_flash_guard_end();
            COUNTER_ADD_BYTES(write, write_size);
            mid_size -= write_size;
            mid_off += write_size;
        }
        if (rc != ESP_ROM_SPIFLASH_RESULT_OK) {
            goto out;
        }
    }

    if (right_size > 0) {
        uint32_t t = 0xffffffff;
        memcpy(&t, srcc + right_off, right_size);
        spi_flash_guard_start();
        rc = spi_flash_write_inner(dst + right_off, &t, 4);
        spi_flash_guard_end();
        if (rc != ESP_ROM_SPIFLASH_RESULT_OK) {
            goto out;
        }
        COUNTER_ADD_BYTES(write, 4);
    }
out:
    COUNTER_STOP(write);

    spi_flash_guard_start();
    // Ensure WEL is 0 after the operation, even if the write failed.
    esp_rom_spiflash_write_disable();
    spi_flash_check_and_flush_cache(dst, size);
    spi_flash_guard_end();

    return spi_flash_translate_rc(rc);
}
Regards,
P

UPDATE: I tried using legacy and leakage still present, I guess problem is somewhere deeper in IDF. Maybe an ESP user can help with this.

Who is online

Users browsing this forum: Baidu [Spider], Gaston1980, Majestic-12 [Bot], ok-home and 98 guests