Page 1 of 1

Panic on i2c operation when powering off the board

Posted: Sat Apr 27, 2019 1:14 pm
by maldus
Hello everyone,
I'm working on a project that needs to save data in a permanent storage (in the form of an external I2C EEPROM) when powered down.
To achieve that, I have an interrupt set up on IO34 that notifies me when voltage is falling before is too late. I have approximately 200 ms from the interrupt to complete loss of power, which should be plenty to save a few dozen bytes.

From a similar work I did last year I know I cannot do i2c operations from an interrupt thread, so I launch an high priority task that just waits on a queue for a message and saves the data when one is received.
The interrupt is fired and the task wakes up, but upon beginning I2C operations the execution is disrupted by an exception:

Code: Select all

Poweroff
second print
W (181381) pwoff: Saving!
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x400852df  PS      : 0x00060031  A0      : 0x8008568c  A1      : 0x3ffb09c0  
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

A2      : 0x00000000  A3      : 0x3ffbda28  A4      : 0x00000000  A5      : 0x00000001  
A6      : 0x00060023  A7      : 0x00060723  A8      : 0x3ffc6128  A9      : 0x6001301c  
A10     : 0xffffffff  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x0000010b  
A14     : 0x3ffc6140  A15     : 0x00060723  SAR     : 0x00000013  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xffffffff  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0xffffffff  
Core 0 was running in ISR context:
EPC1    : 0x400852df  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x00000000
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854


Backtrace: 0x400852df:0x3ffb09c0 0x40085689:0x3ffb09f0 0x40082401:0x3ffb0a20 0x4000bfed:0x3ffbb330 0x4008ea3a:0x3ffbb340 0x400d1790:0x3ffbb360 0x400d29b0:0x3ffbb390 0x400d0fc2:0x3ffbb890 0x4008e039:0x3ffbb8b0
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

0x40085689: i2c_isr_handler_default at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

0x40082401: _xt_lowint1 at /home/maldus/Source/esp-idf/components/freertos/xtensa_vectors.S:1154

0x4008ea3a: vTaskExitCritical at /home/maldus/Source/esp-idf/components/freertos/tasks.c:4590

0x400d1790: esp_task_wdt_reset at /home/maldus/Source/esp-idf/components/esp32/task_wdt.c:322

0x400d29b0: app_main at /home/maldus/Projects/HSW/StiroRotondi/main/source/main.c:110

0x400d0fc2: main_task at /home/maldus/Source/esp-idf/components/esp32/cpu_start.c:494

0x4008e039: vPortTaskWrapper at /home/maldus/Source/esp-idf/components/freertos/port.c:403


CPU halted.
This is the code that saves into the EEPROM:

Code: Select all

extern char *revision;

void IRAM_ATTR lowVoltageSave(void *queue) {
    gpio_set_level(GPIO_NUM_2, 0);
    ets_printf("Poweroff\n");
    xQueueHandle handle = *((xQueueHandle *)queue);
    xQueueSendFromISR(handle, NULL, 0);
    ets_printf("second print\n");
}

void task_lowVoltageSave(void *queue) {
    xQueueHandle handle = *((xQueueHandle *)queue);
    xQueueReset(handle);
    portBASE_TYPE res;
    while (1) {
        res = xQueueReceive(handle, NULL, 10000 / portTICK_RATE_MS);
        if (res == pdTRUE) {
            ESP_LOGW("pwoff", "Saving!");
            esp_err_t        ret;
            i2c_cmd_handle_t cmd;

            uint8_t ControlByte = EEPROM_ADD;
            uint8_t HighAdd = ADD_REV_HIGH;
            uint8_t LowAdd = ADD_REV_LOW;
            ControlByte &= 0xF1;
            ControlByte |= HighAdd << 1;

            cmd = i2c_cmd_link_create();
            i2c_master_start(cmd);
            i2c_master_write_byte(cmd, ControlByte, 1);
            i2c_master_write_byte(cmd, LowAdd, 1);
            i2c_master_write(cmd, (uint8_t*)revision, 11, 1);
            i2c_master_stop(cmd);

            gpio_set_level(GPIO_NUM_25, 0); // Write protect
            ret = i2c_master_cmd_begin(I2C_NUM_0, cmd, 10 / portTICK_PERIOD_MS);
            gpio_set_level(GPIO_NUM_25, 1);
            i2c_cmd_link_delete(cmd);

            ets_delay_us(1000*1000);
            ESP_LOGW("LOW VOLTAGE SAVE", "Not Powered Off");
        }
    }

    vTaskDelete(NULL);
}

xQueueHandle lowvoltagequeue;

void init_low_voltage_save() {
    lowvoltagequeue = xQueueCreate(1, 0);
    gpio_set_intr_type(LOW_VOLTAGE_SAVE, GPIO_INTR_NEGEDGE);
    gpio_install_isr_service(0);
    gpio_isr_handler_add(LOW_VOLTAGE_SAVE, lowVoltageSave, (void *)lowvoltagequeue);
    xTaskCreate(task_lowVoltageSave, "task_lowVoltageSave", 4096, (void *)&lowvoltagequeue, 8, NULL);
}
I'm very confused because the exact same situation works on a different projects running the same ESP32 model. I also find hard to understand the error message itself: how should I proceed to find more about my problem?

Re: Panic on i2c operation when powering off the board

Posted: Sun Apr 28, 2019 1:45 am
by ESP_Sprite
What version of IDF do you use?

In general, you helpfully posted the backtrace here, which indicates what happened. In this case, i2c_master_cmd_begin_static did something wrong, specifically the command at line 854 did. It tried to load something from memory where there is no memory (hence the LoadProhibited), possibly because of a pointer that has crap in it. Specifically, the Xtensa core tells you that the load was from 0xffffffff (because that is the EXCVADDR value).

If you can tell the version of IDF you use, we can look up what's happening at that line and use that to dig further.

Re: Panic on i2c operation when powering off the board

Posted: Sun Apr 28, 2019 9:32 am
by Ritesh
maldus wrote:
Sat Apr 27, 2019 1:14 pm
Hello everyone,
I'm working on a project that needs to save data in a permanent storage (in the form of an external I2C EEPROM) when powered down.
To achieve that, I have an interrupt set up on IO34 that notifies me when voltage is falling before is too late. I have approximately 200 ms from the interrupt to complete loss of power, which should be plenty to save a few dozen bytes.

From a similar work I did last year I know I cannot do i2c operations from an interrupt thread, so I launch an high priority task that just waits on a queue for a message and saves the data when one is received.
The interrupt is fired and the task wakes up, but upon beginning I2C operations the execution is disrupted by an exception:

Code: Select all

Poweroff
second print
W (181381) pwoff: Saving!
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x400852df  PS      : 0x00060031  A0      : 0x8008568c  A1      : 0x3ffb09c0  
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

A2      : 0x00000000  A3      : 0x3ffbda28  A4      : 0x00000000  A5      : 0x00000001  
A6      : 0x00060023  A7      : 0x00060723  A8      : 0x3ffc6128  A9      : 0x6001301c  
A10     : 0xffffffff  A11     : 0x00000000  A12     : 0x00000000  A13     : 0x0000010b  
A14     : 0x3ffc6140  A15     : 0x00060723  SAR     : 0x00000013  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xffffffff  LBEG    : 0x4000c46c  LEND    : 0x4000c477  LCOUNT  : 0xffffffff  
Core 0 was running in ISR context:
EPC1    : 0x400852df  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x00000000
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854


Backtrace: 0x400852df:0x3ffb09c0 0x40085689:0x3ffb09f0 0x40082401:0x3ffb0a20 0x4000bfed:0x3ffbb330 0x4008ea3a:0x3ffbb340 0x400d1790:0x3ffbb360 0x400d29b0:0x3ffbb390 0x400d0fc2:0x3ffbb890 0x4008e039:0x3ffbb8b0
0x400852df: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

0x40085689: i2c_isr_handler_default at /home/maldus/Source/esp-idf/components/driver/i2c.c:854

0x40082401: _xt_lowint1 at /home/maldus/Source/esp-idf/components/freertos/xtensa_vectors.S:1154

0x4008ea3a: vTaskExitCritical at /home/maldus/Source/esp-idf/components/freertos/tasks.c:4590

0x400d1790: esp_task_wdt_reset at /home/maldus/Source/esp-idf/components/esp32/task_wdt.c:322

0x400d29b0: app_main at /home/maldus/Projects/HSW/StiroRotondi/main/source/main.c:110

0x400d0fc2: main_task at /home/maldus/Source/esp-idf/components/esp32/cpu_start.c:494

0x4008e039: vPortTaskWrapper at /home/maldus/Source/esp-idf/components/freertos/port.c:403


CPU halted.
This is the code that saves into the EEPROM:

Code: Select all

extern char *revision;

void IRAM_ATTR lowVoltageSave(void *queue) {
    gpio_set_level(GPIO_NUM_2, 0);
    ets_printf("Poweroff\n");
    xQueueHandle handle = *((xQueueHandle *)queue);
    xQueueSendFromISR(handle, NULL, 0);
    ets_printf("second print\n");
}

void task_lowVoltageSave(void *queue) {
    xQueueHandle handle = *((xQueueHandle *)queue);
    xQueueReset(handle);
    portBASE_TYPE res;
    while (1) {
        res = xQueueReceive(handle, NULL, 10000 / portTICK_RATE_MS);
        if (res == pdTRUE) {
            ESP_LOGW("pwoff", "Saving!");
            esp_err_t        ret;
            i2c_cmd_handle_t cmd;

            uint8_t ControlByte = EEPROM_ADD;
            uint8_t HighAdd = ADD_REV_HIGH;
            uint8_t LowAdd = ADD_REV_LOW;
            ControlByte &= 0xF1;
            ControlByte |= HighAdd << 1;

            cmd = i2c_cmd_link_create();
            i2c_master_start(cmd);
            i2c_master_write_byte(cmd, ControlByte, 1);
            i2c_master_write_byte(cmd, LowAdd, 1);
            i2c_master_write(cmd, (uint8_t*)revision, 11, 1);
            i2c_master_stop(cmd);

            gpio_set_level(GPIO_NUM_25, 0); // Write protect
            ret = i2c_master_cmd_begin(I2C_NUM_0, cmd, 10 / portTICK_PERIOD_MS);
            gpio_set_level(GPIO_NUM_25, 1);
            i2c_cmd_link_delete(cmd);

            ets_delay_us(1000*1000);
            ESP_LOGW("LOW VOLTAGE SAVE", "Not Powered Off");
        }
    }

    vTaskDelete(NULL);
}

xQueueHandle lowvoltagequeue;

void init_low_voltage_save() {
    lowvoltagequeue = xQueueCreate(1, 0);
    gpio_set_intr_type(LOW_VOLTAGE_SAVE, GPIO_INTR_NEGEDGE);
    gpio_install_isr_service(0);
    gpio_isr_handler_add(LOW_VOLTAGE_SAVE, lowVoltageSave, (void *)lowvoltagequeue);
    xTaskCreate(task_lowVoltageSave, "task_lowVoltageSave", 4096, (void *)&lowvoltagequeue, 8, NULL);
}
I'm very confused because the exact same situation works on a different projects running the same ESP32 model. I also find hard to understand the error message itself: how should I proceed to find more about my problem?
As ESP32 IDF developer mentioned that please provide ESP32 IDF version details.

Also, Did you check with commenting I2C begin static code line just to verify that you are facing issue particularly on that call or not?

Because, We have used I2C expander on our board and not facing any issue while communicating using I2C calls.

Re: Panic on i2c operation when powering off the board

Posted: Mon Apr 29, 2019 8:52 am
by maldus
Sorry for the delay,
I'm working with the release/3.2 branch.
Also, Did you check with commenting I2C begin static code line just to verify that you are facing issue particularly on that call or not?
Yes, I can confirm that there is no problem if I comment out the i2c_master_cmd_begin function.

Re: Panic on i2c operation when powering off the board

Posted: Fri May 03, 2019 5:32 pm
by Ritesh
maldus wrote:
Mon Apr 29, 2019 8:52 am
Sorry for the delay,
I'm working with the release/3.2 branch.
Also, Did you check with commenting I2C begin static code line just to verify that you are facing issue particularly on that call or not?
Yes, I can confirm that there is no problem if I comment out the i2c_master_cmd_begin function.
Hi,

Would you please check on latest master branch?

Meanwhile, ESP32 developers can look into that in details to find out reason for failure.

Re: Panic on i2c operation when powering off the board

Posted: Tue May 07, 2019 6:48 am
by maldus
I checked out the master branch (commit df61612f8ffeb61ac46ad9dbfcf6bd00e691a920) with more or less the same result:

Code: Select all

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40083b6f  PS      : 0x00060031  A0      : 0x80083ef1  A1      : 0x3ffb1110  
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

A2      : 0x00000000  A3      : 0x3ffbd5fc  A4      : 0x6001301c  A5      : 0x00ffffff  
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x01000000  A9      : 0x00000800  
A10     : 0x3ffc6f00  A11     : 0x00000000  A12     : 0x3ff53058  A13     : 0x00000000  
A14     : 0x3ff53000  A15     : 0x0000000b  SAR     : 0x00000011  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00ffffff  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  
Core 1 was running in ISR context:
EPC1    : 0x40083b6f  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x00000000
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880


ELF file SHA256: 78feffb99d2ac6c5377ed1a7a5e0ddb82b607bde2ada1b440f42ad1f7c8d9133

Backtrace: 0x40083b6f:0x3ffb1110 0x40083eee:0x3ffb1140 0x40081e9d:0x3ffb1170 0x400d2a33:0x3ffbc680 0x4008c085:0x3ffbc6a0 0x4008af95:0x3ffbc6c0
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

0x40083eee: i2c_isr_handler_default at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

0x40081e9d: _xt_lowint1 at /home/maldus/Source/esp-idf/components/freertos/xtensa_vectors.S:1154

0x400d2a33: esp_vApplicationIdleHook at /home/maldus/Source/esp-idf/components/esp_common/src/freertos_hooks.c:86

0x4008c085: prvIdleTask at /home/maldus/Source/esp-idf/components/freertos/tasks.c:5094

0x4008af95: vPortTaskWrapper at /home/maldus/Source/esp-idf/components/freertos/port.c:403
Line 880 of esp-idf/components/driver/i2c.c is the following, both in master and 3.2:

Code: Select all

esp_err_t i2c_isr_free(intr_handle_t handle)
{
    return esp_intr_free(handle);
} // < Line 880

Re: Panic on i2c operation when powering off the board

Posted: Wed May 08, 2019 8:20 am
by Ritesh
maldus wrote:
Tue May 07, 2019 6:48 am
I checked out the master branch (commit df61612f8ffeb61ac46ad9dbfcf6bd00e691a920) with more or less the same result:

Code: Select all

Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x40083b6f  PS      : 0x00060031  A0      : 0x80083ef1  A1      : 0x3ffb1110  
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

A2      : 0x00000000  A3      : 0x3ffbd5fc  A4      : 0x6001301c  A5      : 0x00ffffff  
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x01000000  A9      : 0x00000800  
A10     : 0x3ffc6f00  A11     : 0x00000000  A12     : 0x3ff53058  A13     : 0x00000000  
A14     : 0x3ff53000  A15     : 0x0000000b  SAR     : 0x00000011  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00ffffff  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  
Core 1 was running in ISR context:
EPC1    : 0x40083b6f  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x00000000
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880


ELF file SHA256: 78feffb99d2ac6c5377ed1a7a5e0ddb82b607bde2ada1b440f42ad1f7c8d9133

Backtrace: 0x40083b6f:0x3ffb1110 0x40083eee:0x3ffb1140 0x40081e9d:0x3ffb1170 0x400d2a33:0x3ffbc680 0x4008c085:0x3ffbc6a0 0x4008af95:0x3ffbc6c0
0x40083b6f: i2c_master_cmd_begin_static at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

0x40083eee: i2c_isr_handler_default at /home/maldus/Source/esp-idf/components/driver/i2c.c:880

0x40081e9d: _xt_lowint1 at /home/maldus/Source/esp-idf/components/freertos/xtensa_vectors.S:1154

0x400d2a33: esp_vApplicationIdleHook at /home/maldus/Source/esp-idf/components/esp_common/src/freertos_hooks.c:86

0x4008c085: prvIdleTask at /home/maldus/Source/esp-idf/components/freertos/tasks.c:5094

0x4008af95: vPortTaskWrapper at /home/maldus/Source/esp-idf/components/freertos/port.c:403
Line 880 of esp-idf/components/driver/i2c.c is the following, both in master and 3.2:

Code: Select all

esp_err_t i2c_isr_free(intr_handle_t handle)
{
    return esp_intr_free(handle);
} // < Line 880
Hi,

As per issue description, it seems that issue is due to while de-allocating interrupt into I2C communication. SO, Would you please check that handle is free or not into I2C Driver itself?

Re: Panic on i2c operation when powering off the board

Posted: Wed May 08, 2019 9:15 am
by maldus
I seem to have figured it out. I got carried away from the real problem because of the backtrace; the problem had nothing to do with interrupts, I was using a variable declared as extern uint8_t* while in reality it is an uint8_t array (as opposed to a pointer). As a result the i2c driver was dereferencing an invalid address.

Thanks for the support, even if in the end it was little more than a typo.

Re: Panic on i2c operation when powering off the board

Posted: Fri May 10, 2019 3:07 am
by Ritesh
maldus wrote:
Wed May 08, 2019 9:15 am
I seem to have figured it out. I got carried away from the real problem because of the backtrace; the problem had nothing to do with interrupts, I was using a variable declared as extern uint8_t* while in reality it is an uint8_t array (as opposed to a pointer). As a result the i2c driver was dereferencing an invalid address.

Thanks for the support, even if in the end it was little more than a typo.
Great. Good findings..