(resolved) I2C call panicking system

[mzimmers]

Hi all -

I've been struggling with this problem off and on for months now. I'd been getting intermittent problems when making calls to the I2C library. I've now got it recurring repeatedly. Here's the trace:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x4015e46a PS : 0x00060830 A0 : 0x8015f41e A1 : 0x3ffd4e40
A2 : 0x3ffd7324 A3 : 0x3ffd4e7c A4 : 0x3ffe1de8 A5 : 0x3ffbc730
A6 : 0x00000000 A7 : 0x3ffd4140 A8 : 0xfefefefe A9 : 0x3ffd4e20
A10 : 0x3ffe1d7c A11 : 0x00000014 A12 : 0x00000010 A13 : 0x3ffe1df8
A14 : 0x3ffd4eb3 A15 : 0x00000000 SAR : 0x0000000d EXCCAUSE: 0x0000001c
EXCVADDR: 0xfefeff0e LBEG : 0x4000c46c LEND : 0x4000c477 LCOUNT : 0x00000000

ELF file SHA256: 65a72938982fcd6111c3ac5a71607486e7fccfe02025477318cb5d80a8dd1f5d

Backtrace: 0x4015e46a:0x3ffd4e40 0x4015f41b:0x3ffd4e60 0x40122355:0x3ffd4eb0 0x4012288d:0x3ffd4f10 0x401119db:0x3ffd4f40 0x40111f85:0x3ffd5080 0x400926c1:0x3ffd50a0

0x40122355 (the 3rd element in the backtrace) is my code (the 2nd i2c_master_write_byte() call):

Code: Select all

    m_i2c_cmd = i2c_cmd_link_create();
    if (m_i2c_cmd != nullptr)
    {
        //ESP_LOGI(TAG, "i2cReadReg(): m_i2c_cmd is %x.", (unsigned) m_i2c_cmd);
        // build the i2c transfer.
        rc = (i2c_master_start(m_i2c_cmd));
        rc |= (i2c_master_write_byte(m_i2c_cmd, addrWrite, true));
        rc |= (i2c_master_write_byte(m_i2c_cmd, regAddr, true));
        rc |= (i2c_master_start(m_i2c_cmd));
        rc |= (i2c_master_write_byte(m_i2c_cmd, addrRead, true));
        rc |= (i2c_master_read(m_i2c_cmd, (uint8_t *) data, len, I2C_MASTER_LAST_NACK));
        rc |= (i2c_master_stop(m_i2c_cmd));

The 2nd element in the backtrace is the return statement for i2c_master_write_byte():

Code: Select all

esp_err_t i2c_master_write_byte(i2c_cmd_handle_t cmd_handle, uint8_t data, bool ack_en)
{
    I2C_CHECK(cmd_handle != NULL, I2C_CMD_LINK_INIT_ERR_STR, ESP_ERR_INVALID_ARG);
    i2c_cmd_t cmd;
    cmd.ack_en = ack_en;
    cmd.ack_exp = 0;
    cmd.ack_val = 0;
    cmd.byte_num = 1;
    cmd.op_code = I2C_CMD_WRITE;
    cmd.data = NULL;
    cmd.byte_cmd = data;
    return i2c_cmd_link_append(cmd_handle, &cmd);
}

And the 1st element in the backtrace is, oddly enough, the closing brace for this function:

Code: Select all

esp_err_t i2c_isr_free(intr_handle_t handle)
{
    return esp_intr_free(handle);
}

I'd have expected to see i2c_cmd_link_append() in the backtrace...not sure what's going on. I can't even find where i2c_isr_free is being called. But anyway...

Can someone see into this at all? I know I'm getting low on heap, but I'd expect a different error if that were the cause.

Thanks...

[mzimmers]

Another data point: what I said about being low on memory evidently was wrong: I put in a call to heap_caps_get_free_size(), and it tells me there are ~118K bytes free.

[mzimmers]

Bump.

I don't know if I'm asking the wrong questions, but I'm hoping that someone can assist with this. Currently, my app is quite fragile, and I don't know whether to look at the CPU, the I2C interface, the OS/drivers or my own software.

Should I consider opening a bug on gitsrv for this?

[ESP_Sprite]

It's hard to say, as your code by itself doesn't seem to do much wrong, Your backtrace gives some indication, however. In menuconfig, does your heap allocation checking happen to be set to 'comprehensive'?

I'm asking because:
- You have a LoadStoreException. For these exceptions, the CPU will put the address it tried to load from in EXCVADDR. In your case, this is 0xfefeff0e.
- A load in Xtensa can only be from a register +/- a certain offset. We can't tell what the offset is exactly, but as the only register haing contents starting with 0xFEFE is A8, we can be sure that that register contains a pointer to the structure the CPU was trying to read a member from.
- The contents of this register (and thus, the pointer) are 0xFEFEFEFE. According to the docs, this is a value left by the memory allocator when a memory region is freed.

So whatever your issue is, it's likely happening because somehow you're using a pointer after the memory is in already has been freed.

[mzimmers]

Hi Sprite -

I agree, except that I don't think it's "my" code...I think that's exactly the error that's occurring in the I2C library. It seems likely that the call to esp_intr_free() is what's causing the panic.

The bigger question, though, is...what is Espressif going to do about it? This is kind of a big deal.

[ESP_Sprite]

Can you provide a minimal example of the code that causes this? If you indeed think this is a bug in ESP-IDF, you're probably better off posting it as an issue on the esp-idf Github page.

[mzimmers]

OK, I'll try to put something together. It might take a little time, given that the I2C portion of my app was the last thing I added, but I'll see what I can do. Thanks, Sprite.

[mzimmers]

Still working on producing a minimal example. I have made a little progress, though. I have a passage of code that uses the I2C device four times:

Code: Select all

uint16_t Max77818::getBatteryRepCap()
{
    uint16_t percent = 0;
    uint16_t sample;

    readReg(MAX77818_ADDR_FUELGAUGE, MAX77818_REG_FUELGAUGE_REPCAP, &sample);

    // register LSB is 0.5mAh, so divide by 2 to attain mAh.
    percent = sample / 2;
    return percent;
}
...
    info->voltage = getBatteryVoltage();
    info->TTE = getBatteryTTE();
    info->SoC = getBatterySoC();
    info->repCap = getBatteryRepCap();

Each of these calls makes a call to readReg(), which invokes the sequence I posted above. If I comment out the call to getBatteryRepCap(), the program starts successfully. But...if I attempt to read too quickly, I'll get this:

CORRUPT HEAP: Bad head at 0x3ffe2538. Expected 0xabba1234 got 0x3ffe2774
assertion "head != NULL" failed: file "C:/esp-idf/components/heap/multi_heap_poisoning.c", line 214, function: multi_heap_free

This may or may not be a related issue, but the backtrace leads into the driver/i2c.c code.

My program should be accessing the I2C library sequentially, so it shouldn't be a problem of code that isn't re-entrant.

I tried using the taskENTER_CRITICAL calls but those blow up on me, too.

Any suggestions here would be GREATLY appreciated. Thanks...

[ESP_Sprite]

Still hard to say without seeing the entire source, sorry.

[fly135]

Looks to me like you are writing outside of allocated memory. What's the function prototype for readReg?