(resolved) I2C call panicking system

[mzimmers]

Hi Fly - here's the prototype:

Code: Select all

    esp_err_t readReg(uint8_t devAddr, uint8_t regAddr, uint16_t *data);

As I was preparing to produce a minimal example, a thought occurred to me. I moved the code that caused the problem to the top of my app_main(), and it works. I now believe that I may be running out of heap, and that the i2c library isn't handling that properly (hence the i2c malloc error I see).

I swear, it almost seems like a timing issue -- if I call the readReg too often (or too quickly in succession) I see this problem. It's as though FreeRTOS needs a little time to clean up the memory used in the I2C library.

[vonnieda]

Create a FreeRTOS task that prints out the free memory every 5 seconds or so. Once your app is up, watch the printout and see if the memory continues to go down. If it does, start commenting out code until the memory stops going down. Then find the culprit. I also recommend printing out internal memory as well.

For reference, here are the ones I print out when I am trying to track something down:

heap_caps_get_free_size(MALLOC_CAP_8BIT),
heap_caps_get_free_size(MALLOC_CAP_32BIT),
heap_caps_get_free_size(MALLOC_CAP_32BIT) - heap_caps_get_free_size(MALLOC_CAP_8BIT),
heap_caps_get_free_size(MALLOC_CAP_INTERNAL),
heap_caps_get_free_size(MALLOC_CAP_SPIRAM),
heap_caps_get_free_size(MALLOC_CAP_DMA),

Additionally, if you are creating and deleting tasks, make sure you are not keeping the IDLE task from running. IDLE is responsible for cleaning up deleted tasks. This will happen if you have a higher priority task that never sleeps.

Also, I can tell you that the i2c master library is pretty darn stable for me. I run tens of transactions per second to multiple devices in my app for weeks at a time without any problems.

Finally, when you see a stack dump that points to a line number that is the end of a function, you can't really trust that stack dump. I don't know the exact reason why this happens - I suspect it's due to stack corruption - but when it does the error is seldom actually in the function indicates by the stack dump.

Jason

[mzimmers]

Hi Jason -

Thanks for the valuable information. I think you might be onto something regarding the IDLE task. While I don't do any task deletion (I create several at start-up), I wonder if the IDLE task is also responsible for memory management. I don't think any of my tasks should be running continuously but I can check. When you said "never sleeps" did you mean that literally? Most of my inter-task communication is through queues, so my tasks spend a lot of time waiting on a queue.

[vonnieda]

Hi Jason -

Thanks for the valuable information. I think you might be onto something regarding the IDLE task. While I don't do any task deletion (I create several at start-up), I wonder if the IDLE task is also responsible for memory management. I don't think any of my tasks should be running continuously but I can check. When you said "never sleeps" did you mean that literally? Most of my inter-task communication is through queues, so my tasks spend a lot of time waiting on a queue.

Blocking on a queue will do the trick, but if you aren't deleting tasks it's probably not an issue. Memory management is synchronous as far as I know. The most likely scenario is that you either have a memory leak or you are double freeing a pointer.

Jason

[mzimmers]

It's almost surely not a memory leak:

I (147593) memreport: memReport(): there are 114072 MALLOC_CAP_8BIT bytes free.
I (147593) memreport: memReport(): there are 163936 MALLOC_CAP_32BIT bytes free.
I (147593) memreport: memReport(): there are 163936 MALLOC_CAP_INTERNAL bytes free.
I (147603) memreport: memReport(): there are 0 MALLOC_CAP_SPIRAM bytes free.
I (147613) memreport: memReport(): there are 114072 MALLOC_CAP_DMA bytes free.

And I doubt it's a double free, mainly because the program runs "for a while." All my tasks have very short loops, and there's not a lot of variant behavior, so I think execution would expose this kind of memory bug right away.

I just increased the idle time in my power manager loop (the one that always gets blamed when the program crashes) from .1 second to .5 seconds. Early indications are that it's running better. I'd really like to know what's at the bottom of this, though.

[mzimmers]

I left this running overnight...it's definitely more reliable than it was, but I still get an occasional error like this:

E (2085053) i2c: i2c command link malloc error
abort() was called at PC 0x4015c4ab on core 1

I make a series of calls to the i2c library every half second, and according to the tick-stamp, this ran for better than 30 minutes before the malloc error. I'm hard pressed to see how this wouldn't be some kind of bug in the i2c library.

[vonnieda]

I left this running overnight...it's definitely more reliable than it was, but I still get an occasional error like this:

E (2085053) i2c: i2c command link malloc error
abort() was called at PC 0x4015c4ab on core 1

I make a series of calls to the i2c library every half second, and according to the tick-stamp, this ran for better than 30 minutes before the malloc error. I'm hard pressed to see how this wouldn't be some kind of bug in the i2c library.

A malloc error likely means that the library tried to allocate some memory and could not. This still points to a memory leak. Did you try my suggestion of printing out the memory usage every 5 seconds? Over the course of 30 minutes, if there is a leak, this should be very clear.

[mzimmers]

Hi - yes I did: the results never moved:

I (619406) memreport: memReport(): there are 114124 MALLOC_CAP_8BIT bytes free.
I (619406) memreport: memReport(): there are 163988 MALLOC_CAP_32BIT bytes free.
I (619406) memreport: memReport(): there are 163988 MALLOC_CAP_INTERNAL bytes free.
I (619416) memreport: memReport(): there are 0 MALLOC_CAP_SPIRAM bytes free.
I (619426) memreport: memReport(): there are 114124 MALLOC_CAP_DMA bytes free

I kept getting these exact numbers right up to the malloc error.

[fly135]

I make a series of calls to the i2c library every half second, and according to the tick-stamp, this ran for better than 30 minutes before the malloc error. I'm hard pressed to see how this wouldn't be some kind of bug in the i2c library.

I use I2C all the time and am not seeing a mem leak. I do reboot my devices every 24 hours just to avoid long term issues. But I have also run them for long periods without the automatic reboot. In general I my I2C reads aren't high speed. But there are several chips on I2C and I'm reading them about once a second.

Also, just wanted to make sure that the function was expecting the address of a 16 bit variable instead of an int.

John A

[fly135]

Why not write the simplest possible program to read the I2C and see if it works without failing? That would get rid of your worries about the underlying library.

John A