ESP32 malloc gives memory in invalid location or guru meditation

[Serverguy]

The ESP32 is allocating memory in an unusable area. There are variations of this error. The first variation is the one I want to work. I am allocating 76800 integers in a single array from the heap. The heap, to start with, is almost 300K. So giving me 76800 is not a problem. It does that. Then I load up the array for a test with integers. When I read back those integers, it stops with a value mismatch at address 0x40000000 (1 GB). Well, what is this tiny controller doing up at 1GB? And why does it allocate my array past that boundary into, apparently, invalid addresses? When I read it back, it gives me random data.

19:32:02.862 -> Array should have 9600 bytes.
19:32:02.862 -> Array located at 0x3ffb2068
19:32:02.862 -> Last element located at 0x3ffbb668
19:32:02.862 -> FreeHeapSize before malloc 290752, after 280936,
19:32:02.862 -> used: 9816
19:32:02.862 -> Array located at 0x3ffb2068

So, I reduced the size to 10000. That causes it to fail in a different way. Now, the very first entry mismatches when I read it back. When I use 10000, it gets guru meditation and that's all for that run.

None of this makes any sense! There is plenty of memory. It allocates it for me but in an address area that is invalid. And random other errors. I will put specifics next and the test program last. Yes, it works just fine in PSRAM but the unit I want to use for this program does not have PSRAM.
The same program fails on T14, T7 and T5 T-Display. All want to allocate invalid memory addresses.

The first lines are setting array elements to 100000 - i. They print back good in the loop but print back bad when reprinted. They also are bad in the checking routine. Locations 0 and 38400 are OK. But anything above 0x40000000 is good, then bad. No idea why it prints good once then bad immediately afterwards. Same print line (code) prints in both places:

Code: Select all

Serial.printf("Location %i contains %i\n", i, array[i]);

19:12:46.898 -> Setting location 76797 at address 0x4000c0fc to 23203
19:12:46.898 -> Location 76797 contains 23203
19:12:46.898 -> Setting location 76798 at address 0x4000c100 to 23202
19:12:46.898 -> Location 76798 contains 23202
19:12:46.898 -> Setting location 76799 at address 0x4000c104 to 23201
19:12:46.898 -> Location 76799 contains 23201
19:12:46.898 -> Location 0 contains 100000
19:12:46.898 -> Location 38400 contains 61600
19:12:46.898 -> Location 76797 contains 124004104
19:12:46.898 -> Location 76798 contains 860554088
19:12:46.898 -> Location 76799 contains 1430979945
19:12:46.898 -> /***************************************************************************/
19:12:46.898 -> /***************************************************************************/
19:12:46.898 -> Checking array
19:12:46.898 -> /***************************************************************************/
19:12:46.898 -> /***************************************************************************/
19:12:46.898 -> Array size to check: 76800
19:12:46.933 -> Location 64446 mismatch at address 0x40000000
19:12:46.933 -> Should be 35554, contains 273270016
19:12:46.933 -> Location 0 contains 100000
19:12:46.933 -> Location 38400 contains 61600
19:12:46.933 -> Location 76797 contains 124004104
19:12:46.933 -> Location 76798 contains 860554088
19:12:46.933 -> Location 76799 contains 1430979945

Now I change the array size from 76800 to 10000 via:

Code: Select all

const uint32_t buffSize = 10000;

Seems like it should run just fine now since it ran OK until location 64446 last time. NOPE! It crashes loading location 1000.

19:19:27.895 -> Setting location 998 at address 0x3ffb3000 to 99002
19:19:27.895 -> Location 998 contains 99002
19:19:27.895 -> Setting location 999 at address 0x3ffb3004 to 99001
19:19:27.895 -> Location 999 contains 99001
19:19:27.975 -> Guru Meditation Error: Core 1 panic'ed (StoreProhibited). Exception was unhandled.
19:19:27.975 -> Core 1 register dump:
19:19:27.975 -> PC : 0x4008bf1e PS : 0x00050033 A0 : 0x4008be27 A1 : 0x3ffb1e30
19:19:27.975 -> A2 : 0x00016ea3 A3 : 0x00000000 A4 : 0x00000001 A5 : 0x4008bda8
19:19:27.975 -> A6 : 0x3ff000e0 A7 : 0x00000001 A8 : 0x80085139 A9 : 0x3ffbeb20
19:19:27.975 -> A10 : 0x3ffbeb58 A11 : 0x00000001 A12 : 0x00000001 A13 : 0x00000001
19:19:27.975 -> A14 : 0x00060021 A15 : 0x00000000 SAR : 0x00000017 EXCCAUSE: 0x0000001d
19:19:27.975 -> EXCVADDR: 0x00016ea3 LBEG : 0x400882d5 LEND : 0x400882e5 LCOUNT : 0xffffffff
19:19:27.975 ->
19:19:27.975 -> ELF file SHA256: 0000000000000000
19:19:27.975 ->
19:19:27.975 -> Backtrace: 0x4008bf1e:0x3ffb1e30 0x4008be24:0x3ffb1e40

This dump decoder does not return any lines at all in the bottom window. It has no idea what it is looking at. That makes 2 of us!

This makes NO SENSE! A smaller array crashes. A larger array works but gets into invalid memory space.

I tried using "static" for the pointer but that did not work any better. Please help me understand what is going on here. I am getting no help from Espressif. They seem mystified although they have all the tools to look into this stuff, nothing is happening. If you want any more tests, please copy the code, below, and try it in the Arduino IDE.

Thanks for any help solving this. Mike.

Code: Select all

#define TFT_WIDTH 320
#define TFT_HEIGHT 240
const uint32_t buffSize = 10000;  // TFT_HEIGHT * TFT_WIDTH;
int preHeapSize;
//static unsigned int *workBuff;
long int i;

/***************************************************************************/
void setup()
/***************************************************************************/
{
  Serial.begin(500000); delay(1000);
  Serial.printf("\n\nArray should have %i bytes.\n", buffSize);
  preHeapSize = xPortGetFreeHeapSize();
  static unsigned int *workBuff = (unsigned int*)malloc(buffSize);  // Fails
  //  workBuff = (unsigned int*)ps_malloc(buffSize);
  //  workBuff = (uint32_t*)malloc(buffSize);
  Serial.printf("Array located at %p\n", workBuff);
  Serial.printf("Last element located at %p\n", workBuff + buffSize);
  Serial.printf("FreeHeapSize before malloc %i, after %i, \nused: %i\n",
                preHeapSize, xPortGetFreeHeapSize(),
                preHeapSize - xPortGetFreeHeapSize());
  Serial.printf("Array located at %p\n", workBuff);
  workBuff[0] = 12;  // Just checking...
  FillArray(workBuff);
  CheckArray(workBuff);
}
/***************************************************************************/
void FillArray (unsigned int *array)
/***************************************************************************/
{
  Serial.printf("Filling array\n");
  Serial.println(array[0]);  // Just checking...
  array[0] = 24;             // Just checking...
  Serial.println(array[0]);  // Just checking...
  for (i = 0; i < buffSize; i++) {
    if (i > buffSize - 500 || i < 1000) {
      Serial.printf("Setting location %i at address %p to %i\n",
                    i, array + i, 100000 - i);
    }

    array[i] = 100000 - i;

    if (i > buffSize - 500 || i < 1000) {
      Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
    }
    delay(0);
  }
  i = 0;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize / 2;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 3;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 2;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 1;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
}

/***************************************************************************/
void CheckArray (unsigned int *array)
/***************************************************************************/
{
  Serial.println("/***************************************************************************/");
  Serial.println("/***************************************************************************/");
  Serial.printf("Checking array\n");
  Serial.println("/***************************************************************************/");
  Serial.println("/***************************************************************************/");
  Serial.printf("Array size to check: %i\n", buffSize);
  for (i = 0; i < buffSize; i++) {
    if (i > buffSize - 500)
      Serial.printf("Verify location %i contains %i\n", i, array[i]); // Just checking...
    if (array[i] != 100000 - i) {
      Serial.printf("Location %i mismatch at address %p\n", i, array + i);
      Serial.printf("Should be %i, contains %i\n", 100000 - i, array[i]);
      break;
    }
  }
  i = 0;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize / 2;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 3;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 2;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
  i = buffSize - 1;
  Serial.printf("Location %i contains %i\n", i, array[i]); // Just checking...
}
void loop() { }

[vonnieda]

In ESP32, an int (or unsigned int) is 32 bits, or 4 bytes. This means that an array of 76800 ints consumes 76800 * 4 = 307200 bytes.

When you malloc you should always malloc the number of objects (76800) times the size of the object (sizeof(uint32_t)), so you malloc should look like malloc(sizeof(int32_t) * 76800).

The root issue is that you are only allocating 1/4 the memory you need, and then you are accessing invalid locations.

Jason

[Serverguy]

I am allocating exactly what I need. If you read what is allocated, it is 76816 bytes from the heap. Apparently, there is 16 bytes of housekeeping. Do you still say that is the wrong size? I can actually read and write about 64% of this area before it blows up. Please look, again, carefully, at what I posted and you will see all of that. If I was allocating 1/4 of what I need, the readback check would fail at about 19,200.

Still waiting for a valid solution. If you think you know how to do it, PLEASE just write the few lines of code that will do what you propose and run it. See if you can actually make this work. Start with mine and modify it. It is that easy to resolve and close this.

[lbernstone]

I am not going to debug your code, but I would guess you are allocating more than the available contiguous space. This works just fine:

Code: Select all

const char* fmtMemCk = "Free: %d\tMaxAlloc: %d\t PSFree: %d\n";
#define MEMCK Serial.printf(fmtMemCk,ESP.getFreeHeap(),ESP.getMaxAllocHeap(),ESP.getFreePsram())
const uint32_t buffSize = 31500;

void setup() {
  Serial.begin(115200);
  MEMCK;
  static unsigned int* buff = (unsigned int*)malloc(buffSize * sizeof(unsigned int));
  MEMCK;
  for (uint32_t x=0; x< buffSize; x++) {
    buff[x] = random(1000);
  }
  for (uint32_t x=0; x< buffSize; x++) {
    log_v(buff[x]);
  }
}

void loop() {
  delay(-1);
}

When run, it shows the max allocation as 126920 bytes, which works out to 31730. If you set buffSize over that, it fails with a StoreProhibited error.

[Vader_Mester]

I am allocating exactly what I need. If you read what is allocated, it is 76816 bytes from the heap. Apparently, there is 16 bytes of housekeeping. Do you still say that is the wrong size? I can actually read and write about 64% of this area before it blows up. Please look, again, carefully, at what I posted and you will see all of that. If I was allocating 1/4 of what I need, the readback check would fail at about 19,200.

Still waiting for a valid solution. If you think you know how to do it, PLEASE just write the few lines of code that will do what you propose and run it. See if you can actually make this work. Start with mine and modify it. It is that easy to resolve and close this.

What I say in your code is that you maybe want to make a buffer for a display.
In this case, using uint or int is indeed 4bytes of data. Using uint8_t gives you one byte.

EDIT:
There is a possibility, that the compiler sees that you are mallocing 32bit variables, so it wants to malloc it inside the IRAM. IRAM can be used for data, but only 32bit word addressable variables can be stored there. So, yeah, it's possible, that the compiler tries IRAM and fails.
If possible, you can use heap_cap_malloc() function to actually specify which memory you want the malloc to be placed into.

I suggest you check Ibernstone's code, in which he prints free heap to serial.

Also, every time you malloc, always check the pointer you malloced. If it is NULL, and it is not checked, a core panic and a LoadProdhibited error will be guaranteed.

A simple:

Code: Select all

if(!workBuf) {
  //code for workBuf being NULL
  //if you return from the function here no "} else {" is needed
 }

BTW, in IDF, I'm pretty much maxing our the free memory of the ESP32 in one of my project, with only 10k left, and I allocate a 100kB region and another 18k region first, then other stuff use up the rest. So it will work, you just need to be careful.

Vader(Ben)

[ESP_Sprite]

I am allocating exactly what I need.

I'm sure you intended that, but it's not what your code does.

Code: Select all

const uint32_t buffSize = 10000;  // TFT_HEIGHT * TFT_WIDTH;
static unsigned int *workBuff = (unsigned int*)malloc(buffSize);

allocates 10000 *bytes* of data (see the malloc() manpage if you want to confirm that). However, in the code after

Code: Select all

void FillArray (unsigned int *array) {
...
  for (i = 0; i < buffSize; i++) {
...
    array[i] = 100000 - i;

uses this memory as an array of *integers* which are 4 times as large as a byte. An array of 10000 bytes takes up (10000*4=)40000 bytes, and you only allocated 10000 of them. The rest of that memory is either free, or in use by other bits of code. In case of the latter, yeah, that code is free to write whatever to their regions. You actually should be happy your code didn't outright cause a crash.