ESP32 Large Array Fails

[aydintopcu]

Hello,

I'm trying to build an ESP32 (ESP32-WROOM-32D 4MB) device to collect at least 30 seconds of analog sensor data (i.e. accelerometer) by 20.000 Hz and store the data in array later in an SD card.

Since each stored the data is Float32; the data will take 30 x 20000 x 4 = 2.400.000 bytes in memory. I couldn't achieve to allocate this amount of data because the program keep crashes with below statement. How can I overcome this problem? (sample code is below. If I keep duration as dataSeconds = 1, everything is fine. But 2 and above exceeds the RAM I guess)

Guru Meditation Error: Core 1 panic'ed (StoreProhibited). Exception was unhandled.
Core 1 register dump:
PC : 0x400d0bc6 PS : 0x00060530 A0 : 0x800e9996 A1 : 0x3ffb1f80
A2 : 0x3ffc2480 A3 : 0x00000001 A4 : 0x0000c381 A5 : 0x00000001
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x800d0bc4 A9 : 0x3ffb1f60
A10 : 0x00000751 A11 : 0x00000005 A12 : 0x0000000a A13 : 0x00000003
A14 : 0x00000001 A15 : 0x00000000 SAR : 0x0000001b EXCCAUSE: 0x0000001d
EXCVADDR: 0x00000001 LBEG : 0x400d1dcc LEND : 0x400d1dd8 LCOUNT : 0x00000000

Backtrace: 0x400d0bc6:0x3ffb1f80 0x400e9993:0x3ffb1fa0

Rebooting...

Code: Select all

#include <esp_heap_caps.h>

const unsigned int samplingFr = 20000;
const unsigned int dataSeconds = 1;                      // 1 second is OK, 2 and above generates error (treshold around 23000 sample)
const unsigned int dataScan = dataSeconds * samplingFr;

byte *dataPart = (byte *) heap_caps_malloc(dataScan * sizeof(int), MALLOC_CAP_8BIT);

void setup() {
  Serial.begin(115200);
  int i = 1;
  for (i = 1; i < dataScan + 1; i++) {
    dataPart[i] = 1;                                    // Normally I put analogRead(pinNo) here, I put "1" to shorten the code here
    delayMicroseconds(1000000 / samplingFr);
  }

  for (i = 1; i < dataScan + 1; i++) {
    Serial.println(dataPart[i]);
    delayMicroseconds(1);
  }
}
void loop() {
}

[ESP_Sprite]

The ESP-Wroom does not contain a RAM chip, so only has the ESP32s internal memory to work with. An array of 2.4MiB is never going to fit in that.

[tommeyers]

Storing 32 bit numbers is one way but can you store a smaller difference value instead? (measure=n-1 - measure-n) in 16 bits with sign? That would cut memory demand in-half.

Tom Meyers

[aydintopcu]

Thanks for the answers.

@ESP_Sprite, Actually best practice would be to write raw data directly ın SDCard but write speed does not allow sampling over 1kHz. I guess wrover with 4MB SRAM will fit my need for this specific purpose but I have package constraint unfortunately.
Isn't there a possibility to allocate memory in SPI Flash to use as RAM (via partition setup or any method)?

@tomeyers, Since I need sign dependant 32bit analog values, I didn't get the point to store it as 16bits/2bytes integer. Could you share me a post/link for more explanation. Thanks in advance.

[ArtemN]

May be you may write ADC data to ESP32 FLASH and analyse it after collecting procedure? See EEPROM or SPIFFS examples and edit "partitions" folder in your espressive/esp32 folder.

[tommeyers]

First, it sounds like you are getting 32bit floating point values from your ADC or are you getting 12bit, 10bit, ... values and converting to floatingpoint and then storing? in that case store the 12, 10, ... bit values.

Second, for storing large array of big numbers numbers store instead d = (measure n - measure n-1) / m and reconstitute measure n by measure n-1 + m * d.

Tom Meyers

[tommeyers]

Here is more info on the encoding: http://www.dspguide.com/ch27/4.htm

Tom Meyers

[aydintopcu]

Here is more info on the encoding: http://www.dspguide.com/ch27/4.htm

Tom Meyers

Thank you @tommeyer; your point as storing values in 16bit format, clear now. I have a specific file format mixed with text and HEX and need a conversion to implement that, worth to deal with to double the amount

By the way wrover works perfect for such amount data storing, if only I could use more compact sibling wroom's flash as RAM.

[ESP_Sprite]

So, in theory you can use flash as swap... but in practice, it may not be worth it: first of all, flash can also be pretty slow to erase and program, and I'm not sure if that rate is enough to keep up with your sample rate. Second of all, flash only has a limited amount of erase/program cycles, and at 20Ksamples/sec, you could be liable to burning through that pretty fast.