Compiler does not report space taken by big char arrays??

[rin67630]

I am a bit confused. I have a code that contains following variables definitions:

Code: Select all

...
static int year;
static int weekday;
static IPAddress ip;
static char charbuff[32];
static char filbuff1[86410];
static char filbuff2[86410];
int pointer;
...

As you can see, I need two file buffers to store one byte per second for one day = 86400 bytes + overhead.
The compiler reports:

Code: Select all

Sketch uses 519034 bytes (39%) of program storage space. Maximum is 1310720 bytes.
Global variables use 37592 bytes (12%) of dynamic memory, leaving 257320 bytes for local variables. Maximum is 294912 bytes.
D:\Users\Michel Firholz\My Documents\Arduino\hardware\espressif\esp32/tools/esptool.exe --chip esp32 --port COM7 --baud 921600

The big char arrays do not seem to have been recognized, are they?
Where are my char arrays supposed to be stored?

[Cellie]

If you declared them in a function, they are not counted as 'Global variables' but are 'Local variables'.

Global declaration:

Code: Select all

char filbuff1[86410];  

void setup() {
  // put your setup code here, to run once:
  filbuff1[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Gives:

Global variables use 97304 bytes (32%) of dynamic memory, leaving 197608 bytes for local variables. Maximum is 294912 bytes.

Local declaration:

Code: Select all

void setup() {
  char filbuff1[86410];  

  // put your setup code here, to run once:
  filbuff1[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Gives:

Global variables use 10888 bytes (3%) of dynamic memory, leaving 284024 bytes for local variables. Maximum is 294912 bytes.

[rin67630]

I declared them global, but static.
If I omit static the sketch will not compile:
/arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld.exe: region `dram0_0_seg' overflowed by 8784 bytes.

P.S. also with
static char filbuff1[86410];
I got a compilation error
"`...dram0.bss' will not fit in region `dram0_0_seg'" and
" /../../../../xtensa-esp32-elf/bin/ld.exe: region `dram0_0_seg' overflowed by 8784 bytes"
as soon as I tried to read from the array. Funnily writing to the array even with big indexes did not produce any error.

Even if the each byte of the array takes an int, the total size should be 172.820 bytes, enough to fit into the 284252 bytes left for for local variables, isn't it?

[Cellie]

I just compiled this test:

Code: Select all

static char filbuff1[86410];  

void setup() {
  // put your setup code here, to run once:
  filbuff1[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Which gives:

Global variables use 10888 bytes (3%) of dynamic memory, leaving 284024 bytes for local variables. Maximum is 294912 bytes.

Without the 'static':

Global variables use 97304 bytes (32%) of dynamic memory, leaving 197608 bytes for local variables. Maximum is 294912 bytes.

It is the 'static' keyword that makes the arrays 'disappear'.
I think ( but am not 100% sure ) that the 'static' declaration makes the compiler treat the array as a local variable.

[Cellie]

I just tried this:

Code: Select all

char filbuff1[86410];  
char filbuff2[86410];  

void setup() {
  // put your setup code here, to run once:
  filbuff1[0]=0;
  filbuff2[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Which gives:

region `dram0_0_seg' overflowed by 68512 bytes

Which basically means not enough RAM.

[kolban]

When you declare data as follows:

static char name[length];

You are creating uninitialized but globally addressable storage. This data is in fact initialized ... to zeros. It is also known as BSS data ... see:

https://en.wikipedia.org/wiki/.bss

Since the data is uninitialized, there is no need to "copy" it from your compilation to the flash memory. For example, if you allocate 80K of buffer space that is binary zero, no need to copy 80K into flash storage of your ESP32. Instead... at run time, the environment is told that it needs to dynamically "allocate" 80K of RAM and initialize it to zeros. Now, whether you actually have enough RAM to fulfill the job, that may only be able to be detected at execution time.

[rin67630]

I just tried this:

Code: Select all

char filbuff1[86410];  
char filbuff2[86410];  

void setup() {
  // put your setup code here, to run once:
  filbuff1[0]=0;
  filbuff2[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Which gives:

region `dram0_0_seg' overflowed by 68512 bytes

Which basically means not enough RAM.

OK, for me too.
But if I use only one buffer:

Code: Select all

char filbuff1[86410]; 

void setup() {
  // put your setup code here, to run once:
  filbuff1[0]=0;
}

void loop() {
  // put your main code here, to run repeatedly:
}

Then it compiles with:
Sketch uses 157964 bytes (12%) of program storage space. Maximum is 1310720 bytes.
Global variables use 97068 bytes (32%) of dynamic memory, leaving 197844 bytes for local variables. Maximum is 294912 bytes.

There is somthing wrong here. If I have 197844 bytes left for local variables, 97068 bytes more for the second buffer should be possible, isn't it?

[rin67630]

When you declare data as follows:

static char name[length];

You are creating uninitialized but globally addressable storage. This data is in fact initialized ... to zeros. It is also known as BSS data ... see:

https://en.wikipedia.org/wiki/.bss

Since the data is uninitialized, there is no need to "copy" it from your compilation to the flash memory. For example, if you allocate 80K of buffer space that is binary zero, no need to copy 80K into flash storage of your ESP32. Instead... at run time, the environment is told that it needs to dynamically "allocate" 80K of RAM and initialize it to zeros. Now, whether you actually have enough RAM to fulfill the job, that may only be able to be detected at execution time.

I fear a misunderstanding:
The arrays should be ram, not flash storage. I expects the compiler to report the reserved RAM storage correctly.
Conferring to the example with two buffers and the example with only one, the math do not match.

[rin67630]

It seems that an ESP32 can handle a maximum of ~100.000 bytes for variables. The reported 32% of the dyn RAM is misleading!
When variables are declared static their volume will not be reported, but counts for the limit.
When libraries are included, their variables will be counted as well.
With #include <WiFi.h>, #include <WiFiUdp.h>, and #include "SSD1306.h" and some strings used around, it seems that I can't get an array
of 86410 bytes (one byte every second for a day) any more.

[kolban]

My understanding is that when we compile an application, from a RAM usage perspective, some will be initialized RAM (.data), some will be uninitialized data (.bss). Sum those together and we have the amount of RAM that is pre-allocated by the application. Within the ESP32 there is an available amount of RAM which I think is on the order of 180K. We subtract 180K - .data - .bss and what is left then becomes heap accessible storage. When you perform malloc() requests (explicitly or implicitly) this is drawn from heap. That includes creating FreeRTOS tasks whose stack (if I understand it correctly) are also drawn from heap.