ESP32 hangs after a few days

[Phillip]

I have an Adafruit HUZZAH32 ESP32 MCU that gathers the following data from a domestic water tank: water level, temperature, humidity and LiPo battery voltage. This data is printed to a 16x2 LCD. It also sends daily texts three times a day and if the water level drops below eight (8) feet, it sends a low water level text.

Startup is always normal - no issues, but after a few days of operation the unit quits completely. I have never been in the pump house when this happens so I do no know exactly what happens. The only way that I become aware of this malfunction is when the daily reports are not sent at their preset times.

Startup IDE info is as follows:

Code: Select all

Sketch uses 1058985 bytes (80%) of program storage space. Maximum is 1310720 bytes.
Global variables use 47816 bytes (14%) of dynamic memory, leaving 279864 bytes for local variables. Maximum is 327680 bytes.

As I understand it, the above data is FLASH memory usage, not PSRAM info.

I think that my Heap and Stack memory data are overlapping and corrupting the data thus shutting down the MCU.

I currently have a sketch that reports numerical data from the following calls:
- ESP.getHeapSize()
- ESP.getFreeHeap()
- ESP.getMinFreeHeap()

I have also tried getting data from the following calls:
- ESP.getPsramSize()
- ESP.getFreePsram()
- ESP.getMinFreePsram()
- ESP.getMaxAllocPsram()

The latter Psram calls all return a value of '0'. These are the memory calls that I would like to get data back from.
Do anyone know why these calls return zeros?

Sample START data is as follows:

Code: Select all

14:19:25.561 -> 	 ... HEAP SIZE is 332352
14:19:25.561 -> 	 ... Free heap memory is 200792
14:19:25.596 -> 	 ... MIN FREE HEAP is 195704
14:19:25.596 -> 	 ... MAX ALLOC HEAP is 110580
14:19:25.596 -> 	 ... Free heap SIZE is 153628
14:19:25.596 -> 
14:19:25.596 -> 	 ... SPRAM SIZE is 0
14:19:25.596 -> 	 ... FREE SPRAM SIZE is 0
14:19:25.596 -> 	 ... MIN FREE SPRAM is 0
14:19:25.596 -> 	 ... MAX ALLOC SPRAM is 0

Sample ENDING data (after three(3) hours of run time) is as follows:

Code: Select all

17:06:59.642 -> 	 ... HEAP SIZE is 332352
17:06:59.642 -> 	 ... Free heap memory is 181664
17:06:59.642 -> 	 ... MIN FREE HEAP is 127388
17:06:59.642 -> 	 ... MAX ALLOC HEAP is 81908
17:06:59.642 -> 	 ... Free heap SIZE is 134236
17:06:59.642 -> 
17:06:59.642 -> 	 ... SPRAM SIZE is 0
17:06:59.642 -> 	 ... FREE SPRAM SIZE is 0 
17:06:59.642 -> 	 ... MIN FREE SPRAM is 0
17:06:59.642 -> 	 ... MAX ALLOC SPRAM is 0

If someone can show me how to get SPRAM data I would be most grateful.

Thank you.

[MicroController]

"PSRAM", i.e. pseudo static RAM, a.k.a. SPI-RAM, a.k.a. external RAM, refers to RAM in an extra chip, which the ESP can access via one of its SPI buses.
The ESP32-WROOM module on the HUZZAH32 doesn't have this external RAM chip, so that RAM's size is returned as 0.
The difference between the WROOM and the WROVER modules is that the latter do have PSRAM installed.

Your application may or may not have a memory leak which could cause it to stop working after some time when it finally runs out of heap memory.
But that's only one potential reason for the issue you're seeing.

The quickest and dirtiest way around the problem would probably be to just have the ESP restart itself e.g. every 24 hours.

[Phillip]

@MicroController

Thank you VERY MUCH for your in-depth response to my query. Today's research has made me wonder if I am using the right MCU for this project (which is embarrassing me because I cannot solve this hanging issue).

That said your final suggestion may resolve this problem altogether. I will certainly add that code and hope for the best.

Thank You!, once again for your help. I am a lot wiser now.

[Phillip]

Update!

I will be allowed to update my code to the pump house enclosure HUZZAH32 MCU tomorrow afternoon sometime. I have added a restart function that will send me a text when the ESP32 is restarted. The restart time is 2400 so will see the text at 0800 or so.

I will report back after it has been on-line for a week or so. This MCU has always hung after a few days of operation so if it does indeed stay live for a week then I will believe that @MicroController's restart suggestion will have solved this problem.

Later.

[username]

Why not just get another and leave it hooked up at home to the PC so when it restarts you will see what is the issue.

[Phillip]

@MicroController

Update
After the most recent upload, the HUZZAH32 has not hung. Here are few graph grabs that illustrate the new behavior.

Screenshot 2024-09-18 at 10.17.52.png (37.05 KiB) Viewed 1798 times

This shows the approximate start of the previous upload. The squiggly lines are feed data.

Screenshot 2024-09-18 at 10.18.08.png (35.68 KiB) Viewed 1798 times

This shows the data just prior to the MCU hanging. The straight line to upper right is crash behavior. From this it can be assumed that the board hung within less than 12 hours of run time.

Screenshot 2024-09-18 at 10.18.49.png (33.14 KiB) Viewed 1798 times

The above grab is the start of the most recent upload with the restart code installed. Actual upload time was 1515.

Will add the most recent grab below.

[Phillip]

And here is the most recent data grab:

Screenshot 2024-09-18 at 13.13.54.png (29.86 KiB) Viewed 1794 times

From this it can be seen that the HUZZAH32 has run continuously for nearly 48 hours without hanging up.

Bottom line: From this current behavior, your restart suggestion appears to have been the solution to this problem.

I cannot thank you enough for your help.

[MicroController]

Thanks for reporting back. It's always good to hear if problems could actually be solved

For future stability of your application I encourage you to go on and try and fix the actual cause of the issue (memory leak?) though.

[Phillip]

You are correct about actually determing what the root cause of the hanging was. This may be a challenge because I do not have full access to the tank well house. Monitoring memory may be the only option.

I sincerely appreciate your help.

[MicroController]

There are many way to approach the issue.
Personally, I would first comb through my code looking specifically for memory leaks, i.e. check if every memory allocation I make (malloc, new,...) has a corresponding deallocation (free, delete,...) in every possible branch of execution. If one hasn't I make sure that's actually intended, i.e. the object/memory allocated is supposed to live 'forever'.

If this isn't fruitful, the next step would be running the application 'on the bench' while monitoring the heap space. These test runs can possibly be speed-run by cutting out any idle times the 'real' application would have.

Then there's also the IDF's heap debugging facility to help in locating leaks.

This is of course only productive if a memory leak is likely to be the issue. Other bugs in the code should also be checked for an ruled out.