ESP32 stops parsing JSON, heap RAM keeps getting smaller

[jemamena]

Hi,

I am trying to use the twitter api library or any other social media api to receive stats.

Everything seems ok at the start but at some point it stops receiving the json message. It happens with different requests, not only twitter (Facebook, etc.).

I have checked if there is a RAM problem and it seems like every time there is a request the heap RAM keeps getting smaller to the point that it doesn't parse anymore. I have also checked if the ESP has been disconnected but it keeps connected.

I have also tried to use StaticjsonBuffer and clearing the jsonbuffer but it doesn't change.

This is my code:

Code: Select all

#include <TwitterApi.h>

/*******************************************************************
 *  An Example of getting data back from the TwitterAPI            *
 *  https://github.com/witnessmenow/arduino-twitter-api            * 
 *                                                                 *
 *  Written by Brian Lough                                         *
 *******************************************************************/


#include <WiFi.h>
#include <WiFiClientSecure.h>

//------- Replace the following! ------
char ssid[] = "ssss";       // your network SSID (name)
char password[] = "kkkkk";  // your network key

// Normally we would use these to generate the bearer token but its not working yet :/
// Use steps on the readme to generate the Bearer Token

#define BEARER_TOKEN "zzzzz"

//Using curl to get bearer token
// curl -u "$CONSUMER_KEY:$CONSUMER_SECRET" \
//    --data 'grant_type=client_credentials' \
//    'https://api.twitter.com/oauth2/token'

WiFiClientSecure client;
TwitterApi api(client);

unsigned long api_mtbs = 5000; //mean time between api requests
unsigned long api_lasttime = 0;   //last time api request has been done
bool firstTime = true;

//Inputs

String screenName = "xxx";
bool haveBearerToken = false;

void setup() {

  Serial.begin(115200);

  // Set WiFi to station mode and disconnect from an AP if it was Previously
  // connected
  WiFi.mode(WIFI_STA);
  WiFi.disconnect();
  delay(100);

  // Attempt to connect to Wifi network:
  Serial.print("Connecting Wifi: ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(500);
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  IPAddress ip = WiFi.localIP();
  Serial.println(ip);

  Serial.printf("Internal Total heap %d, internal Free Heap %d\n",ESP.getHeapSize(),ESP.getFreeHeap());
  Serial.printf("SPIRam Total heap %d, SPIRam Free Heap %d\n",ESP.getPsramSize(),ESP.getFreePsram());


  api.setBearerToken(BEARER_TOKEN);
  haveBearerToken = true;
  getTwitterStats(screenName);

//   uint8_t* ps=(byte *)ps_malloc(900000);
}

void getTwitterStats(String name) {
  Serial.println("Getting twitter stats for " + name);
  String responseString = api.getUserStatistics(name);
    Serial.println(responseString);
    DynamicJsonBuffer jsonBuffer;
    JsonObject& response = jsonBuffer.parseObject(responseString);
    if (response.success()) {
      Serial.println("parsed Json");
      // Use Arduino Json to parse the data
      int followerCount = response["followers_count"];
      Serial.println(followerCount);
    } else {
      Serial.println("Failed to parse Json");
    }
    Serial.printf("Internal Total heap %d, internal Free Heap %d\n",ESP.getHeapSize(),ESP.getFreeHeap());
    Serial.printf("SPIRam Total heap %d, SPIRam Free Heap %d\n",ESP.getPsramSize(),ESP.getFreePsram());
    
}



void loop() {
  if(haveBearerToken){
    if ((millis() > api_lasttime + api_mtbs))  {
      getTwitterStats(screenName);
      api_lasttime = millis();
      if (WiFi.status() == WL_CONNECTED) {
        Serial.println("Still connected");
      } 
    }
  } 
}

This is what I get in the Serial Monitor:

Code: Select all

Connecting Wifi: sssss
...
WiFi connected
IP address: 
192.168.1.xxx
Internal Total heap 337244, internal Free Heap 282376
SPIRam Total heap 4194116, SPIRam Free Heap 4190924
Getting twitter stats for xxxx
{"......very long Json......."}
parsed Json
219
Internal Total heap 336328, internal Free Heap 234060
SPIRam Total heap 4194100, SPIRam Free Heap 4190788
ps=0x3f801288
Getting twitter stats for xxxx

Failed to parse Json
Internal Total heap 336900, internal Free Heap 247180
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx
{"......very long Json......."}
parsed Json
219
Internal Total heap 336148, internal Free Heap 199752
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

Failed to parse Json
Internal Total heap 336764, internal Free Heap 213076
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx
{"......very long Json......."}
parsed Json
219
Internal Total heap 336040, internal Free Heap 165672
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

..........


Internal Total heap 335664, internal Free Heap 63468
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

Failed to parse Json
Internal Total heap 336192, internal Free Heap 76104
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

Failed to parse Json
Internal Total heap 336196, internal Free Heap 76096
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

Failed to parse Json
Internal Total heap 336156, internal Free Heap 75676
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxxx

Failed to parse Json
Internal Total heap 336160, internal Free Heap 75668
SPIRam Total heap 4194084, SPIRam Free Heap 3290772
Still connected
Getting twitter stats for xxxx

........

According to an Arduino JSON assistant, the size of the JSON message is:

ESP32/ESP8266 1464+1994 = 3458 Bytes

I really don't know what else to do. The instagram library for Arduino works fine but it uses JsonStreamingParser.h. Would it be possible to use that library instead? Will it change the result?

It does seem like once the internal Free Heap RAM gets to around 7000, it doesn't work anymore.

I am also calling the function every 5 seconds. I have tried to increase the time but seems like it happens the same at some point. It also seems like the Free Heap RAM increases a bit too.

Thanks

[chegewara]

Try to free/delete one or both variables:

Code: Select all

    DynamicJsonBuffer jsonBuffer;
    JsonObject& response = jsonBuffer.parseObject(responseString);

[venquessa]

Try to free/delete one or both variables:

Code: Select all

    DynamicJsonBuffer jsonBuffer;
    JsonObject& response = jsonBuffer.parseObject(responseString);

So the first object is a local stack variable which cannot be freed/deleted. The second is reference, which really shouldn't be deleted. Using a reference suggests the memory for that object is handled elsewhere. For example in this case the response memory has no-doubt been allocated by the DynamicJsonBuffer. Deleting "response" may then lead to a NULL reference within the DynamicJsonBuffer and bad things may occur randomly.

For the OP. If you can't parse a single JSON string after boot, then basically you can't parse it this way period. If however you are finding your memory is expanding as you parse multiple messages one after the other, then you are not clearing up the memory from previous messages.

I would look at the documentation for this DynamicJsonBuffer closely to find out how to free the memory.

Alternatively, you can trust that it implements a destructor and put the code that uses the DynamicJsonBuffer into a method/function. It will be deleted when the method exits. If it implments a destructor correctly it will free/delete all the memory it used and the DynamicJsonBuffer memory will be freed and the stack pointer restored as if it never existed. The downside is allocating all that memory every time you create a new buffer.

[jshep321]

Hi,
I had similar problems. There's a nice FAQ explaining your problem here:
https://arduinojson.org/v6/faq/i-found- ... e-library/

clearing the document after parsing will likely stem the memory leak!