Strange networking issues with WROOM-32U vs. 32D

[Zwirbeljupp]

Hi all,

I need your help with a bee hive monitoring device I am working on.

The design is based on a custom PCB which contains an ESP32-WROOM module and a couple of sensors. The plan is to read these sensors, upload the data via wifi to an influxDB database and go into deepsleep for 60 seconds.
The development prototype (for my desk) is equipped with an ESP32-WROOM-32D (integrated PCB antenna), the actual bee hive unit has a WROOM-32U with an external 2.4 GHz antenna connected. Power supply is based on LiFePo4 cells directly powering the ESP32s.

Problem description:
I noticed that the “production unit” with the external antenna (WROOM-32U) often runs into communication issues, where it is not able to deliver the payload to the database. In these cases it is able to connect to wifi and retrieve time stamp via NTP, but the http connection to the influxDB server fails with http result codes -1 (in 90% of cases) or -11. There are periods of several hours, where the unit delivers data without any issues, then there are periods, where it hardly can deliver any data for the next couple of hours.
If I keep my development unit (WROOM-32D) running at the same time, connected to the same router, running the same code, it does not show any connectivity issues and flawlessly delivers the data to the database.
RSSI reading of both units is inconspicuous, usually around -47.
I don’t think that I have a general problem with my PCBs, since one unit is working just fine and even the “faulty” unit sometimes operates as expected for a couple of hours.

Here I plotted the number of database entries per 5 minutes (expecting 4-5) over a period of 22 hours. As you can see, the "development unit" with the 32D reliably delivers 4-5 entries, the "production unit" with the 32U (upper diagram) fails completely during several periods:

data integrity monitoring 2020-01-26 081006.jpg (79.54 KiB) Viewed 10856 times

What I tried so far:

• tried different external antennas from different vendors (all specifically designed for 2.4 GHz wifi)
• tried to upload data to different suppliers (tested Thingspeak, now using Corlysis) with same results
• replaced the WROOM-32U on the PCB once already since I was suspecting a faulty ESP32, but also with a new WROOM-32U the issues are identical
• completely stripped down my code to only the networking part: connect to wifi, get time via NTP, send test data to influxDB, go into deepsleep (see code for reference below). This code still shows the same behavior
• verified, that there are no settings in my router (AVM fritzbox 7560) which might block network communication for a specific MAC / IP address
• completely erased flash (esptool.py erase_flash) before uploading sketch

My questions:

• Are there any configuration steps I need to perform for the WROOM-32U which I don’t know? Maybe adjust RX/TX power to match the external antenna?
• Do you have any explanation, why the networking performance of the 32U is so bad compared to the 32D?
• Do you have any ideas, how to further debug this issue? I have little experience with this networking stuff and don’t really know where to start…

Please let me know what else you need to help me with my problem (e.g. datasheets, schematic, code,...).

Thank you & best regards,
Daniel

my code:

Code: Select all

#include <WiFi.h>
#include "SPI.h"
#include "time.h"
#include <HTTPClient.h>

#ifndef ONBOARD_LED
  #define ONBOARD_LED 2
#endif

#define SECRET_SSID "---"    // replace MySSID with your WiFi network name
#define SECRET_PASS "---"  // replace MyPassword with your WiFi password

#define MAC_DEVUNIT "30:AE:A4:--:--:--"    // Wifi MAC of development unit (integrated PCB antenna)
#define MAC_PRODUNIT "30:AE:A4:--:--:--"    // Wifi MAC of production unit (external antenna)

char ssid[] = SECRET_SSID;   // your network SSID (name) 
char pass[] = SECRET_PASS;   // your network password
int keyIndex = 0;            // your network key Index number (needed only for WEP)
WiFiClient  client;

// Corlysis Setting - click to the database to get those info
const char* dbname = "NewDB";
const char* dbpassword = "--";

const char* dbname_test = "TestDB";
const char* dbpassword_test = "--";

const unsigned long delayTimeMs = 10000;

HTTPClient http;
char payloadStr[255];

const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

unsigned long delayTime;
unsigned long last_timestamp;
unsigned long modem_start;

#define uS_TO_S_FACTOR 1000000ULL  /* Conversion factor for micro seconds to seconds */
#define TIME_TO_SLEEP  60        /* Time ESP32 will go to sleep (in seconds) */

#define WIFI_TIMEOUT    100    // 100 retries for Wifi connection equals 20 seconds

RTC_DATA_ATTR int bootCount = 0;

String mac = "";
int8_t rssi = 127;
uint16_t wifitimeout;


/*
Method to print the reason by which ESP32
has been awaken from sleep
*/
void print_wakeup_reason(){
  esp_sleep_wakeup_cause_t wakeup_reason;

  wakeup_reason = esp_sleep_get_wakeup_cause();

  switch(wakeup_reason)
  {
    case ESP_SLEEP_WAKEUP_EXT0 : Serial.println("Wakeup caused by external signal using RTC_IO"); break;
    case ESP_SLEEP_WAKEUP_EXT1 : Serial.println("Wakeup caused by external signal using RTC_CNTL"); break;
    case ESP_SLEEP_WAKEUP_TIMER : Serial.println("Wakeup caused by timer"); break;
    case ESP_SLEEP_WAKEUP_TOUCHPAD : Serial.println("Wakeup caused by touchpad"); break;
    case ESP_SLEEP_WAKEUP_ULP : Serial.println("Wakeup caused by ULP program"); break;
    default : Serial.printf("Wakeup was not caused by deep sleep: %d\n",wakeup_reason); break;
  }
}


void printLocalTime()
{
  struct tm timeinfo;
  if(!getLocalTime(&timeinfo)){
    Serial.println("Failed to obtain time");
    return;
  }
  Serial.println(&timeinfo, "%A, %B %d %Y %H:%M:%S");
}


void setup() 
{
  // initialize digital LED_BUILTIN on pin 13 as an output.
  pinMode(ONBOARD_LED, OUTPUT);
  digitalWrite(ONBOARD_LED, LOW);   // switch on LED
  
  Serial.begin(115200);//Start Serial comms

  //Increment boot number and print it every reboot
  ++bootCount;
  Serial.println("Boot number: " + String(bootCount));

  //Print the wakeup reason for ESP32
  print_wakeup_reason();
  
  WIFIconnect();
  NTPUpdate();

  DataUpload();
  WIFIdisconnect();
  ActivateDeepSleep();

}


void loop() 
{
  ; 
}


  // +--------------------------------------------------------------------+
  // |    Switch on WiFi                                                  |
  // +--------------------------------------------------------------------+
void WIFIconnect()
{
  modem_start = millis();

  WiFi.persistent(false);
  WiFi.disconnect();
  WiFi.mode(WIFI_OFF);  
  WiFi.mode(WIFI_STA);   

  // Connect or reconnect to WiFi
  wifitimeout = 0;
  Serial.print("Attempting to connect to SSID: ");
  Serial.println(SECRET_SSID);
  WiFi.begin(ssid, pass);  // Connect to WPA/WPA2 network. Change this line if using open or WEP network
   
  while((WiFi.status() != WL_CONNECTED) && (wifitimeout < WIFI_TIMEOUT) )
  {
   
    Serial.print(".");
    delay(200);    
    wifitimeout++;
  } 

  if(wifitimeout == WIFI_TIMEOUT)
  {
    Serial.println("\nWIFI Connection timed out!");
  }
  else
  {
    Serial.println("\nConnected.");  

    mac = WiFi.macAddress();
    Serial.println(mac);
  
    rssi = WiFi.RSSI();
  }
}

// +--------------------------------------------------------------------+
// |    Get current time from NTP Server                                |
// +--------------------------------------------------------------------+
void NTPUpdate()
{
  char sprintfBuffer[100];
  
  if(WiFi.status() == WL_CONNECTED)
  {
    //init and get the time
    configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  
    struct tm timeinfo;
    if(!getLocalTime(&timeinfo))
    {
      Serial.println("Failed to obtain time");
    }
    else
    {
      strftime(sprintfBuffer, 24, "%d-%m-%Y - %H:%M:%S, ", &timeinfo);  
      Serial.println(sprintfBuffer);

    }
  }
}     


// +--------------------------------------------------------------------+
// |    Send Data to Corlysis / InfluxDB                                |
// +--------------------------------------------------------------------+
void DataUpload()
{
    static long counter = 0;

    char pstr[400];
    char corlysisUrl[200];    
 
    pstr[0] = '\0';
  
  if(WiFi.status() == WL_CONNECTED)
  {
      // create payload string (with fixed test values)
      sprintf(payloadStr, "ALS=%u,T1=%.2f,Hum1=%.2f,SOC=%u,Ibat=%d,Ubat=%u,weight=%.2f,Psolar=%.2f", 1111, 20.5f, 30, 10, 0, 3333, 0.0f, 0.0f);
      Serial.println(payloadStr);


      // identify test device based on Wifi MAC and select influxDB database
      if (strcmp(mac.c_str(), MAC_DEVUNIT)  == 0)
      {
        Serial.println("Development Unit detected.");
        sprintf(corlysisUrl, "http://corlysis.com:8087/write?db=%s&u=token&p=%s", dbname_test, dbpassword_test);
        strcat(pstr, "test_data ");      
      }
      else if (strcmp(mac.c_str(), MAC_PRODUNIT)  == 0)
      {
        Serial.println("Production Unit detected.");
        sprintf(corlysisUrl, "http://corlysis.com:8087/write?db=%s&u=token&p=%s", dbname, dbpassword);
        strcat(pstr, "beehive ");
      }
      else
      {
        Serial.println("Unknown MAC.");
        return;
      }

      // assemble payload string
      strcat(pstr, payloadStr);
      Serial.println(pstr);

      // initiate connection to influxDB
      http.begin(corlysisUrl);
      
      http.addHeader("Content-Type", "application/x-www-form-urlencoded");  
      int httpCode = http.POST(pstr);
      Serial.print("http result:");
      Serial.println(httpCode);

  
      if(httpCode != 204)
      {
        // print error message to serial if available
        http.writeToStream(&Serial);

        Serial.print("Corlysis error. Data not sent. httpCode = ");
        Serial.println(httpCode);        
      }
      else
      {
          counter = 0;
          Serial.println("Data successfully sent.");        
      }
      
      http.end();

  }
  else
  {
    // Wifi connection failed...
  }
}

// +--------------------------------------------------------------------+
// |    Switch off WiFi                                                 |
// +--------------------------------------------------------------------+
void WIFIdisconnect()
{
  if(WiFi.status() == WL_CONNECTED)
  {
    WiFi.disconnect();
    WiFi.mode(WIFI_OFF);  
    Serial.println("Modem activation time: " + String(millis() - modem_start) + " ms");
  }
}


// +--------------------------------------------------------------------+
// |    activate ESP32 deep sleep                                       |
// +--------------------------------------------------------------------+
 void ActivateDeepSleep()
 {
  /*
  First we configure the wake up source
  We set our ESP32 to wake up every 5 seconds
  */
  esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
  Serial.println("Setup ESP32 to sleep for every " + String(TIME_TO_SLEEP) +
  " Seconds");

  Serial.println("Going to sleep now");
  Serial.flush(); 

  digitalWrite(ONBOARD_LED, HIGH);
  
  esp_deep_sleep_start();
  Serial.println("This will never be printed");
 }
 

[tachin]

Very intesting thread, Did you solve this? Meanwhile some expert may reply here, did you replace the U module to a D module in the production onit? (or put a U module into Dev unit) to discard totally the pcb, can you share the full part number of each number? maybe different ram / flash sizes?

[Zwirbeljupp]

Hi tachin,
unfortunately I was not able to solve this so far. Kept both systems running in parallel for some more days and the symptoms are still there:

data integrity monitoring 2020-02-09.jpg (84.37 KiB) Viewed 10656 times

By the way, I noticed that the phases of bad connection mostly start around 0:00 and then last for a couple of hours. Was not yet able to correlate this timing to any specific activities in my home network etc.

I did not change the U module to a D yet, since in the end application the D will most likely not have enough antenna gain to do the job. However, seems like one of the best and last options, to rule out issues with power supply etc. on this specific PCB.

How should the full numbers you requested look like? These are the FCC IDs lasered onto the metal can, but I assume you don't mean these:
Production unit: 2AC7Z-ESP32WROOM32U
Dev. unit: 2AC7Z-ESP32WROOM

So any ideas how to further debug this problem?
I'm still wondering why the system never has issues to access the SNTP server but the simple http.POST operations fail with results -1 and -11?

Thanks & best regards,
Daniel

[ESP_Angus]

Hi Daniel,

Thanks for the detailed description of the problem.

One thing that springs to mind is that the power, battery, external connections in the "production" box appear different, is it possible there's something like noisy power or a sensor that's causing inteference? Have you tried mounting the working 32D board into the box with 100% identical other connections and verified it's still stable?

I'm still wondering why the system never has issues to access the SNTP server but the simple http.POST operations fail with results -1 and -11?

Can you provide a snippet of the code showing the exact function that returns these results, please?

Also, is there any chance you can capture a UART console log from the failing device? That's probably the quickest way to get more details about what's happening.

[Zwirbeljupp]

Hi ESP_Angus,

thanks for your feedback!
Following your recommendation I swapped only the logic boards of both units (the ESP32 with PCB antenna now sits in the sealed enclosure). Will keep it running for as long as it is required to gather some meaningful data and will keep you updated.

Please see below the marked line in my code from the same sketch as above which returns the -1 or -11 http result code:

Code: Select all

      // initiate connection to influxDB
      http.begin(corlysisUrl);
      http.addHeader("Content-Type", "application/x-www-form-urlencoded");  
      int httpCode = http.POST(pstr);   // <--- this function returns the -1 or -11 result codes
      Serial.print("http result:");
      Serial.println(httpCode);

I am capturing the UART debug output of the device all the time, but it is not giving me meaningful debugging information (would probably need to modify the default Arduino libraries to add additional debug outputs).

Thanks,
Daniel

[Zwirbeljupp]

Hello all,

I finally figured out how to enable additional debugging output for httpClient and WifiGeneric. I was also able to identify the meaning of the http result codes -1 and -11 which were returned by the POST function:

#define HTTPC_ERROR_CONNECTION_REFUSED (-1)
#define HTTPC_ERROR_READ_TIMEOUT (-11)

The console output now reveals more details about what goes wrong:

Code: [Select all] [Expand/Collapse]

1.  [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 0 - WIFI_READY
2. Attempting to connect to SSID: xxxxx
3. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 2 - STA_START
4. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 4 - STA_CONNECTED
5. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 7 - STA_GOT_IP
6. [D][WiFiGeneric.cpp:376] _eventCallback(): STA IP: 192.168.178.77, MASK: 255.255.255.0, GW: 192.168.178.1
7.  
8. Connected.
9. RSSI: -44
10. 07-03-2020 - 17:12:16,
11. Development Unit detected.
12. [D][HTTPClient.cpp:290] beginInternal(): host: corlysis.com port: 8087 url: /write?db=TestDB&u=token&p=f4762f03bf2ade70b91bxxxxxxxx
13. [E][WiFiGeneric.cpp:678] hostByName(): DNS Failed for corlysis.com
14. HTTPClient::connect failed connect to corlysis.com:8087
15. [D][HTTPClient.cpp:1062] connect(): failed connect to corlysis.com:8087
16. [W][HTTPClient.cpp:1379] returnError(): error(-1): connection refused

GeSHi © Codebox Plus Extension

And this is the second failure mode I see:

Code: [Select all] [Expand/Collapse]

1.  [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 0 - WIFI_READY
2. Attempting to connect to SSID: xxxxxxxxxxxx
3. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 2 - STA_START
4. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 4 - STA_CONNECTED
5. [D][WiFiGeneric.cpp:332] _eventCallback(): Event: 7 - STA_GOT_IP
6. [D][WiFiGeneric.cpp:376] _eventCallback(): STA IP: 192.168.178.77, MASK: 255.255.255.0, GW: 192.168.178.1
7.  
8. Connected.
9. RSSI: -43
10. 07-03-2020 - 17:04:41,
11. Development Unit detected.
12. [D][HTTPClient.cpp:290] beginInternal(): host: corlysis.com port: 8087 url: /write?db=TestDB&u=token&p=f4762f03bf2ade70b91bxxxxxx
13. [D][HTTPClient.cpp:1069] connect():  connected to corlysis.com:8087
14. HTTPClient::connect connected to corlysis.com:8087
15. HTTPClient::handleHeaderResponse HTTPC_ERROR_READ_TIMEOUT
16. [W][HTTPClient.cpp:1379] returnError(): error(-11): read Timeout
17. [D][HTTPClient.cpp:1381] returnError(): tcp stop

GeSHi © Codebox Plus Extension

So the first problem seems to be related to the DNS.
Any ideas how to continue debugging this issue from here? Is there a chance that my router is leading to these weird connection issues?

Thanks,
Daniel

EDIT:
at the same time I have a second unit running the same code but does not show the DNS or connection errors.

[Zwirbeljupp]

Hi all,

my problem is (kind of) solved, so I want to give a quick update.

The limited debugging options in the Arduino IDE started to be quite frustrating, so I decided to port the project to the ESP-IDF 4.0. There also was a little hope that the issues with the device somehow were related to the Arduino implementation, but that of course was not the case
So after spending some weeks to port the entire code I was facing exactly the same issue: one of the prototypes had problems to connect to the influxDB server, where the sensor data should be uploaded. As with the Arduino code, wifi connection and time update via NTP mostly worked OK. Now with the ESP-IDF code I was able to dig a little bit deeper on the root cause of the connection issues. However, it showed that the http client core functions were failing at somehow random steps with weird error codes, but only on the unit with the external antenna. The prototype accomodating the ESP32 with PCB antenna operated as expected.

Long story short: I bought a different 2.4 GHz antenna with a 1.5m cable connection and positioned this antenna about 1m away from the circuit board. Issues gone.

I still have not understood, what is causing the connection problem. Something on the circuit board seems to interfer with the 2.4GHz signal, but so far I could not figure out the source. There is no switching regulator on the board, power directly comes out of a LiFePo4 battery pack. Supply rail checked with an oscilloscope, looks perfectly clean. All sensors on the board are very low power and mostly in sleep mode when the wifi connection is established.
I also don't understand, why the ESP32 with the PCB antenna is NOT affected by the interference (at least on one of the prototypes).
Any ideas how to further investigate the problem are welcome.

Best regards,
Daniel