Newbie Question: Reliable Message Based Communication to ESP32?

[neltnerb]

Apologies for the newbie questions, I'm having a hard time using search terms for the forum since I think the stuff I'm asking about I don't know the right words for.

I build custom scientific equipment, and I'd like to shift from USB based communication to a flexible scheme that can allow wifi or ethernet control. The usual way I set this up is to have the computer emulate a serial port and I send a command to my microcontroller that it then interprets, does something, and then replies based on whether it succeeded in executing the command.

I am not quite sure how to do something along these lines over TCP/IP.

I tried using the Arduino toolchain for ESP32 (which maybe is buggy?) and set it up as a server on port 22222. Then I connect to it with ncat or LabView on that port and IP address, send a message, wait for a reply, and then (notionally) close the port. My sketch is a derivative of the SimpleWifiServer server that I excised all the HTML from.

This is my main loop():

Code: Select all

std::vector<char> commandstring;

void loop(){
 WiFiClient client = server.available();   // listen for incoming clients

  if (client) {                             // if you get a client,
    Serial.println("New Client.");           // print a message out the serial port
    String currentLine = "";                // make a String to hold incoming data from the client
    while (client.connected()) {            // loop while the client's connected
      if (client.available()) {             // if there's bytes to read from the client,
        char c = client.read();             // read a byte, then
        if (c == 13) {   // if the byte is a newline or carriage return character...
          commandstring.push_back(13);
          commandstring.push_back(10);
          commandstring.push_back('\0');
          Serial.println(commandstring.data());
          client.println(commandstring.data());
          commandstring.clear();
        }
        else if ((c >= 0x20) && (c <= 0x7E)) { // Otherwise, check that the character is standard and add the character to the message.
          commandstring.push_back(c);
        }
      }
    }
    client.stop();
    Serial.println("Client Disconnected.");
    // close the connection:
  }
}

This very nearly works, just not reliably. What I see is that I send it data and it reports the new client connection. Then I see the command I sent (with the carriage return) echoed properly on both the serial port and the TCP/IP connection. However, if I try to send it a second command it sometimes works, and by the third time it completely fails and I have to restart the ESP32 to make it communicate again. At that point it is no longer echoing the received command to the serial port either.

I think I'm probably doing this in a very naive and wrong way, but unfortunately my last networking course was in 2001... websockets and such are new terminology to me and I don't remember how ports work under the hood. The ESP32 is *not* reporting when my client disconnects, which is probably the main issue -- ncat doesn't exit on its own after sending a file or piping an echoed command, and LabView explicitly closes the connection and the ESP32 doesn't seem to register the closed connection.

Can anyone help me either make what I'm trying to do work reliably, or help me identify a better approach? I'd like to be able to give this to a client and trust that they won't run into issues. I'm not sure if I'm doing it wrong and need to add a handler to identify when the connection is closed (the example code didn't, and seemed to identify when HTTP connections closed correctly anyway), if the Arduino server implementation isn't properly handling closing connections from the client side, or if I should just use a different approach entirely to do this kind of thing.

I am also unsure what best practices are to implement a user configuring the wifi connection. I assume that I have to have them connect over USB and configure it manually, but in this day and age I am curious if there is a better way. I imagine that the initialization problem is still tricky since it can't be communicated with over wifi until it knows the ssid and password?

[neltnerb]

Update: I managed to get some fairly sophisticated stuff working reasonably well, but I think there may frankly be some bugs in the library. I'd love to know if they are not bugs and that I'm doing it wrong!

Okay, so first, I wanted to make the connection optional, but WiFi.status() doesn't seem to return what I think it should. I tried this:

Code: Select all

  while ((WiFi.status() != WL_CONNECTED) and (WiFi.status() != WL_CONNECT_FAILED)) {
    delay(500);
    Serial.print(".");
  }
  if (WiFi.status() == WL_CONNECTED) {
    Serial.println("");
    Serial.println("WiFi connected.");
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());
    server.begin();
  }
  else {
    Serial.println("");
    Serial.println("WiFi Connection Failed.");
    Serial.print("SSID: ");
    Serial.println(ssid);
    Serial.print("Password: ");
    Serial.println(password);
  }

The idea here is that if the connection attempts reach their failure limit (which the documentation doesn't define) it would return WL_CONNECT_FAILED which would tell me that I can stop waiting for it to connect. However, this doesn't actually ever happen. I instead relied on a timeout, but I'd much rather do something cleaner.

Next, in my main loop I wanted to check for a valid client and then try to connect again if it isn't currently trying so that you can configure it via the serial port and then connect without rebooting. I implemented it like this:

Code: Select all

WiFiClient client;

void loop() {
  checkSerial(Serial);
  if (client) checkWiFi(client);
  // If the wifi is connected, connect the client to an open socket.
  else if (WiFi.status() == WL_CONNECTED) client = server.available();
  // If it's not connected nor trying to connect, try to connect. 
  else if (WiFi.status() != WL_IDLE_STATUS) WiFi.begin(ssid.c_str(), password.c_str());
}

However, this resulted in it complaining about trying to run too many instances of WiFi or something, the error message wasn't entirely clear. But I'm pretty sure it's resulting from WiFi.begin() being called repeatedly. I'm looking in the documentation and I see nothing like WiFi.reconnect(ssid, password), is there a correct way to try to initialize a reconnect sequence?

I'm also finding that WiFiClient doesn't seem to detect when the remote system disconnects and then presumably since the client thinks it's still connected it ignores future connection attempts. I've tested that with LabView running the TCP/IP Close function as well as with ncat hitting Ctrl-C to exit the program. I think this might be a known bug, but I can't quite tell how to fix or workaround it.

This is the possible bug I'm referencing:
https://github.com/espressif/arduino-esp32/issues/377
but it's over a year old and the thread implied it was fixed so I'm not sure if I'm just doing it wrong or if it's still a bug.

The checkWiFi(client) function is a basic parser.

Code: Select all

std::vector<char> wifistring;

void checkWiFi(WiFiClient& interface) {
  if (interface.connected()) {
    while (interface.available() > 0) {
      char incomingByte = interface.read();
      
      // If the byte is a carriage return or newline (since I can't guarantee which if either will come
      // first), then send the command previously read to evaluateCommand() and clear the commandstring
      // buffer. This has the convenient effect of rendering irrelevant whether LabView or other such
      // GUI sends something reasonable for a termination character, as long as it sends *something*.
      
      if ((incomingByte == 0x0D) || (incomingByte == 0x0A)) {
        // This tests that there's a string to return in the buffer. If not, ignore. This is both
        // to avoid testing when it's an empty command, and also to deal with sequential CR/NL that
        // can happen with some GUIs sending serial data.
        
        if (wifistring.size() != 0) {
          // Append the termination character to the command string.
          wifistring.push_back('\0');
    
          // Evaluate the data.
          InterfaceClass port = InterfaceClass(&interface);
          evaluateCommand(wifistring.data(), port);
        }
        wifistring.clear();
      }

      // If the byte is not a carriage return, and is a normal ASCII character, put it onto the commandstring.
      else if ((incomingByte >= 0x20) && (incomingByte <=0x7E)) {
        wifistring.push_back(incomingByte);
      }
    }
  }
}

Don't worry about the InterfaceClass part, that was a class I built to contain either a HardWareSerial or WiFiClient pointer (with an enum to tell it which to use) so that evaluateCommand knows which interface the command was received on and I could implement a reply() function which sends responses on the correct interface even though there is only one evaluateCommand() function. That part seems to work like a charm, though there's probably a cleverer way to do it.

I never in here explicitly close the WiFiClient, but the wifiserver example didn't seem to either and it did detect when it had finished sending a HTML page, which is extra weird to me. This is the part that makes me think I may be simply using it wrong. It's never worked, even in simple toy code, I just have to restart the ESP32 module to reconnect over wifi after I close the first (sometimes second) connection.

What's the actual data flow here? I was figuring that what would happen is that the client when checked with if(client) would return false if the socket had been closed, thus triggering the first else if that would do client=server.available() each time through the main loop until client returns true again. But I'm not sure what makes client return false; should I be manually checking server.available() or something each time through the loop to modify the client?

Perhaps worth mentioning that I am absolutely, positively okay with shifting to another library system. I'm just getting started, so no time like now to port over stuff from a buggy library to a better one if it will work better. I will use this template I use in basically everything I program, so I'm happy to spend an unreasonable amount of time getting it perfect.