TCP Receive Buffering with WiFiClient

[maciekish]

Hi, I need to receive about 100 bytes every 33ms on an ESP32. I have a Python server that sends this out and the ESP32 needs to react to each message immediately. However, I'm seeing these messages get buffered by the ESP. There is nothing else running, receving the bytes in loop(). I'm using the standard wiFiClient.h and tried setting TCP_NODELAY on both ends, however it only seems to affect transmitting on the Python side. What can i do to get the data immediately as its received by the ESP32? I cant just split the received data, it's already too late as im using it to for example control LEDs depending on whats going on in the game.

I have also tried using client.read() without relying on client.available(), which results in the same issue.

Code: Select all

void ESP32WiFiSlave::loop() {
		if (!client.connected()) {
			int n = MDNS.queryService("_dcs-bios", "_tcp");

			if (n > 0) {
				master_ip = MDNS.IP(0);
				master_port = MDNS.port(0);
				client.setNoDelay(true);

				while (!client.connected()) {
					Serial.println("Connecting to " + master_ip.toString() + ":" + master_port);
					client.connect(master_ip, master_port, 1000);
				}

				Serial.println("Socket is connected");
				
				send("register");
				
				lastReceivedTime = millis();
				lastKeepAliveTime = millis();
				setStatus(CONNECTED);
			} else {
				Serial.println("No services found");
				delay(3000);
			}
		} else {
			int packetSize = client.available();

			if (packetSize > 0) {
				Serial.println("Received " + String(packetSize));
				uint8_t incomingPacket[packetSize];
				int len = client.read(incomingPacket, packetSize);
				lastReceivedTime = millis();

				// Print the received data to serial
				for (int i = 0; i < len; i++) {
					Serial.print((char)incomingPacket[i]);
				}
				Serial.println();
			} else {
				// Check for timeout
				if (millis() - lastReceivedTime > timeoutDuration) {
					client.stop();
					setStatus(ASSOCIATED);
					Serial.println("Connection timed out");
				}
			}

			// Check if it's time to send a keep-alive message
			if (client.connected() && (millis() - lastKeepAliveTime >= keepAliveTimeout)) {
				send("check-in");

				lastKeepAliveTime = millis();
			}
		}
	}

The Python is properly sending small packets every 33ms. I have also confirmed this with Wireshark.

Code: Select all

Sent 111 {"type": "message", "data": "VVVVVQYABAAyACAgKgQEACA5EABMBAIAI/aiEAQAdx2QHbAQAgC6E8YQAgAokmYTBAAwgDMw/v8CAMIG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 103 {"type": "message", "data": "VVVVVQoABAAgICAgLgQEAAkARTRMBAIAJ/aiEAQAgB2XHcYQAgA5kmoTBAAuMDAw/v8CAMMG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 103 {"type": "message", "data": "VVVVVQ4ABAAgICAgMgQEABOQQgBMBAIAKvaiEAQAhx2fHcYQAgBJkm4TBAAgAAAA/v8CAMQG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 111 {"type": "message", "data": "VVVVVRIABAAgICAgNgQEAPnqMDlMBAIALvaiEAQAjR2kHcYQAgBWktIQAgDCqXITBAAAAAAA/v8CAMUG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 127 {"type": "message", "data": "VVVVVQAAAgBBLRYAAgAgIDoEBAA6MjA6TAQCADH2ohAEAJMdqR2uEAIA5RPGEAIAYpLSEAIAwal2EwQAAAAAAP7/AgDGBg=="} to slave-1 at address ('10.0.0.60', 55332)
Sent 127 {"type": "message", "data": "VVVVVQIABAAxMENfPgQEADMzIDFMBAIANfaiEAQAmB2uHbAQAgC5E8YQAgBtkgIRAgAyMZgSAgBbIHoTBAD//wAA/v8CAMcG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 103 {"type": "message", "data": "VVVVVQYABAAyACAgQgQEADk0MSBMBAIAOPaiEAQAnB2xHcYQAgB1kn4TBAAAADAw/v8CAMgG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 103 {"type": "message", "data": "VVVVVQoABAAgICAgRgQIAAAAQIMCADv2ohAEAKAdtR3GEAIAfZKCEwQAMTMzLv7/AgDJBg=="} to slave-1 at address ('10.0.0.60', 55332)
Sent 111 {"type": "message", "data": "VVVVVQ4ABAAgICAgSgQEAAIAQPaiEAQApR2yHa4QAgDmE8YQAgCGkkYTAgBgUoYTBAAwMDAA/v8CAMoG"} to slave-1 at address ('10.0.0.60', 55332)
Sent 107 {"type": "message", "data": "VVVVVRIABAAgICAgTAQGAEP2IDEuMAAQBAAyNDAgohAEAKcdqB3GEAIAjJLSEAIAwKn+/wIAywY="} to slave-1 at address ('10.0.0.60', 55332)
Sent 119 {"type": "message", "data": "VVVVVQAAAgBBLRYAAgAgIAAEBAAwLjcuTAQCAEb2BBAEACAxMjCiEAQAqh2fHcYQAgCSktIQAgC+qf7/AgDMBg=="} to slave-1 at address ('10.0.0.60', 55332)

However, on the receiving end, it looks like this (Note how there are multiple concatenated messages in each receive loop):

Code: Select all

Received 650
{"type": "message", "data": "VVVVVQoABAAgICAgRgQIAAAAQIMCAHONohAEAHQdrh3GEAIAIpIkEQQAMCAgVf7/AgD/Bg=="}{"type": "message", "data": "VVVVVQ4ABAAgICAgSgQEAAIAd42iEAQAgB2wHbAQAgC5E8YQAgA6kigRBAAFAIBE/v8CAAAG"}{"type": "message", "data": "VVVVVRIABAAgICAgTAQGAHqNIDEuMKIQAgCJHcYQAgBNkiwRBAD//////v8CAAEG"}{"type": "message", "data": "VVVVVQAAAgBBLRYAAgAgIAAEBAAwLjcuTAQCAH6NohAEAJAdpx2uEAIA5xPGEAIAXJIwEQQA/+92fv7/AgACBg=="}{"type": "message", "data": "VVVVVQIABAAxMENfBAQEADUwTmVMBAIAgY2iEAQAlx2eHcYQAgBrkjQRBABKAAAA/v8CAAMG"}{"type": "message", "data": "VVVVVQYABAAyACAgCAQEAHcgY2FMBAIAhI2iEAQAnh2WHcYQAgB4ktIQAgCCuAIRAgA1MzgRBACD/AAAmBICAFsg/v8CAAQG"}
Received 797
{"type": "message", "data": "VVVVVQoABAAgICAgDAQEAGxsc2lMBAIAiI2iEAQAox2QHa4QAgDmE8YQAgCCktIQAgCAuDwRBAAAAP///v8CAAUG"}{"type": "message", "data": "VVVVVQ4ABAAgICAgEAQEAGduICBMBAIAi42iEAQAqB2KHbAQAgC6E8YQAgCMktIQAgB/uEARBAD//wAA/v8CAAYG"}{"type": "message", "data": "VVVVVRIABAAgICAgFAQEACAgICBMBAIAjo2iEAQArB2EHa4QAgDlE8YQAgCWkkQRBACBEgAA/v8CAAcG"}{"type": "message", "data": "VVVVVQAAAgBBLRYAAgAgIBgEBAAgICAgTAQCAJKNohAEALAdgB3GEAgAnZKIkrFRw1FIEQQAAAAAAP7/AgAIBg=="}{"type": "message", "data": "VVVVVQIABAAxMENfHAQEACAgTilMBAIAlY2iEAQAsx17HcYQAgClkkwRBAAAAAAA/v8CAAkG"}{"type": "message", "data": "VVVVVQYABAAyACAgIAQEAHcKIfBMBAIAmY2kEAIAdx2wEAIAuxPGEAYApJKIkrBRUBEEAAAA///+/wIACgY="}{"type": "message", "data": "VVVVVQoABAAgICAgJAQEACAgMTZMBAIAnI2iEAQAqh10Ha4QAgDkE8YQAgCQklQRBAAAAAAA/v8CAAsG"}
Received 111
{"type": "message", "data": "VVVVVQ4ABAAgICAgKAQEACAgIDlMBAIAn42iEAQAoB1xHcYQAgB9ktIQAgB9uFgRBAAQdgAA/v8CAAwG"}
Received 448
{"type": "message", "data": "VVVVVRIABAAgICAgLAQEABAACQBMBAIAo42iEAQAmB1yHcYQAgBsktIQAgB8uFwRBAAAAP9//v8CAA0G"}{"type": "message", "data": "VVVVVQAAAgBBLRYAAgAgIDAEBABFNBOQTAQCAKaNohAEAJEdex3GEAIAXpJgEQQA/38wNv7/AgAOBg=="}{"type": "check-in"}{"type": "message", "data": "VVVVVQIABAAxMENfNAQEAEIA+epMBAIAqo2iEAQAix2CHcYQAgBRkmQRBAA3WP9//v8CAA8G"}{"type": "message", "data": "VVVVVQYABAAyACAgOAQEADA5OjJMBAIArY2iEAQAhR2JHcYQAgBFkmgRBACEBP///v8CABAG"}

[lbernstone]

UDP (and AsyncUDP) is probably more appropriate for this use.
You have a lot of extra stuff (esp the mDNS) in your loop that is going to be taking time, such that the polling happens less frequently than you want. If you make a separate task that is only polling for received data, and pushes it into a queue, you will be able to capture this the way you want, depending on how much additional processing you have going on in the background. https://github.com/espressif/arduino-es ... /Queue.ino
There is an AsyncTCP library which will trigger callbacks whenever a packet is received, but it is not designed for ease of use. https://github.com/me-no-dev/AsyncTCP/b ... yncTCP.cpp. For this use case, it will probably not be significantly different than polling in a separate task (which is what it does too, just then fires off implicit callbacks).

[maciekish]

I solved it by tagging the packets with NODELAY, setting the TOS and sending them OOB on the server side. Buffering must have happened at WiFi AP or ESP32.

Code: Select all

s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.setsockopt(socket.IPPROTO_IP, socket.IP_TOS, 0x10)
s.send(message.encode(), socket.MSG_OOB)