Socket issues

[permal]

Looks like they both send 7x tcp_output_segment, but one case receives many unacceptable reset and the other only receives 1 or none. So is there firewall or something else rejecting these packets before they get to wireshark/docker?

There is a firewall, but it is not blocking anything going to port 1883, other clients can connect while the ESP32 cannot. I cannot disable the FW since docker requires it to run.

There might some difference between our test:

As I wrote earlier, I can only reproduce the behavior when Docker is involved - running the MQTT broker directly on a PC yields in the expected results. Also, during your tests, did you restart the ESP32 during your 4:th step or did you program the application to attempt reconnection? I've found that resetting the ESP32 usually results in a successful connection even if it couldn't connect before the reset, which makes sense since the problem only occurs when the first connection attempt fails.

And could you paste the packet capture log after you setup server? did ESP32 send SYN to server?

Do you mean the packets seen from tshark after the server is started? There are none - the ESP32 seemingly doesn't send anything once the problem has been seen.

I'm quite certain that the issue only happens when lwIP reports "Software caused connection abort" and not when it reports "Connection reset by peer".

[WiFive]

There may not be any firewall rules on that port, but say a client tries repeatedly to connect to a closed port, a firewall may detect that as an attacker and impose a rate limit or block.

Esp32 is receiving reset packets in response to the syn so they have to be coming from somewhere. Somewhere before the packets get to the interface tshark is monitoring.

[permal]

There may not be any firewall rules on that port, but say a client tries repeatedly to connect to a closed port, a firewall may detect that as an attacker and impose a rate limit or block.

Esp32 is receiving reset packets in response to the syn so they have to be coming from somewhere. Somewhere before the packets get to the interface tshark is monitoring.

What you say is sound reasoning, but it doesn't explain why the ESP32 is able to connect if you reset it, as I'm sure I have mentioned earlier. It's still the same IP & MAC after reset so to the firewall it'll look as the same client.

[WiFive]

Yes that's true, except maybe when esp32 is reset its ARP and DNS caches are cleared and that combined with docker network layer somehow affects routing.

[permal]

Could be. That still points to the ESP32 though.

[heyinling]

@permal could you push your docker image to docker hub and also provide application code so we can reproduce this issue?

[permal]

@permal could you push your docker image to docker hub and also provide application code so we can reproduce this issue?

Docker image info is in this post: viewtopic.php?f=2&t=2665&start=10#p12632

Like I wrote in an earlier, as far as I can tell, tcp_perf shows the same problem during the connection phase when connecting against Docker so you should be able to use that. If you want to run something bigger that has reconnect behavior built-in, you can clone my "test" repo for my Smooth framework: https://github.com/PerMalmberg/Smooth-test, be sure to cone it recursively so that you get it all and update the file wifi-creds.h with your credentials.

As I'm actively working against the repo, I suggest you checkout commit cf732d26c2db26d8449a76043145c7de1a8d903f

[jimmy.rivers]

Hi Permal. I have experienced the same issue. Using simple TCP Client application I am able to determine after a valid socket is opened, data is sent the remote side closes the connection and reopens listeneing for a new connection. If I simply do a connect on the same socket it always fails. If I call ip = socket(AF_INET, SOCK_STREAM,IPPROTO_TCP); I get a new number in ip this increases every time I call socket();. It starts with 0 then increases. If I keep closing & opening the remote listening socket, I can keep doing this till the ip number on a call to socket(); returns 9. so having gone from 0 to 9. there after I get a response to socket(); of -1 and the errno returned is 9 translated to "Bad file number". From then onwards I can never connect and need to restat the whole module.
I understand bigger system have thousands of sockets, but is this a limitation I should expect. If so is there anything I can call to reinitialise the socket system. Hope I am on topic here.

[permal]

Seems similar regarding not being able to reconnect, but that you are running out of free sockets is likely due to not calling close() on the socket you already tried to open.

[permal]

@heyinling Any progress reproducing the issue?