Device can't receive data after reconnection without significant delay

[muzos07]

Introduction
I have my head scratching by one thing regarding the HWCDC and serial communication for couple of days now.
It seems like serial.open() in various frameworks (reproducible with QtSerialPort and PySerial) does some initialization that needs to finish before serial.write() goes through, BUT only if the device reconnection happened during software runtime. I would like to know why is there difference between "device reconnection" vs "device connected beforehand".

Code example
Let me explain it by code. Consider this minimal device code:

Config

Code: [Select all] [Expand/Collapse]

1. [env:tdisplay]
2. platform = espressif32
3. board = lilygo-t-display-s3
4. framework = arduino

GeSHi © Codebox Plus Extension

Device code
It basically waits for 5B header and returns 256B of data.

Code: [Select all] [Expand/Collapse]

1. #include <Arduino.h>
2.  
3. void setup()
4. {
5.     Serial.begin(115200);
6. }
7.  
8. void loop()
9. {
10.     // Set timeout to -1 so the "readBytes" waits forever for the first 5 bytes
11.     Serial.setTimeout(-1);
12.  
13.     // Read header
14.     uint8_t header[5];
15.     Serial.readBytes(header, 5);
16.  
17.     // Set timeout to 3 seconds in case something goes wrong, so we don't get stuck but handle the error
18.     Serial.setTimeout(3000);
19.  
20.     ... <handle the header here, header includes number of bytes that should follow> ...
21.  
22.     ... <in my test case, the code decides to return specific array of 256 bytes> ...
23.  
24.     Serial.write(responseBuffer, 256);
25. }

GeSHi © Codebox Plus Extension

Python code
Now consider this Python code. It goes through infinite loop and just sends the command for obtaining said 256B of data and if the device returns less, print the "Response too short \<number_of_bytes_read\>" message.

Code: [Select all] [Expand/Collapse]

1. import serial
2.  
3. ser = serial.Serial(
4.     port=None,
5.     baudrate=115200,
6.     timeout=0.5,
7.     write_timeout=None,
8. )
9.  
10. ser.port = <some specific port, "/dev/cu.usbmodem101" in my case>
11. command = <bytearray with command that expects 256B back>
12.  
13. while True:
14.     print("----------------------")
15.     try:
16.         # open the port
17.         ser.open()
18.         print("Opened serial port")
19.  
20.         # CRUCIAL PART, WITHOUT THIS, THE DEVICE NEVER RECEIVES DATA AFTER RECONNECTION
21.         # time.sleep(1.5) # commented out now to introduce the problem in the next code block
22.  
23.         # write command that instruct the device to sent 256B back
24.         bw = ser.write(bytes(command))
25.         print("Wrote command " + str(bw))
26.  
27.         # read 256B of data
28.         br = ser.read(256)
29.         print("Read response 256")
30.  
31.         if len(br) != 256:
32.             print("Response too short " + str(len(br)))
33.     except serial.SerialException as ex:
34.         # We don't care in this test, most errors are due to device being disconnected from the PC (Mac in my case)
35.         print("Ex: " + str(ex))
36.     finally:
37.         # Close the port so it is ready for the next iteration
38.         ser.close()
39.         time.sleep(1)

GeSHi © Codebox Plus Extension

Current behavior
Now, let me run the python code and print output here (I will describe my physical actions and thoughts in \<\>):

Code: [Select all] [Expand/Collapse]

1. <I connect device to the PC>
2. <I start the python code>
3.  
4. ----------------------
5. Opened serial port
6. Wrote command 7
7. Read response 256
8. ----------------------
9. Opened serial port
10. Wrote command 7
11. Read response 256
12. ----------------------
13. Opened serial port
14. Wrote command 7
15. Read response 256
16. ----------------------   <So far so good, device is working as expected>
17. Opened serial port
18. Wrote command 7
19. Ex: read failed:         <I disconnect the device from the PC here>
20. [Errno 6] Device not
21. configured    
22. ----------------------
23. Ex: [Errno 2] could not
24. open port /dev/cu.usbm
25. odem101: [Errno 2] No
26. such file or directory:
27. '/dev/cu.usbmodem101'
28. ----------------------
29. Ex: [Errno 2] could not.  
30. open port /dev/cu.usbm
31. odem101: [Errno 2] No.   <These errors are expected as no device is connected to the PC >
32. such file or directory:
33. '/dev/cu.usbmodem101'
34. ----------------------
35. Ex: [Errno 2] could not
36. open port /dev/cu.usbm
37. odem101: [Errno 2] No
38. such file or directory:
39. '/dev/cu.usbmodem101'
40. ----------------------
41. Opened serial port       <I connect device back to the PC here>
42. Wrote command 7
43. Read response 256
44. Response too short 0     <Device is not returning any data?>
45. ----------------------
46. Opened serial port
47. Wrote command 7
48. Read response 256
49. Response too short 0     <This continues indefinitely.>
50. ----------------------
51. Opened serial port
52. Wrote command 7
53. Read response 256
54. Response too short 0
55. ----------------------
56. Opened serial port
57. Wrote command 7
58. Read response 256
59. Response too short 0
60. ----------------------   <How to fix this? Introduce delay after serial.open(). But why?>

GeSHi © Codebox Plus Extension

Additional information
Here is some more relevant info I didn't put in code samples:

• I put LCD blinking at the end of setup() function. This tells me when exactly setup finished and loop() started
  • in DFRobot with basic HardwareSerial, it blinks almost immediately after introduction of power
  • in T-Display-S3 with HWCDC, it takes about 3 seconds to finish setup()
• I have some local flash storage logging on and I can see, that the device is always ready to receive commands after reconnection, but it never receives any, thus it has no reason to sent 256B of data back.
  • It basically call setup() only once after reconnection
  • This indicates problem on the OS level, not on the device level?
• The delay between serial.open() and serial.write() must be about 1.5-ish seconds or higher in order to "survive" device reconnection.
  • The delay itself was found out in https://stackoverflow.com/questions/361 ... 9#36736569 by user "r21"
• I suppose this is a problem related to HWCDC since I tested this with two devices:
  1. FireBeetle 2 ESP32-E:
     • using the most basic HardwareSerial class as Serial (but I need to install CH34xVCPDriver in order to see it in my Mac)
     • no problems at all
  2. LilyGO T-Display S3:
     • using HWCDC class as Serial
     • reproducible

Questions

My main question is, why does this happen? Why do I have to introduce delay between serial.open() and serial.write()? The device may have not been ready in the first moments after reconnection, I understand that. But why does it happen indefinitely? The port must be ready at some point, right?.

My secondary question is, why does this happen only when physical device reconnection during software runtime occurred? I mean, the device is in the same state (waiting for commands) some moments after reconnection as it would have been connected before the software started.

What is wrong?
One could argue something like "serial.open() needs some time to initialize before it can write, HWCDC is more complicated, needs more time", but why is then immediately ready in the first few iterations when the Python code is started and the device was connected beforehand?

There, it should also require some time to initialize, right? If answer would be "well, the device was connected before hand and was initialized on the OS level" or something like this, then why wasn't the conversation (5B <-> 256B) repaired after the device was reconnected and took some time to initialize on the OS level. You see my thought process?

Please, if you have answer to any of my questions, let me know.

Expected behavior
This is expected behavior, that occurs when trying the same thing with my first DFRobot device that uses HardwareSerial:

Code: [Select all] [Expand/Collapse]

1. ----------------------
2. Opened serial port
3. Wrote command 7
4. Read response 256
5. ----------------------
6. Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
7. ----------------------
8. Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
9. ----------------------
10. Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
11. ----------------------
12. Opened serial port        <first iteration after reconnection fails, this is fine>
13. Wrote command 7
14. Read response 256
15. Response too short 0
16. ----------------------
17. Opened serial port        <second iteration after reconnection works>
18. Wrote command 7           <all other iterations work>
19. Read response 256
20. ----------------------

GeSHi © Codebox Plus Extension

All I want is that my device starts sending data again after physical reconnection. I understand some delay is needed to startup the device, but I don't understand why this delay is indefinite.

[muzos07]

Solved by setting -DARDUINO_USB_MODE on 0 in platformio.ini:

Code: [Select all] [Expand/Collapse]

1. board_build.extra_flags =
2.     -DARDUINO_LILYGO_T_DISPLAY_S3
3.     -DARDUINO_USB_MODE=0           ; <-- 1 is here by default
4.     -DARDUINO_USB_CDC_ON_BOOT=1
5.     -DARDUINO_RUNNING_CORE=1
6.     -DARDUINO_EVENT_RUNNING_CORE=1

GeSHi © Codebox Plus Extension

Default values are from: https://github.com/platformio/platform- ... ay-s3.json