ESP-IDF monitor eating random newline characters (IDF-6647)

[pawel.cern]

Hi,

I observed occasional phenomenon - some (not all) newline characters are lost. Quick debugging with strace proved all data delivered to /dev/ttyACM interface is complete and correct, there is a bug in python code. I made some investigation and found problem in tools/idf_monitor_base/serial_handler.py. Patch below:

diff --git a/tools/idf_monitor_base/serial_handler.py b/tools/idf_monitor_base/serial_handler.py
index b3a71f64a2..394dca021e 100644
--- a/tools/idf_monitor_base/serial_handler.py
+++ b/tools/idf_monitor_base/serial_handler.py
@@ -88,13 +88,8 @@ class SerialHandler:
if self._last_line_part != b'':
# add unprocessed part from previous "data" to the first line
sp[0] = self._last_line_part + sp[0]
- self._last_line_part = b''
- if sp[-1] != b'':
- # last part is not a full line
- self._last_line_part = sp.pop()
+ self._last_line_part = sp.pop()
for line in sp:
- if line == b'':
- continue
if self._serial_check_exit and line == console_parser.exit_key.encode('latin-1'):
raise SerialStopException()
if gdb_helper:

[ESP_Roland]

Hi pawel.cern,

Thank you for pointing out the issue and suggesting a patch for it. Before adding the patch to ESP-IDF, I'd like to reproduce the issue and see that it is not causing any regression (I'm worried about the removed lines which are there for a reason). Could you please give me some hints how where you able to reproduce this? Do you have any simple project to reproduce?

By any chance, have you observed the content of "sp"? Was it correct? Are you using "\r\n" (Windows) line endings?

Thank you again!
Roland

[pawel.cern]

OK, I managed to prepare code, which reproduces the problem. Please set up sdkconfig in such way that line endings are LF (driver won't do conversions) and try the following code:

#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>

void app_main(void)
{
while(1)
{
printf("Text 1\r\n");
fflush(stdout);
usleep(1000000);
printf("Text 2\r");
fflush(stdout);
usleep(1000000);
printf("\nText 3");
fflush(stdout);
usleep(1000000);
printf("\r\n");
fflush(stdout);
usleep(1000000);
}
}

Expected result (patched serial_handler.py) is:

Text 1
Text 2
Text 3
Text 1
...
...

Current result (existing esp-idf) we have:

Text 1
Text 1
Text 1
...
...

but if you observe console, you will see overwriting Text2 / Text3.

[ESP_Roland]

Thank you for the additional information. That explains it. Because the content of "sp" was created by splitting the buffer content based on "\n".

The issue is tracked and we will get back to this.

[ESP_Roland]

This should been fixed in esp-idf-monitor 0.1.1 included with ESP-IDF v5.1 (current master branch).