usb_serial_jtag_write_bytes() doesn't always write all bytes

[ESP_Sprite]

Nope. I want to test your code, but I need some time and concentration to get all the parts (korvo board, sdcard) together. If you could simplify your app a bit more so I can run it on a random devkit, that'd probably make life easier for me.

[jmadsenee]

Hi @ESP_Sprite, This should run on a random -S3 dev kit. When the program starts, it creates a ~550KB file in spi flash, which it then tries to YModem to a PC.

After downloading the program to the dev kit, open a terminal program such as Tera Term, connect, and press the reset button. You'll see the messages:

I (366) storage: spiflash Filesystem mounted
I (506) storage: Create text file
I (11286) storage: File written

Sending file "/data/text.txt", please start YModem receive on host ...
CCCCCCCC

As you can see, it takes a few seconds between Create text file and File written.

On Tera Term I do a File>Transfer>YMODEM>Receive... and some portion of the file transfers and then it stops. Pressing cancel (within 10s) outputs the rest of the buffer.

On occasion, the transfer completes, but most of the time it stops in random spots.

Thanks so much for looking into this!

John

[jmadsenee]

Hi @ESP_Sprite, I know you're busy, but this is becoming critical for my project. Have you had a chance to look at the modified version yet? Thanks so much!

[ESP_Sprite]

Yes, but it's still puzzling me a bit... smells like maybe a hardware race condition of some sort?

Can you double-check what I'm seeing and see what results you get when you change the '64' in this line in esp-idf to '63'?

[jmadsenee]

@ESP_Sprite, Well, that seems to have done it! I ran my minimal program 10 x without a single failure. I then made the change in my real project and ran it 3 x without failure.

Is that a real solution or a kluge? Do I have to worry about it causing other problems? The console part of my app seems to work, but I did not do extensive testing.

Thanks very much for your help!

[ESP_Sprite]

It's somewhat of a kludge; I think there's some hardware race condition when 64 bytes are written to the FIFO (which implicitly triggers a flush) and a manual flush done afterwards; by only writing 63 bytes, the automatic flush never happens and whatever triggers the bug doesn't occur. I need to poke the digital team to figure out the root cause here to see if there's a more elegant fix, but I think this should hold up as a workaround for now.

[jmadsenee]

Hi @ESP_Sprite, did you learn any more about this issue? Is it hardware? Is there a better fix?

[ESP_Sprite]

Not yet, our digital team is still squirreling out the root cause for this one.

[ESP_Sprite]

It actually looks like a quirk of USB that is not handled correctly. If a packet is sent for the full 64 bytes and nothing after, the host OS will think that it's only part of an USB transfer, and will hold onto that information (=not send it to the client program) until it receives a non-64-byte packet. Normally, in an interactive session, this tends to not lead to an issue, but in your case it does, as it leaves the ymodem program waiting for data that is held by the OS and the ESP waiting for an ack.

As such, the solution to only send 63 bytes at a time is a bit crude, but certainly a valid workaround for this issue. I'll see if I can add a feature to the driver to only do this when it's needed, but looking at what I tested, there's not much speed gain to be gotten by 'properly' working around it.