Page 1 of 1

Wierdly encoded characters when receiving data.

Posted: Wed Feb 08, 2017 7:40 pm
by Feelfree
Hi! I'm writing IRC bouncer on esp32. Implemented basic functionality, but when I receive data, I'm randomly getting characters replaced like ToggleSwitchWid�Ž�Žû?�gû?�, cherryh.freenode.net @�û?Í352 - Hmm, maybe its something with this buffer to string operation? Thanks!
https://gist.github.com/feelfreelinux/8 ... cbda0f463e

Re: Wierdly encoded characters when receiving data.

Posted: Wed Feb 08, 2017 10:16 pm
by ESP_Angus
Hi Feelfree,

You don't seem to have tried fixing the buffer overflow issue that I mentioned on github (receiving the full size of the buffer doesn't leave space for a terminating null byte, so you can't treat it as a string). Are you sure that's not the problem?

Angus

Re: Wierdly encoded characters when receiving data.

Posted: Thu Feb 09, 2017 10:34 am
by Feelfree
ESP_Angus wrote:Hi Feelfree,

You don't seem to have tried fixing the buffer overflow issue that I mentioned on github (receiving the full size of the buffer doesn't leave space for a terminating null byte, so you can't treat it as a string). Are you sure that's not the problem?

Angus
Forgot to update gist - resized buffer to 500, problem still persist. Looks like it's only seen when there's a lot of messages in short peroid of time(Like IRC motd).

Re: Wierdly encoded characters when receiving data.

Posted: Thu Feb 09, 2017 5:51 pm
by ESP_igrr
Can you update the gist now? It still has the problem Angus has described, that is you aren't zero-terminating your string.
Second thing: if you increase buffer size, make sure you increase the stack size as well.

Re: Wierdly encoded characters when receiving data.

Posted: Fri Feb 10, 2017 11:06 am
by Feelfree
Ok, updated gist.

Re: Wierdly encoded characters when receiving data.

Posted: Fri Feb 10, 2017 4:38 pm
by Feelfree
And I ported this to PC, linux. This code works correctly here. Looks like its something with lwip/esp32. https://gist.github.com/feelfreelinux/9 ... 44a9ae6186

Re: Wierdly encoded characters when receiving data.

Posted: Tue Feb 14, 2017 3:10 am
by ESP_Angus
The Linux version doesn't (yet) buffer overflow because it's receiving less than 500 bytes at a time (a combination of more CPU power and faster standard output as it's not speed limited by the serial port). The potential buffer overflow is still there on Linux, it just isn't triggered as easily (some combination of CPU load and network load would eventually trigger it).

EDIT: actually a better explanation of why the Linux version doesn't crash is that the first byte of data after the 500 byte buffer happens to be zero. You can't rely on this , though!

Here's the ESP32 main.cpp file modified to avoid the buffer overflow. This works for me, whereas the previous version triggers a fatal exception or produces garbage output: https://gist.github.com/projectgus/8596 ... a6f0b5fbfa

Some tips for things to read about in order to gain more skills with this kind of network code development:
  • Learn what a null-terminated string is in C, and what the difference between a string and a char buffer is in C. Knowing this difference will help you understand why the code above fixes the problem, whereas increasing the buffer size to 500 did not.
  • Read the 'man' pages for recv, connect, socket. Update your code to check results from these.
  • You'll probably want to adapt your code to use 'select' at some point.
  • In your code, remember that a single call to recv() can return any number of bytes (TCP is a byte stream not a packet stream) - so you can receive anything between 0 whole lines of output (ie a partial incomplete line) and multiple lines of output from the server in one hit. You'll need to accumulate that raw data into a buffer, and then parse that buffer for lines of output.
Enjoy!