You CAN. It has the huge advantage that it's human readable. You can just print whatever is going over the wire using whatever terminal emulator makes you happy with a couple of wire clips and you can instantly know if the sender sent what you thought and the receiver received that. Don't give that up trivially.
"Obvious" problems with your current code are that it handles only a single byte (maybe you want to scanf the buffer?) and there's no acknowledgement/retry/cancellation/ if the receiver missed it (perhaps it wasn't plugged up yet, reset, or whatever) and more. You're in the ESP-IDF group, but you asked with an Arduino example, so it's not clear what you really want, but you can find TONS of tutorials on the basic topic online and in youtube.
Now if you're sending a zillion bytes a second across a thousand comm lines and are sending more than one digit, you might guess this may not be the most compact representation. AFTER (and only after) you've mastered a single number at low speed AND you've hit the wall on performance, data size, cell phone bill, or whatever, you may need to look at at alternative ways to encode and read those numbers. The magic words are "serialization formats" and then you can learn about protobuf, flatbuffers, cereal, json (ick), CBOR and
more. Don't even think about those until you've mastered the easy case and run into the limitations, though.
The ability to see what you sent AND send/parse with code that looks somewhat like what you just showed is too great to give up casually.
Let easy things be easy until they actually have to be not-easy!