ESP32S3 files sharing and transfering

[DoDobro!]

Hello!

I've bough those ESP32S3 boards with integrated WiFi, Bluetooth, Display, SD-card reader to use as sensors data logger to file.

http://pan.jczn1688.com/directlink/1/ES ... 48S050.zip

After sensors data will be logged to files I need to transfer those log files from this device SD-card to another devices (PC or Smartphone) without needing to insert&remove SD all the time. Which options do I have?

A) USB - connect ESP32 board via USB and open SD-card as USB memory storage device (but USB-port can be also used for another purpose as another devices connection, power, etc).
B) WiFi network - share SD-card as folder with files via WiFi network.
C) WiFi direct - transfer files from ESP32S3 device to another peer-to-peer.
D) Bluetooth - not sure is it possible to also share SD-card as folder with files via Bluetooth? Or just send files from SD-card on each BT connection.

Unfortunately I can't find needed info and libraries&samples at Arduino IDE for a such "file sharing" tasks at ESP32.

Please, answer to the next questions:
1) Which file sharing firmware (USB/WiFi/BT) will occupy more or less of ESP32S3 memory?
2) Which file sharing firmware (USB/WiFi/BT) will use more of ESP32S3 CPU power?
3) Via which bus USB/WiFi/BT are connected to ESP32 core? SPI, I2C, another?
4) Does ESP32S3 WiFi have option to minimize radio signal power when not used (like Bluetooth LE)?
5) Do you have a such libraries and samples for A/B/C/D?

[MicroController]

After sensors data will be logged to files I need to transfer those log files from this device SD-card to another devices (PC or Smartphone) without needing to insert&remove SD all the time. Which options do I have?

Running an HTTP server on the ESP is pretty much the standard way to do it.
MQTT may also be an option.

Unfortunately I can't find needed info and libraries&samples at Arduino IDE for a such "file sharing" tasks at ESP32.

Plenty of examples for HTTP server (or MQTT client) are available.

Please, answer to the next questions:
1) Which file sharing firmware (USB/WiFi/BT) will occupy more or less of ESP32S3 memory?
2) Which file sharing firmware (USB/WiFi/BT) will use more of ESP32S3 CPU power?
3) Via which bus USB/WiFi/BT are connected to ESP32 core? SPI, I2C, another?
4) Does ESP32S3 WiFi have option to minimize radio signal power when not used (like Bluetooth LE)?
5) Do you have a such libraries and samples for A/B/C/D?

3) completely irrelevant, but the answer is "others"
4) Yes, including shutting down WiFi completely or using "modem sleep" mode
5) see above; look for Arduino ESP32 HTTP server

[DoDobro!]

Running an HTTP server on the ESP is pretty much the standard way to do it. MQTT may also be an option. Plenty of examples for HTTP server (or MQTT client) are available.

Thanks for answers!
Yes, web-http now must be the most used data transferring protocol just because of internet.
But I'm not sure that it's the best option for simple small files transferring because of:
- Firmware require code for WiFi.
- Firmware require code for HTTP-server.
- Developer needs to turn both of them and make some kind of "HTTP frontend for files listing and downloading".
- Suppose this "server" to be accessible through WiFi must work for 100% all the time, so it will not have possibility for any "sleep modes" and it will drain much more power and emitting much more radio signals then special "auto-initiated Bluetooth 5 LE waiting mode" or even "WiFi Direct" which also start to work at 100% after peering request. Also suppose such HTTP-server will take some CPU power all the time.
- If you change the network you will need to search this HTTP-server IP somehow each time.

Compare it with Bluetooth 5 LE files transferring:
+ Firmware require code only for BT.
+ No need of any custom coding, frontend, etc.
+ 10-100 times less power drain and radio signals emitting, because BT LE sleep/waiting mode is hardware function it must not waste any CPU power.
+ Hardware MAC is always stay the same - once connected it will be accessible fast&easy always.

Compare it with USB files transferring (open SD-card as "USB mass storage device"):
+ Firmware require code only for USB MSD.
+ No need of any custom coding, frontend, etc.
+ No any additional power drain and no any radio signals emitting, no CPU power wasting when not used.
+ Ready immediately upon connection.

So I still asking for some code sample of ESP32S3 Bluetooth 5 LE files transferring (better for Arduino IDE).

[MicroController]

Yes, web-http now must be the most used data transferring protocol just because of internet.
But I'm not sure that it's the best option for simple small files transferring because of:
- Firmware require code for WiFi.
- Firmware require code for HTTP-server.
- Developer needs to turn both of them and make some kind of "HTTP frontend for files listing and downloading".

This is true. But it's even more true for BLE: BLE provides no "file sharing" protocol out of the box (see e.g. https://github.com/espressif/esp-idf/issues/7025). So instead of using available HTTP server code, you'll have to write your own "file server" protocol on top of BLE, which is somewhat cumbersome given the architecture and limitations of BLE.

- Suppose this "server" to be accessible through WiFi must work for 100% all the time, so it will not have possibility for any "sleep modes" and it will drain much more power and emitting much more radio signals then special "auto-initiated Bluetooth 5 LE waiting mode" or even "WiFi Direct" which also start to work at 100% after peering request. Also suppose such HTTP-server will take some CPU power all the time.

Being automatically accessible 24/7 is a new requirement. If the user can be made to press a button to initiate a WiFi transfer, this can be avoided.
Inactive/waiting components using "CPU power" is not a thing.

- If you change the network you will need to search this HTTP-server IP somehow each time.

Indeed. Plus you may need a way for provisioning WiFi credentials.

Compare it with Bluetooth 5 LE files transferring:
+ Firmware require code only for BT.
+ No need of any custom coding, frontend, etc.

This is an incorrect assumption, see above. Additionally, every(*) IP-networked device already has a web browser and hence can access files via HTTP, while for a custom BLE transfer protocol you'll also have to provide the software for receiving the files.

+ 10-100 times less power drain and radio signals emitting, because BT LE sleep/waiting mode is hardware function it must not waste any CPU power.

For better or worse, the energy saving potential of BLE over WiFi isn't that huge on ESPs.

+ Hardware MAC is always stay the same - once connected it will be accessible fast&easy always.

True.

Compare it with USB files transferring (open SD-card as "USB mass storage device"):
+ Firmware require code only for USB MSD.
+ No need of any custom coding, frontend, etc.
+ No any additional power drain and no any radio signals emitting, no CPU power wasting when not used.
+ Ready immediately upon connection.

Depends on what "no" custom coding means, but mostly correct.

So I still asking for some code sample of ESP32S3 Bluetooth 5 LE files transferring (better for Arduino IDE).

As stated above, since there's no standard way (except for maybe OTS) in BLE, you'll have to search for custom solutions/libraries; and have compatible code running on the other device.

[DoDobro!]

But it's even more true for BLE: BLE provides no "file sharing" protocol out of the box (see e.g. https://github.com/espressif/esp-idf/issues/7025). So instead of using available HTTP server code, you'll have to write your own "file server" protocol on top of BLE, which is somewhat cumbersome given the architecture and limitations of BLE. As stated above, since there's no standard way (except for maybe OTS) in BLE, you'll have to search for custom solutions/libraries; and have compatible code running on the other device.

Agree that WiFi HTTP server is the most common standard now but it have it's own disadvantages which I've wrote. That why I want to try BT. Answer to your text quote: Then how Windows for many years can send&receive files through Bluetooth without any custom app??? Suppose BT already have it's own file transfer protocol standard. https://www.bluetooth.com/specification ... ofile-1-3/

What about WiFi Direct on ESP32S3 - do you have a such samples?
What about USB mass storage device (USB flash drive) on ESP32S3 - do you have a such samples?

[MicroController]

how Windows for many years can send&receive files through Bluetooth without any custom app???

By using Bluetooth I.e. "Bluetooth classic", which is much more different from Bluetooth Low Energy than one would initially think.

Basically, what I'm trying to say is probably: The requirements for the (ESP-)software side of things are not as you stipulated above. USB+MSC stack are absolutely not "free" w.r.t. resource (flash, RAM, CPU) usage or required development effort. Neither are WiFi-HTTP or BLE.
You will struggle to find an easy to use, interoperable BLE solution/example. WiFi+HTTP is well established, documented, and has plenty of examples out there. If WiFi doesn't fit into your energy budget, BLE can be used, but there are reasons why people keep developing their own custom solutions for file transfer via BLE.
I don't see how WiFi Direct would solve any energy issues 'normal' WiFi would have.
And USB-MSC... is probably the most resource-hungry option, albeit likely the most energy efficient overall.

Again, an HTTP server is readily available and maintained in both ESP-IDF and Arduino, the other options aren't. I suggest you decide on your definitive requirements (ease of use, power consumption, code complexity, memory usage, documentation & support,...) and then on which trade-offs you want to take between them.

[DoDobro!]

By using Bluetooth I.e. "Bluetooth classic", which is much more different from Bluetooth Low Energy than one would initially think. I don't see how WiFi Direct would solve any energy issues 'normal' WiFi would have. And USB-MSC... is probably the most resource-hungry option, albeit likely the most energy efficient overall.

Very very pity that nor ESP32-IDF, nor Arduino - don't have nor Bluetooth LE power efficient radio files transferring SW, nor WiFi Direct - after so many years of they exist... it's like a joke or huge mistake.
WiFi Direct can sleep (wait for incoming initiation) while not using and because of peer-to-peer connection it's MAC address stay the same - so with WiFi Direct you don't need to reconnect to each new network again and again!
Also HTTP-server by it's basal own and specially "always on HTTP-server" - is potentially the MOST highest vulnerability to attacks!!!
Thus at security aspect maybe it's better to use FTP - this protocol is more simple but also more stable.
Still absent of BT LE and WiFi Direct file transferring libraries frustrates extremely high!

What do you think about?
https://espressif-docs.readthedocs-host ... b_msc.html
https://github.com/adafruit/Adafruit_TinyUSB_Arduino

[MicroController]

I forgot to mention: There's also support for BLE "Connection-oriented Channels" in NimBLE (https://github.com/espressif/esp-idf/tr ... _l2cap_coc) which could facilitate the transfer of file data via BLE.
Still not a "file sharing" protocol, but could be built upon.

[DoDobro!]

Because all good (functional, power efficient, not too bulky, etc) libraries for file transferring via Bluetooth 5/LE and USB MSC is absent, this forced me to use "Web HTTP server".

Unfortunately standard sample "ESP32-SDWebServer" is synchronous which means that when anyone connected to this WebServer it blocks any(!) other task at ESP32 device! Which is very bad.

Then I've found newer asynchronous ESPAsyncWebServer - https://github.com/me-no-dev/ESPAsyncWebServer
But it lack of samples which works with SD.

Then I've found tutorial of how to use ESPAsyncWebServer with SD - https://randomnerdtutorials.com/esp32-w ... rosd-card/
But it only display very simple "index.html" from SD to WebServer users which open it's IP, but files manager as listing and dowloading/uploading from SD is absent.

I've try to use "index.htm" and "edit.htm" from "ESP32-SDWebServer" with "ESPAsyncWebServer with SD" but they don't work with each other.

After all I've found very simple and enough functional and nice looking "files manager" for "ESPAsyncWebServer" which does not require any additional web files which is very nice! But it works with ESP32 internal memory SPIFFS only and does not work with SD - https://github.com/smford/esp32-asyncwe ... d-example/
(Not sure, but the only thing which maybe it lacks is working with folders on SD - list folders, enter folder, create folder, delete folder, etc.?)

One more similar project for "ESPAsyncWebServer + SPIFFS" with more functions but very bad&outdated aesthetic:
https://github.com/G6EJD/G6EJD-ESP-File-Server - https://www.youtube.com/watch?v=TmV6JAUtrvw

So can anyone make a mix-fork of "smford/esp32-asyncwebserver-fileupload-example" for "ESPAsyncWebServer with SD and SPIFFS" + directories support and maybe system info (Status tab) from G6EJD-ESP-File-Server, please!

[username]

Espressif has an example doing this. But its for IDF, should not be any trouble to get it working for Arduino
https://github.com/espressif/esp-idf/tr ... le_serving