Using an HTTP GET method to download large files (upto 8MB) to an external EEPROM of ESP32-WROVERB

[kyleneville]

I have a custom made chip which has a ESP32 WROVERB chip. The chip has an external flash memory/EEPROM of 256Mb (32MB). My task is to download a large .bin file from the internet (say dropbox.com) and save it to the external EEPROM. After that's done, I read and process the data from the EEPROM.

As an alternative, I have tried SPIFFS, but I don't think that I can make use of the external flash for it. It says that the memory is full.

For the application that I am building, I need to make this work over the internet. I'm looking for some pointers because I've tried looking on Google and I've found only things like SPIFFS or Websockets Server examples. I even tried uploading my file through a local websocket, but the WiFi-AP just restarts when I select a big file.

Thanks in advance!

[Scott.Bonomi]

I think there is likely a problem with getting such a file all at once. You would need to have something like DMA compatable RAM to drop the entire file into and I do not think any of the ESPs have enough of that that today. If you send it in small blocks with TFTP or some more robust protocol, you should be able to move blocks to your external memory and with the robust protocols request resend for any dropped blocks.

You may be able to set your external memory up as DMA accessible, I do not know.

[dastoned]

The ESP IDF includes a reasonably capable HTTP(S) client:
https://docs.espressif.com/projects/esp ... lient.html#

If I understand this correctly the HTTP client will download the file in chunks to a RAM buffer (the size of which is up to you to define). On receiving each chunk of data your callback is invoked, which allows you to copy the chunk into the EEPROM or do whatever else you like.

Since you're writing data to the EEPROM on the fly, a bit of extra care needs to be taken to handle communication disruptions gracefully (i.e. delete the incomplete file and retry; or resume from the last received byte).

[PeterR]

Pray that the HTTP client does the flow control.
The emac task (ESP32 mac) will just malloc on frame (rather inefficientlty if you ask me) & close connection on fail.
So if throughput is high and memory low....
I have only seen this become an issue on local loop. Network traffic is slower I guess.
But for you I would ask if waiting on slow EEPROM also give flow contol and so then buffer/memory issues?
I would not ask except that the CAN driver will lock fail on overflow & issue corrupt frames etc. Oops, bringing my own issues in
Simple test though. Setup a client and delay 500mS each callback. Does anything bad happen?

[dastoned]

The emac task (ESP32 mac) will just malloc on frame (rather inefficientlty if you ask me) & close connection on fail.

That sounds like a disaster waiting to happen. I'd like to have a look. PeterR, would you mind pointing out the file and function where this code is implemented?

[PeterR]

emac_esp32_rx_task()

I guess a way of dealing with it is to limit any queue within emac->eth->stack_input(emac->eth, buffer, length); and block.

To be fair you have to be hammering the input to get this to fail, but it will/can & especially when the handler is slow and memory is limited.
Only seen on local loop which has higher RX rate but my processing was low. You have lower RX rate (over the wire) but much higher blocking period.
Run a quick test, would be interested.

[dastoned]

OK, this specifically is Ethernet code which I don't use. But when the heap is exhausted, it simply drops incoming packets - which seems like a reasonable strategy to me. That shouldn't cause a TCP connection to drop, unless its flow control and re-transmission are stretched to a breaking point. I guess we'd need to process packets *very* slowly for that to happen.

[PeterR]

Well, you have been warned!
It is not that the emac task will cause connection drops directly. As you say there should be retransmission etc.
The exhaustion of memory will cause others to drop the connection however, no memory left!
Say the HTTP client/server or the Websocket client/server attempt to malloc() a buffer whilst decoding the next frame.
I know that at least one drops and suspect another corrupts.
The answer seems simple though, throttle the emac tasks malloc() & (as you say) let the link layer take over.

[dastoned]

It is not that the emac task will cause connection drops directly. As you say there should be retransmission etc.
The exhaustion of memory will cause others to drop the connection however, no memory left!

Ah, right. NW gobbles up all available RAM as buffer and starves every other task. This launches a smorgasbord of nasty, unrecoverable errors. I wonder if there are similar gotchas lurking in WiFi packet handling.

Anyway, this discussion reminds me to write a high-level watchdog to reboot the system when connectivity is lost for unreasonably long periods.