What's everyone using for database or flash storage?

[tpbedford]

I'm after some high-performance storage options. I've tried the Sqlite ESP-IDF port on FatFS and LittleFS, but both are pretty slow. I've looked at LittleDB but there's a lack of error checking and it just doesn't feel robust.

We're using an external SPI flash chip, so I'd like something that works with 4k erase sectors (same as the internal ESP32-WROOM SPI flash). We're able to run daily scheduled tasks to truly erase "deleted" flash sectors, so these DON'T need to be erased in real-time - I don't want to be blocking on flash sector erase during normal operation.

Any suggestions?

Constraints:
- I'll have a few DB "tables" dedicated to the ESP32 host
- I'll have up to 100 "clients" attached, each having their own set of tables so I'm happy to have a folder for each client (each client sends data to the host over wifi, and the client list changes infrequently)
- generally I'm just logging, so something that appends fast is ideal
- generally I don't need random-access reads, especially not of the large tables.

So structurally a no-sql or KV-pair schema is OK, so long as I create a folder per-client. Some tables for each client will get quite large, but many will be quite small (several being just single records).