If this is something homebrew, are you 100% certain (checked with a scope/logic analyzer) that your power is stable?
Flash is pretty power hungry. Writing it is worse than reading it, but it's still a thing. Is your power source to the chip sagging? Do you have decoupling and power caps in the places that need them? (I'm not an EE and can't tell you where that is or the appropriate values.)
It's also possible that you have a subtle locking issue. If your directory and filesystem traffic is interleaving - in ways approved by whatever your filesystem's rules are - you may be issuing back to back SPI address cycles or back-to-back SPI data cycles instead of interleaving them. A logic analyzer on that trace under load (ideally, a fraction of a second before a crash, but we all know that's harder than it sounds) may be telling. If your entire system is homebrew, that kind of thing is more likely than if you're using, say, Nuttx and are "just" doing the UI yourself. SPI transfers aren't really atomic on their own; you can sometimes start getting responses on a read while you're still clocking out addresses and you have to guard against your own code stuffing out another address cycle in between the time you've started a MOSI and the time the Peripheral has acked that cycle with the appropriate (via agreement, since CS is unidirectional and there is no ack line) number of cycles. It's all on you to not interrupt your own cycles with other cycles. If you're using SPI libraries, they''ll usually reduce the number of tears spent debugging this. (Except for the times when they INCREASE the number of tears. You'll only know which it is when you've finished.
Ah, there's another possible source of entertainment - do you have anything else sharing this SPI bus? If your display is on SPI and its updating the time seek bar and the SPI is getting involved in the filesystem's SPI bus, that can provide MORE hours of debugging joy!
It's usually worth the investment (perhaps counterintuitively) to make it crash MORE often. Debugging something that crashes every minute is WAY easier than debugging a crash that happens only once a day. It's also worth hardware instrumentation to snoop the bus cycles on the SPI and the voltage levels of everything involved.