I think you should be able to keep the ESP32 in reset (as in: make EN low) and program the flash if you have access to the SPI pins. If the module you have doesn't give you that, then well... I'd imagine all you can do is lift the can, and that's not very production-safe
Note that from my experience with higher baudrates and USB flashing, you get diminishing results in trying to make the interface faster: at some point you're spending most of the time waiting for an erase or write cycle to complete. The absolute fastest way to get your flashed chips may actually be to not flash the chips at all but to contact
sales@espressif.com: from a certain MOQ (iirc 1 reel) we offer pre-programming services, selling you modules with your own firmware already flashed into it.