May be the I-cache still 'warming up'.rare instances of lags shortly after boot up
And/or other initialization activities (WiFi connect,...) running concurrently and hogging up the the flash's bandwidth.
You could enable the UART's loop back to send 'yourself' some dummy bytes first to have as much of the relevant code as possible loaded from flash before going to town.