SPI version is approximately 10x quicker than I2C, but needs an extra chip select line.

Have you tried running the MCP23017 faster than 400KHz?

I recently did some testing with both the I2C MCP23017 and the SPI MCP23S17, but with the esp8266 Lolin v3 board. Reading a 8 bit port, 128*2(256 bytes) line size at 16-bits per pixel ... (SQCIF - https://en.wikipedia.org/wiki/Common_Intermediate_Format) MCP23017 (700KHz) => 19+ ms MCP23S17 (10MHz) ...