To be honest, the ESP32 is so incredible compared to every other chip I've used before at that I find it hard to think of improvements!
Adding hardware USB 2.0 would be nice so that I can avoid adding a USB converter chip and I imagine potentially higher speeds, so long as it's still simple to upload code over USB. I'd also suggest using a switching regulator on your development boards since this chip is frequently targeted at low power applications, or provide a way to give the board 3.3V directly and bypass the linear regulator, but that's not really an ESP32 complaint.
If you're asking for a ridiculous wish list, add a FPGA to your modules so that we can add custom peripherals
Here's one of the few things that I simply cannot do efficiently right now. I often build equipment that requires high resolution data logging. A chip that I want to be able to use is the AD7768 which is an 8-channel, simultaneous sampling, 24-bit ADC that can be daisy chained to give an arbitrary number of ADC channels at 256kSPS. Problem is, to get data out that fast you're basically required to use octal-SPI.
I just barely got this to work by dropping the sampling frequency (to unacceptable speeds, but at least it worked), monitoring the CS line with an interrupt, and then using polling to read each of the eight data lines, but it's simply not fast enough to do that way. I tried just using the clock edge for interrupts, but the interrupt call time is too long to reliably read the data pins.
It would be amazing if there were a way to do arbitrary-width SPI slave in hardware to let me read a full byte per clock pulse. Of course, an integrated FPGA in your module would take care of that
I'd personally also find it a little nice to have a higher resolution DAC, 2x 10-bit DAC outputs seems to be common and 1024 steps is a lot nicer looking in a synthesized sine wave than 256 steps. I wouldn't prioritize it, but 8-bit DAC is a bit low in 2018.
I'd also add a formal ground pin to the external pins on your SIP package like the ESP32-PICO-D4 so that we actually don't have to solder the pad under the chip. The datasheet is actually very misleading on this one, but it was so hard to submit a note to espressif that I gave up. The datasheet for ESP32-PICO-D4 says that the pad doesn't need to be soldered for thermal performance, but it's also literally the only ground pin on the part so it obviously has to be. It would be much easier to solder by hand if I could rely on an outside pin for the ground connection and just leave the thermal pad touching the ground copper but not soldered.
I'm definitely planning to standardize on the ESP32-PICO-D4 or it's spiritual successor with the next ESP chipset, it's a beautiful product. My use case would be to connect the antenna through a u.Fl connector since I always use metal enclosures, it would be really incredible if the PICO had an integrated u.Fl connector somehow. I can't imagine how you'd do it though. I'm currently figuring to just use a super short trace to a u.Fl connector right next to it to try to avoid needing to build an impedance controlled board, which will hopefully work okay on a 2-layer PCB.