Double buffering is not faster, what it does is prevent tearing. If you only have a single buffer, the screen can refresh while LVGL is writing to it, leading to strange artifacts as the screen shows half-before and half-after images for one frame. Double buffering prevents that.
Sorry, I have no idea how to get 120MHz to work in idf <5.1.