Is that reasonable?
No.
50ms is an eternity compared to the overhead of a timer callback. Even if you
could shave off 5 micro-seconds per timer event, that'd reduce CPU load by 0.01%.
That said, the FreeRTOS timer does callbacks from the IDLE task, and the ESP HR timer uses a dedicated task to do the same thing. You can however set the ESP HR timer to run callbacks directly from the timer's ISR (or use a dedicated hardware timer with your own ISR), which would a) reduce jitter due to tasks' priority-based scheduling and b) mitigate the overhead of task switching to and from the timer task. Point a) may be valid if you require low jitter, point b) becomes meaningless when talking about milli-seconds, or when using the IDLE task to execute callbacks as FreeRTOS does.