The slowness is “how long does it take to wake the car when you launch the app”. This can easily be a good 20-30 seconds before you can then interact with the unlock port button. So: phone out of pocket, unlock phone, find Tesla app and launch it, wait 20-30 seconds (if the car is asleep), press unlock port button. I just timed my Model 3, and it took 23 seconds to wake the car from launching the app. The fob and Tesla cable button are instant (I assume they use Bluetooth).
Certainly the first-est of first world problems!