Most likely it's the phone. Pixel 3 and 3XL seems to work flawlessly from what I've seen anecdotally here on the forums. @SDKoala and I tried to work through some of the Pixel's issues together. He recently moved on to a Pixel 3 and things just work. No changes on the car side. My theory is that the car is having a hard time "hearing" the phone's broadcasts, so it's hit or miss. If you're standing at the wrong corner of the car, or you're turned the wrong way, or you're wearing jeans, etc. the car can't tell how far away you are. But if you see that the phone is BT-connected, hitting the unlock button always works.
There are also edge cases where something about the phone-car connection gets corrupted, and then nothing works until you toggle BT or airplane mode, but I find that error condition to be quite rare these days. It's just proximity woes.
Just want to confirm that my Pixel 3 still works all the time. One failure in the past 6-7 months randomly that has not been reproducible.