I was in contact with Google Pixel support and they are aware of the issue. Apparently it will be fixed in the next update... No ETA.
However, they gave me a workaround that seems to work each time for me. Reboot the device. As soon as I reboot, the phone key works for another 24 hours or so.