Sometimes it depends which direction you approach the car from as to how well it works. when I reverse onto my drive I have to walk around the car to get in and it works great. if I drive straight on then when I come back to the car I walk straight up to the drivers door it is less reliable.
The other option is that there is something there that is interfering with the Bluetooth signal.
Do you have an android? if so you can always touch it on the drivers side B pillar.