The only drawback would be (lack of) convenience. You may eventually find that it's a little bit of a hassle to mount and plug in every time you get in the car, so you don't always mount the phone/camera. One of those times will be when you really want it. You'll also miss out on the parking mode of having the camera potentially record while you are away from the car.
This is very similar to backups on your computer. If they don't happen for you automatically, eventually, you're going to reduce the frequency of backups, and potentially give up on them all together.