I don't believe that's true -- at least, not by default. You can download the app, yes, and even link it to your account if you know the password. You could use this to unlock the car to get into it. There's a separate process to set up the phone as a key that requires a key card, though, so you couldn't drive the car from a freshly-configured phone without a key card. At least, that was the case when I last set up a phone as a key.
That said, it's possible that Tesla has policies and procedures to bypass these steps in special circumstances, so you might be able to do it with the help of a customer support call. I don't know if that's the case, though.