Rather than keeping the key card in my wallet, I keep my key card in the phone case under my iPhone. Just in case
Surely that's the worst place for it to be?
If the phone battery has died, in the car, while you're outside (as per OP) you still can't get to the keycard
If you lose your phone, you've also lost your keycard
If someone steals your phone, they've also stolen your keycard
Having the keycard somewhere different to your phonekey is the logical way to reduce the risk of being caught with no way in.