As far as I know, the charge port will open without a key being present, but you cannot unlock the connector without the keyfob being around!
The strange thing is that I also have a few people having a model X where you can open the charge port when you push it, but not with the connector radio...