The problem is, what could happen between the time those Tokens are changed. That Is the risk one has to assume and an individual has no control over, no matter how small the possibility may be. Perhaps loosing one's FOB has more risk associated with it. IDK. There are just too many variables IMHO to speak with absolutes.
Me? I've vaselated on what makes sense for me. I've tried TeslaFi and its promising, the optional daily part of EVTripping is very helpful for what I tend to care most about, and others... But each time, I've gone through the process of disabling the functionality on all the tools, change my Tesla.com password (14 chars, upper/lower alpha with numeric and special character), store that in 1Password, then change the passwords on the Tesla App on both my iPhone and iPad. It's not a trivial process, but as you suggest is do-able. RIght now, it's only my official Tesla App and Remote S for me. I'd love to have the data some of the other tools provide, but just have not gotten to a comfort level to use anything else that "owns" a copy of my all-powerful Token for some period of time. ...but to each their own. It's great we have options to suit each of our needs and risk tolerance levels.