Would make a lot of sense to have a phone be able to be associated with a drive profile...not sure why Tesla haven't looked at it from this angle.
That way whoever gets in the car with whatever fob or not (S, 3, X and Y) in whatever order does not matter...what matters is Phone A goes with Driver Profile A and if Drive A gets in the car and this is the selected driver profile then Phone A is the prority.