Another user here with the same issue. Setting priority device in Bluetooth does not work either, now I know that's only for infortainment purposes.
I'm the main driver and I've got a key fob, I might try if putting the keyfob in my pocket and other half walks to passenger door as normal...