@Gellie did you associate your key cards/phone keys to profiles? Maybe you missed this step.
For me, walking up to the car, if my partner last drove, the seat will be in her configuration, but as soon as I pull the door handle with my phone on me, the seat starts moving to mine--and vice versa. The only time this can get "confusing" is when we are both present (with our phones) and in close proximity to the car. Occasionally it guesses wrong and we have to manually change the active profile on the screen.
EDIT: have a look at page 45 of the owner's manual
https://www.tesla.com/sites/default/files/model_y_owners_manual_north_america_en.pdf, section titled Linking a Driver Profile to a Key