I also have the exact issue on my brand new Model X and Samsung Fold 4. I am not able to Set Up Phone Key. It fails and states its unable to connect.
However, I was able to Set Up Phone Key on my wife's Samsung Galaxy S22 Ultra (Both are running Android 13). So it may be a Samsung Fold 4...