Even if different corners are encountering different conditions, its more or less "+/-" from some average.
But if you consider that various tires have varying abilities when comparing brands or models of tires, then you might have a "compounded" problem where the mismatched tire is normally a little less effective on one side with "equal" environmental conditions, now simply has even worse conditions on top. Sure, you could potentially have a situation where the "better" tire is the one experiencing the worse condition, supposedly "balancing out."
So, by keeping the tires as similar as possible, you minimize the chances of it going from "meh, a little iffy" to "well, now I need to change my pants."
Sure, a car doesn't know the tires, its just one type of indicator of potential problems. Having mismatched tires is a potential source of problem (the car won't be able to warn about), that doesn't need to be made.
You say yours drove completely fine... did you have to panic brake for any reason? I mean, if you're just driving "normally," sure, mismatched tires won't mean much. It's when you NEED to do something, like slow down really fast cause the guy in front of you wasn't paying attention and slammed into the car ahead of them that had slowed down for increased traffic or something broken down in the lane. You couldn't really see directly in front of that guy and other lanes seemed to still be moving pretty quick.
Because you've driven your car for so long and had a set of tires for a while, you felt relatively confident you could stop in time. But forgot about the new mismatched tire, that triggered ABS when it normally wouldn't and now it takes a few extra feet to stop... just enough tap that guy that crashed. And a string of possible things could happen afterwards... but all the while you might wonder if it could've been avoided if the tires had more closely matched.