Burt isnt it more complex than that? You really have three categories: (a) "pure" comfort, where the passengers dont feel unsafe, they just feel jerked around, (b) "imagined" safety issues, when the car is executing a technically safe maneuver that would make a human uneasy (e.g. squeezing through a narrow gap where a human would be anxious), (c) "real" safety issues, when the car attempts something that is or could be dangerous.
I think there is overlap between all these categories, but the overlap between (b) and (c) is problematic, since it undermines confidence it the ADAS system (examples like driving closer to the car in front). In effect, the ADAS needs to be "de-tuned" so that it simulates a human driver even in circumstances when it could do better. Turning left into oncoming traffic comes to mind. The car can do the math to figure out it has plenty of time (and be right), but scare the passengers (and oncoming traffic) half to death since humans need a wider margin of safety.
Ultimately of course you end up with something like this: