That is totally understandable. Two things here that can avoid this dangerous situation:
1. Let the car move with the speed that was set to in AP (65, 70 mph etc..) without interfering with posted speed and this should be controlled by the driver not posted speed in maps
2. There should be a setting in AP that can give us options a) you want to follow the speed as per maps b) manually control the speed in AP.
True but manually setting a speed like that would negate the very important lowering speed when going around a bendy offramp. Where if It didn't automatically change the speed you might slide right off the road.