I drive the same way, extremely cautious, I do not let the car "learn" or whatever one wants to call it; it can get sooo close to curbs/other vehicles/garbage cans etc, I just can't risk it. And don't even think about construction zones with workers.
The driver is 100% to blame if there is a mishap of any kind, and Tesla makes that very clear in their disclaimers. I would be quite angry with myself if I even scratched a wheel cutting corners too close while in AP. I don't blame the guy for taking over immediately when he senses the car is not going to do what he would do in that situation.