Basically same experience for me.. I love that we universally hover our right foot over the accelerator to override the phantom events. But mostly don't even bother using it. Also want to point out that coming from 2 years and 32K miles driven on Model 3, Vision is the worst. Many years later to have it feel worse than a 2018 M3 auto pilot is quite insane, but for me, true.
It also sounds strange, but mine is deathly afraid of large white objects and I usually get ready to takeover when I see them coming.