What you're saying is exactly what people said about self-driving cars, and still saying. God forbid cars driving themselves! But here we're, every carmaker will have self-driving capabilities in the next 20 years.
The whole point of HUD is that you don't have to press buttons and you don't have to look away. You keep your eyes out front where it's the safest. LOL. Of course, I'm not advocating displaying anything that's distracting. Like every good feature, it needs to be carefully thought out, but I think technology will inevitably lead us that way, just a matter of time.