OK, it happens, but not nearly as often as being hit by a car.
Jason Wakeford, director of campaigns for road safety charity Brake, said: "The rise in the numbers of pedestrians killed or injured by cyclists is concerning but the fact remains that vehicles are responsible for 99 per cent of road user fatalities.
The number of pedestrians fatally or seriously injured in collisions with cyclists has doubled since 2006 (from the UK to match your anecdotal evidence)
I also think you overestimate the utility of a car's crumple zone to a pedestrian. However I agree that getting hit with handlebars would really suck.
A key difference is that pedestrian safety is continuing to improve in cars - the Euro NCAP safety rating now includes "AEB Pedestrian", for example. This isn't happening for bikes.
Also, I think most modern cars (incl. Tesla) do pretty well with front impact pedestrian protection - this is at 25mph: