You can't benchmark against humans as humans have certain flaws and machines have a separate set of flaws.
As far as image recognition, we've seen imageNet competitions where the computers outscore humans. In the real world they can sometimes spot things humans miss while at the same time totally failing on something humans can get with ease.
In writing this post I thought it might be cool to do a AI where's waldo but it appears someone has done that:
You cannot always use lane centering either. Many times humans tend to purposely hug the outside if being passed by a semi or oversized vehicle. This is a matter of comfort and preference. We might veer off center to avoid a pothole or to give a car parked on the side of the road some space. Sometimes people hug the line separating the lane they wish to merge into just before they put their blinker on (if they actually care to use one
). MobilEye's test vehicles will actually hug the line before merging just like a human.
The one true benchmark is going to eventually be the fatality rate. This is pretty much a standard measure done in many countries to benchmark both drivers and the average safety of vehicles on the road.
However, another benchmark, for which human data doesn't accurately exist, would be incidents where the vehicle collides with something else. Most of this goes unreported. It'd be nice if FSD cars acted like bumper cars with invisible bumpers, in that ideally they shouldn't run into anything and should have buffer zones of "personal space".
My benchmark is this:
Can a car get me to where I'm going safely, without incident, and comfortably so that it feels natural?
