Autonomous Car Progress

[diplomat33]

Ford is using Mobileye. This seems similar to SuperCruise.

Yes, it is basically Ford's version of supercruise. The only real difference that I can see is they use differing alerts, icons and colors that they think are more friendly for color bind people.

 

[mark95476]

Intel bought Mobileye a while back. I don't think their investment has paid off. I don't even see them getting much press as a technical leader and showing off their system.

Intel has a new CEO, one who has a technical background, and the marketing hype or mishmash might be running it's course.

It's very clear to me that Mobileye is a sort of Nikola-esque company / approach right now. There are many logical disconnects for critical thinkers. They're not making meaningful company agreements. There's a lot of marketing jargon that have little underlying engineering sense.

 

[scottf200]

The system only works on divided highways that Ford has approved the system for. Right now, the system works on about 100,000 miles of divided highways in US and Canada. The system is not designed for city streets. It is only for highway driving.

This test of a Mach-e recently with 'co-pilot' was dreadful.
Start 1:23 into it (83 seconds).

 

[diplomat33]

This test of a Mach-e recently with 'co-pilot' was dreadful.
Start 1:23 into it (83 seconds).

To be clear, "co-pilot" is different from "bluecruise". They are different systems. The previous article was about hands-free highway was about "bluecruise", not "co-pilot".

 

[Tam]

How do lidar/radar only systems read stoplights?

Right. So how can there be any claim for an independent LIDAR/RADAR system? Isn't that what Shashua is proposing?

He first believes in the possibility of vision driving only while the industry was laughing at that idea.

However, he is a realist and believes that after he figured out the vision challenge, he would add additional sensors such as LIDAR + RADAR as he is doing now.

That's because vision still has some limitations.

But his method of additions of sensors is different from the rest of the industry.

Each Vision and LIDAR + RADAR has its own independent system and each system has its own limitations (the camera thinks it's a white sky in 2016 Autopilot V Semi-truck and the LIDAR would have no problem in confirming that it's not a white sky, it's a big giant white truck and it would override the camera to initiate braking).

So, "independent LIDAR/RADAR system? Isn't that what Shashua is proposing?" is yes, but in the context that there's still another INDEPENDENT camera system to intervene when there's a limitation for LIDAR such as failure to recognize the changing of color of traffic lights.

 

[Backup Account]

He first believes in the possibility of vision driving only while the industry was laughing at that idea.

However, he is a realist and believes that after he figured out the vision challenge, he would add additional sensors such as LIDAR + RADAR as he is doing now.

That's because vision still has some limitations.

But his method of additions of sensors is different from the rest of the industry.

Each Vision and LIDAR + RADAR has its own independent system and each system has its own limitations (the camera thinks it's a white sky in 2016 Autopilot V Semi-truck and the LIDAR would have no problem in confirming that it's not a white sky, it's a big giant white truck and it would override the camera to initiate braking).

So, "independent LIDAR/RADAR system? Isn't that what Shashua is proposing?" is yes, but in the context that there's still another INDEPENDENT camera system to intervene when there's a limitation for LIDAR such as failure to recognize the changing of color of traffic lights.

Well first off as context I work in areas of both machine learning and signal processing / sensor fusion so I fully understand the general ideas. So maybe he is just trying to dumb it down for explanation purposes, but saying there is an independent LIDAR / RADAR model when it cannot (ever?) possible perform certain required functions means it is not independent. But maybe I'm wrong and certain LIDARs can?

Otherwise, he is just talking about sensor fusion, where perhaps the point in the overall architecture where the information is "fused" is different than with other AV companies.

It's basically a hedge so he can say "we can do a camera-only system if we want" whereas other AV players are seemingly "locked" into depending on LIDAR

 

[powertoold]

Well first off as context I work in areas of both machine learning and signal processing / sensor fusion so I fully understand the general ideas. So maybe he is just trying to dumb it down for explanation purposes, but saying there is an independent LIDAR / RADAR model when it cannot (ever?) possible perform certain required functions means it is not independent. But maybe I'm wrong and certain LIDARs can?

Otherwise, he is just talking about sensor fusion, where perhaps the point in the overall architecture where the information is "fused" is different than with other AV companies.

It's basically a hedge so he can say "we can do a camera-only system if we want" whereas other AV players are seemingly "locked" into depending on LIDAR

Mobileye's system is sensor fusion but marketed as some kind of "different" approach. The lidar/radar subsystem is still dependent on the vision-derived REM maps and also vision-based traffic light recognition.

Also, I have no idea how they're going to reliably localize the car with lidar based on a vision-derived REM map.

There are so many holes in this approach, I don't even know where to start.

 

[Tam]

...saying there is an independent LIDAR / RADAR model when it cannot (ever?) possible perform certain required functions means it is not independent. But maybe I'm wrong and certain LIDARs can?...

LIDAR as an L3 can work independently in most scenarios including stop signs but NOT at active traffic lights. That's when the L3 LIDAR needs to give humans 45 seconds warning to take over and handle the traffic lights.

Thus, in rural roads with lots of intersections and stop signs but no traffic lights, LIDAR can independently work as if it's an L4 in this scenario--driverless in specific limitations such as no stormy/snowy weather, Waymo's 50 square mile Chandler, Az, and this scenario where no traffic lights around....

Same with a Camera system that works independently in most scenarios but not in some poorly lit or overlit scenarios such as in Florida 2016 autopilot incident.

Working independently doesn't mean it doesn't have faults. They can derive faulty conclusions (white sky instead of white semi-truck) but they can be faulty independently.

 

[Dan D.]

LIDAR as an L3 can work independently in most scenarios including stop signs but NOT at active traffic lights. That's when the L3 LIDAR needs to give humans 45 seconds warning to take over and handle the traffic lights.

Thus, in rural roads with lots of intersections and stop signs but no traffic lights, LIDAR can independently work as if it's an L4...

To be truly redundant a Lidar/Radar system has to have some ability to sense light signals. There's just too much vital information: traffic signals, turn signals, emergency vehicles, crosswalks. Even on the highway or rural roads you will encounter these. Maybe they have an independent camera system, maybe not (since they stressed that there are no cameras). Perhaps they just mean redundancy in road sensing but for light signals they rely on the cameras only.

 

[diplomat33]

Mobileye's system is sensor fusion but marketed as some kind of "different" approach. The lidar/radar subsystem is still dependent on the vision-derived REM maps and also vision-based traffic light recognition.

Also, I have no idea how they're going to reliably localize the car with lidar based on a vision-derived REM map.

There are so many holes in this approach, I don't even know where to start.

The REM map is just created with camera vision. It is still compatible with lidar. The lidar can localize itself on it, just like any other HD map. I am not really seeing the problem there.

And the charts are pretty clear IMO:

If you compare both charts, you can see a big difference between the two approaches. Traditional sensor fusion just feeds all the sensors into a single planning model and outputs controls to the car. Mobileye's "true redundancy" basically feeds the sensors into separate planning models and them compares them to output vehicle controls.

The "world model" from camera would be the REM map + all vision data.

The "combined world model" would be the REM map + all sensor data

The "world model" from lidar/radar would be the REM map + radar and lidar data

The RSS from the camera based "world model" would do planning based on the camera "world model".

The RSS from the radar/lidar based "world model" would do planning based on the radar/lidar "world model".

The Policy would be in charge or reconciling the different planning models and outputting vehicle control.

The disadvantage with traditional sensor fusion is that any mistake in camera, lidar or radar can affect the whole thing since all the sensors feed into one single planning model directing vehicle controls.

The advantage of Mobileye's approach is that a mistake in one sensor will only affect one planning model, not the whole thing. So, the car will only fail if both planning models make the same mistake, which is less likely than just one planning model making a mistake.

For example, take the example of the Tesla that hit the semi truck a few years back. Camera vision sees the white side of the semi and is not sure what it is, maybe it has a confidence level of 40% that there is a truck. So the RSS from the camera based "world model" says to keep driving. The radar/lidar does detect the semi truck with 100% confidence level. The RSS from the radar/lidar "world model" says to brake to avoid a collision. The Policy looks at the 40% confidence from the camera vision and the 100% confidence from the lidar/radar and concludes there is very likely a truck there and brakes and avoids the collision.

 

[diplomat33]

Perhaps they just mean redundancy in road sensing but for light signals they rely on the cameras only.

Yes, I think so.

 

[Tam]

...he is just talking about sensor fusion, where perhaps the point in the overall architecture where the information is "fused" is different than with other AV companies...

I think that's it. At which point the info from sensors is fused.

Commonly, the info from sensors are fused and decided at an early step such as the 2016 Florida scenaro.

1) Camera sees white sky--crossed check with radar that sees speed of zero

2) The 2 info are fused.

3) The decision is it's safe to continue with no brake deployment

MobilEye involves many steps:

1) Camera sees white sky

2) the info is processed with the World Model. It could be caught that in another previous sample, a white sky might be a white semi-truck but let's say the mistake was not caught here.

3) It then is processed with the Responsibilty-Sensitivity Safety algorithm: The error could have been caught here also because it would compare is it responsible to decide in an overlit condition. Let's say the error was not caught: It's a white sky so it's totally responsible to continue to drive.

4) It then goes to the Policy algorithm where it competes with RADAR


In the meantime, the RADAR also goes through the above steps:

1) RADAR sees a zero speed object and just like any others with that speed, it's to be ignored.

2) the info is processed with the World Model

3) It then is processed with the Responsibilty-Sensitivity Safety algorithm. Although an object at the speed of zero is to be ignored but it realizes that Road Experience Management indicates that the coordinates point to an intersection on that highway. Yes, normally, it has the right of way. Yes, normally, it's safe to drive but it might be prudent to initiate brakes.

4) It then goes to the Policy algorithm where it competes with Camera

At Policy algorithm, it could review the rationales for no-brakes from camera and yes-brakes from RADAR's Responsibilty-Sensitivity Safety algorithm and goes with RADAR's Responsibilty-Sensitivity Safety algorithm as a conservative approach.

5) Vehicle control: Brakes deployed

So, common sensor fusion takes fewer steps while MobilEye's method takes multiple separate pathways with independent decision-makers before letting the common decision-maker make the final call.

 

[scottf200]

To be clear, "co-pilot" is different from "bluecruise". They are different systems. The previous article was about hands-free highway was about "bluecruise", not "co-pilot".

Certainly and I explicitly put quotes around 'co-pilot' for the current owners video vs future 2021 software update. I do wonder how many Ford customers will be confused about this marketing-speak.

 

[diplomat33]

On a side note, I do think "Bluecruise" is a silly name. But "Supercruise" was already taken so...

 

[JHCCAZ]

Certainly and I explicitly put quotes around 'co-pilot' for the current owners video vs future 2021 software update. I do wonder how many Ford customers will be confused about this marketing-speak.

Yes, not like Tesla names AutoPilot and Full Self Driving that are perfectly unambiguous

 

[Bladerskb]

Right. So how can there be any claim for an independent LIDAR/RADAR system? Isn't that what Shashua is proposing?

yes, its not 100% independent. There's hundreds/thousands of modes and almost all are independent.

Completely independent would give you a complete dot product. If your MTBF was 10,000 hours for each system. 10,000 hours * 10,000 hours = 100 million hours.

Since its not completely independent, its lower. If you end up somewhere around 10 million hours that's alot better than 10,000 hours.

 

[Bladerskb]

Well first off as context I work in areas of both machine learning and signal processing / sensor fusion so I fully understand the general ideas. So maybe he is just trying to dumb it down for explanation purposes, but saying there is an independent LIDAR / RADAR model when it cannot (ever?) possible perform certain required functions means it is not independent. But maybe I'm wrong and certain LIDARs can?

Otherwise, he is just talking about sensor fusion, where perhaps the point in the overall architecture where the information is "fused" is different than with other AV companies.

It's basically a hedge so he can say "we can do a camera-only system if we want" whereas other AV players are seemingly "locked" into depending on LIDAR

Modern high resolution lidars can see and detect everything a camera can other than the status of traffic light.
Its quite simple. Lidar, Cameras, Radars has different fail modes and different optimal operational modes.

Lidar can work in pitch darkness and in direct sun light, gives you direct 3d dimensions of objects and their range but is sub-optimal in adverse weather (heavy rain, heavy snow, fog, mist)

Camera has the best resolution and is great for semantics but fails in direct sun light, low light conditions and is sub-optimal in adverse weather

Radar doesn't really have a fail mode and can see in heavy rain, heavy snow, fog, mist, direct sunlight, low light conditions.
The only problem being resolution which imaging radars solve.

Here are the failure modes:

Lidars

• Adverse weather
• Object Semantics

Camera

• Adverse weather
• Direct Sunlight
• Low light condition
• Object detection and range accuracy

Radar

• Object Semantics

Lets take a pedestrian at night wearing all black with no street lights, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar see the peds and classifies it as a ped and assigns it distance and velocity.
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Lets take a pedestrian at night wearing all black with no street lights in adverse weather, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar fails to see the peds
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Because the fail modes are different, all three sensors missing that one person in that instance is so improbable it will essentially never happen or since we are talking in probabilistic term. The probabilities will be very very very low.

 

[Dan D.]

On a side note, I do think "Bluecruise" is a silly name. But "Supercruise" was already taken so...

Yes, not like Tesla names AutoPilot and Full Self Driving that are perfectly unambiguous

The names may be confusing, hopefully the systems are functional, clear to use, and safe.

To me, the hands-free Level 2 looks like a recipe for falling asleep, I like Tesla's hand-monitoring as it makes you do something. With the Ford scheme you have to watch the road and can otherwise remain motionless.

I get that there is eye monitoring, and you can perhaps fiddle with the touchscreen (& phone) more than with the Tesla system, and that if your attention becomes unsafe the car is going to beep/slow/stop.

But this guy is going to be drowsy very quickly.

 

[diplomat33]

Modern high resolution lidars can see and detect everything a camera can other than the status of traffic light.
Its quite simple. Lidar, Cameras, Radars has different fail modes and different optimal operational modes.

Lidar can work in pitch darkness and in direct sun light, gives you direct 3d dimensions of objects and their range but is sub-optimal in adverse weather (heavy rain, heavy snow, fog, mist)

Camera has the best resolution and is great for semantics but fails in direct sun light, low light conditions and is sub-optimal in adverse weather

Radar doesn't really have a fail mode and can see in heavy rain, heavy snow, fog, mist, direct sunlight, low light conditions.
The only problem being resolution which imaging radars solve.

Here are the failure modes:

Lidars

• Adverse weather
• Object Semantics

Camera

• Adverse weather
• Direct Sunlight
• Low light condition
• Object detection and range accuracy

Radar

• Object Semantics

Lets take a pedestrian at night wearing all black with no street lights, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar see the peds and classifies it as a ped and assigns it distance and velocity.
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Lets take a pedestrian at night wearing all black with no street lights in adverse weather, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar fails to see the peds
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Because the fail modes are different, all three sensors missing that one person in that instance is so improbable it will essentially never happen or since we are talking in probabilistic term. The probabilities will be very very very low.

Thanks. This is a good explanation for why AV companies use different sensors.

I think it also explains why Tesla's "camera-only, not even radar" approach is a mistake. Yes, Tesla's FSD may end up working well in conditions where cameras are optimal but it will most likely fail in conditions where cameras are suboptimal. For this reason, Tesla will need to keep driver supervision. I just don't see Tesla being able to remove the driver supervision requirement other than maybe limited ODD where cameras are optimal.

 

[Backup Account]

Modern high resolution lidars can see and detect everything a camera can other than the status of traffic light.
Its quite simple. Lidar, Cameras, Radars has different fail modes and different optimal operational modes.

Lidar can work in pitch darkness and in direct sun light, gives you direct 3d dimensions of objects and their range but is sub-optimal in adverse weather (heavy rain, heavy snow, fog, mist)

Camera has the best resolution and is great for semantics but fails in direct sun light, low light conditions and is sub-optimal in adverse weather

Radar doesn't really have a fail mode and can see in heavy rain, heavy snow, fog, mist, direct sunlight, low light conditions.
The only problem being resolution which imaging radars solve.

Here are the failure modes:

Lidars

• Adverse weather
• Object Semantics

Camera

• Adverse weather
• Direct Sunlight
• Low light condition
• Object detection and range accuracy

Radar

• Object Semantics

Lets take a pedestrian at night wearing all black with no street lights, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar see the peds and classifies it as a ped and assigns it distance and velocity.
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Lets take a pedestrian at night wearing all black with no street lights in adverse weather, you need to detect not only the pedestrian but their distance (range) and velocity.

Camera fails to see the peds
Lidar fails to see the peds
Imaging Radar sees the peds and classifies it as a ped and assigns it distance and velocity.

Driving Policy navigates around the pedestrian

Because the fail modes are different, all three sensors missing that one person in that instance is so improbable it will essentially never happen or since we are talking in probabilistic term. The probabilities will be very very very low.

Yes this describes sensor fusion.

I'm just trying to understand Shashua's attempt to make Mobileye's approach distinct vs Waymo / Cruise's approach to sensor fusion.