Tesla Model 3 will have the new Autopilot 2.0 with dual cameras

[Lucy Lee]

Tesla model X that were recently delivered already has the Dual Facing Camera built in.

Elon will announce a newer / better Autopilot 2.0 very soon and our Tesla Model 3s will have this built in!

 

[SΞXY P100D]

Much sooner than expected! I thought it would take another year. Any pics of the dual camera system, and will it include enhancements recently tweeted by Elon Musk in response to the fatal AP crash?

 

[S'toon]

Tesla Model X delivered with dual front-facing camera housing hinting at Autopilot 2.0

 

[jaguar36]

The model X has always had this housing. It doesn't have dual cameras, just the housing. Obviously AP 2.0 is coming soon, but there is no evidence as to if thats tomorrow, next month or a year from now.

While the Model 3 will obviously have some sort of AP there is no evidence at all that it will be the same as the S/X current hardware, the 2.0 hardware or something else entirely.

 

[EaglesPDX]

I wonder how Tesla was getting stereoscopic camera view for distance prior to dual cameras? Subaru's Eyesight is dual cameras and that is how it tracks vehicles front for distance and speed with just one camera?

 

[Snow Drift]

I wonder how Tesla was getting stereoscopic camera view for distance prior to dual cameras? Subaru's Eyesight is dual cameras and that is how it tracks vehicles front for distance and speed with just one camera?

Subaru makes the best system thanks to dual cameras to see depth. They were as of 2014 the only OEM to pass all of the IIHS crash avoidance tests.

MBLY believes they have a superior tech that can work off one camera. Clearly it is not better.

 

[EaglesPDX]

Subaru makes the best system thanks to dual cameras to see depth. They were as of 2014 the only OEM to pass all of the IIHS crash avoidance tests. MBLY believes they have a superior tech that can work off one camera. Clearly it is not better.

Hopefully the T3 will have the stereoscopic cameras. Love the Eyesight system on the Subaru. Very smart system. An example if someone cuts in front of you on the freeway, the system knows how close they are and if they accelerating away from you or slowing in front of you and reacts accordingly. The blind side indicators likewise use the approaching car's speed to warn you. It's lane keeping requires active driver input. It activates if you start to drift out of your lane so you can't just turn it on watch Harry Potter movies, you have to drive the car which will help you not make mistakes.

 

[stopcrazypp]

I wonder how Tesla was getting stereoscopic camera view for distance prior to dual cameras? Subaru's Eyesight is dual cameras and that is how it tracks vehicles front for distance and speed with just one camera?

Tesla (or rather the Mobileye chip) does so with visual processing. You can get a visual estimate of distance simply from knowing the height of the camera and the position of the bottom of the vehicle relative to the video frame. This is further confirmed by the radar data. Details here:
http://www.mobileye.com/wp-content/uploads/2011/09/VisiobBasedACC.pdf

Here's a link that talks about the Mobileye system in general:
Exclusive: The Tesla AutoPilot - An In-Depth Look At The Technology Behind the Engineering Marvel

Other's have also speculated depth from motion is used (where different frames are compared and you can get an estimate of depth by using a stationary visual reference, camera fps, and your vehicle speed; similar to how people do the 2 or 3 second rule for following distance).

Subaru makes the best system thanks to dual cameras to see depth. They were as of 2014 the only OEM to pass all of the IIHS crash avoidance tests.

MBLY believes they have a superior tech that can work off one camera. Clearly it is not better.

You do not need a dual camera to see depth, especially at the distances for vehicles. There are far more monocular visual depth cues than there are binocular ones.
Depth perception - Wikipedia, the free encyclopedia

Having dual cameras gives you more accuracy than is possible with a single camera, but it isn't strictly required.

 

[Cosmacelf]

I'll second what stop crazy app (damn spell check) said. Dual cameras are NOT needed for depth perception. In particular, two cameras spaced a few inches apart can't discern depth stereoscopically past 20-30 feet anyways IIRC. Instead you use motion, scene understanding algorithms, and radar to come up with a fused understanding of where everything is. The second camera is much more likely to be used as a short range wide angle view camera.

 

[EaglesPDX]

I'll second what stop crazy app (damn spell check) said. Dual cameras are NOT needed for depth perception. In particular, two cameras spaced a few inches apart can't discern depth stereoscopically past 20-30 feet anyways IIRC. Instead you use motion, scene understanding algorithms, and radar to come up with a fused understanding of where everything is. The second camera is much more likely to be used as a short range wide angle view camera.

Subaru's top rated (iihs.org) system which uses stereoscopic cameras seems to be a proven better, simpler system. That Tesla is going to stereoscopic cameras would confirm it. As noted above, in practice, it is a very sophisticated and effective system. In practice the range seems to be close to 300 feet when it first detects objects and gives audible warning. It will then pull within range (four different following distance settings) and start to slow.

 

[stopcrazypp]

Subaru's top rated (iihs.org) system which uses stereoscopic cameras seems to be a proven better, simpler system. That Tesla is going to stereoscopic cameras would confirm it. As noted above, in practice, it is a very sophisticated and effective system.

The Subaru got the top rating because it was the only vehicle that came to a complete stop in a 25mph test collision avoidance test (I should note IIHS did not test any Tesla vehicles yet, so we don't know how it compares in the same test). It is not necessarily because of the superiority of stereoscopic cameras (the logic and rear end collision risk that an automaker programmed into the system can have a lot to do with it).

Also, I should dispel the misconception that the two cameras that Tesla might release has to do with a stereoscopic system.

Just to make people aware. Subaru's Eyesight system uses two cameras mounted 14 inches apart and has no front radar backup to confirm distance to objects in the front (the latest vehicles have side and rear radar for other systems, but still no front radar). Stereo systems rely on something called disparity mapping to generate a 3D depth map. It can't function properly even if only one of the cameras are blocked or misaligned.
Embedded Vision: Detect Pedestrians | EE Times

See how far the cameras are mounted from each other in the Subaru system:

Subaru Forester review: The best small SUV thanks to EyeSight | ExtremeTech

The Model X housing however has a camera housing that allows a distance of only around 3 inches apart for two cameras (I did a rough estimate using the Model X photo on the Tesla website):

This is presumably to support the Mobileye EyeQ4 system (or something similar given recent announcement of the end of the partnership) which despite supporting up to three cameras, does not use the cameras for stereoscopic depth sensing purposes. Such a short distance between the cameras would make stereoscopic based depth sensing poor esp. for the long range.

Rather the multiple cameras is used to support different field of views, suitable for objects/pedestrians both in normal (50 degree as in the current single camera), close (pedestrians in the city), and far range (past the 100-200 meters of today's system). Here's an article that describes this and an illustration that shows how the three different cameras would be used:

The Linley Group

In practice the range seems to be close to 300 feet when it first detects objects and gives audible warning. It will then pull within range (four different following distance settings) and start to slow.

Even the oldest version of the single camera Mobileye system can do vehicle detection out to 70 meters (230 feet) and 100 meters (328 feet) for version 2 using a cheap VGA camera. If using a higher resolution camera, that extends to 200 meters (656 feet). Tesla currently uses version 3 (which I presume may have even longer range).
Vehicle Detection - Mobileye

 

[EaglesPDX]

The Subaru got the top rating because it was the only vehicle that came to a complete stop in a 25mph test collision avoidance test It is not necessarily because of the superiority of stereoscopic cameras (the logic and rear end collision risk that an automaker programmed into the system can have a lot to do with it).

In the Subaru's case it is necessarily due to the stereoscopic cameras since they provide the depth perception that activates the braking system to stop. It likely was not only the cost efficiency but the accuracy that appealed to Subaru.

Subaru's Eyesight system uses two cameras mounted 14 inches apart and has no front radar backup to confirm distance to objects in the front (the latest vehicles have side and rear radar for other systems, but still no front radar).

Nature uses the same system. Human eyes are typically just 2-3 inches apart and provide excellent stereoscopic views and depth perception.

The Model X housing however has a camera housing that allows a distance of only around 3 inches apart for two cameras (I did a rough estimate using the Model X photo on the Tesla website):

More than the average driver that the system is trying to emulate.

Using the stereoscopic cameras eliminates the need for the forward radar system and would provide a cost savings and a system upgrade at the same time. Something that would be appealing to Tesla with the T3 being Tesla's "economy car".

 

[stopcrazypp]

In the Subaru's case it is necessarily due to the stereoscopic cameras since they provide the depth perception that activates the braking system to stop. It likely was not only the cost efficiency but the accuracy that appealed to Subaru.

This is completely unfounded speculation. An automatic emergency braking system involves a lot more than simply depth perception. The available processing power and algorithm (which effects processing time), allowed speed differential (maximum difference between vehicle speeds where system activates, this is 20mph for the Eyesight system, only 9mph for the Volvo system), minimum and maximum activation speeds, maximum brake force applied by the system, maximum speed difference the system allows after activation, just to name a few I thought of. All this will affect whether a system would be able to pass that test.

Nature uses the same system. Human eyes are typically just 2-3 inches apart and provide excellent stereoscopic views and depth perception.

More than the average driver that the system is trying to emulate.

That ignores the distances we are discussing. Horizontal disparity (which a stereo system relies on to do depth sensing) is directly related to the distance between two eyes (or two cameras in this case).
Stereoscopic perception of real depths at large distances | JOV | ARVO Journals

So if a 14 inch distance in the Subaru system yields 80 m (262 ft) of range, 3 inches will yield only about 17 m (56 ft) of range for effective depth perception. This is definitely not acceptable or useful in an automobile setting and far below a radar system (which is 100m or more), even though for our eyes it is plenty useful. After the recent accident, there were calls for Tesla to have sensors for even longer range.
Subaru Gives EyeSight to the Distracted

Again, I should remind you that human depth perception relies largely on monocular cues (AKA single camera analogy) and only on binocular cues (AKA stereo camera) for the shorter distances.

Using the stereoscopic cameras eliminates the need for the forward radar system and would provide a cost savings and a system upgrade at the same time. Something that would be appealing to Tesla with the T3 being Tesla's "economy car".

The idea with a Autopilot 2.0 is to improve upon the existing one, not a step backwards. I would not expect Tesla to eliminate the front radar sensor. It is still required to see in lower visibility conditions (something radar can handle, but a camera system can't).

 

[EaglesPDX]

This is completely unfounded speculation. An automatic emergency braking system involves a lot more than simply depth perception.

No it's not since the stereoscopic camera is the only detection system the Subaru has for braking it is key to providing the depth perception necessary. We are talking about the sensor input data,not the data processing of it which is the same in the all vehicles with auto braking.

That ignores the distances we are discussing. Horizontal disparity (which a stereo system relies on to do depth sensing) is directly related to the distance between two eyes (or two cameras in this case).

No it doesn't ignore it. It does note that three inches is more that humans have for depth perception which is the distance between the cameras in the Tesla.

So if a 14 inch distance in the Subaru system yields 80 m (262 ft) of range, 3 inches will yield only about 17 m (56 ft) of range for effective depth perception.

So human depth perception is only 56 ft?

 

[stopcrazypp]

No it's not since the stereoscopic camera is the only detection system the Subaru has for braking it is key to providing the depth perception necessary. We are talking about the sensor input data,not the data processing of it which is the same in the all vehicles with auto braking.

This is completely false. I already pointed out large differences between the Subaru and Volvo system completely independent of the detection system. Subaru set their parameter such that a maximum closing speed of 20mph will still have the system activate. Volvo set it at 9mph (which pretty much insures it can't past the 25mph test). There are also systems that don't allow the car to autobrake to a stop (so it will never get full points on the test).

No it doesn't ignore it. It does note that three inches is more that humans have for depth perception which is the distance between the cameras in the Tesla.

So human depth perception is only 56 ft?

I'm talking about binocular depth perception (ones that relies on two eyes or two images). This is done by comparing differences between two images (a close object will have a larger difference, while a far object will have less difference).

Monocular depth perception (which can be accomplished by one eye or picture using visual cues) can extend much further. For example, when you look at an airplane in the sky, you can tell how far away it is just from the relative size of it, even though the distance is already far beyond the point where comparing the two images from your different eyes can give you an idea.

Also a camera and human eye has completely different resolutions. The typical VGA camera in many systems only has 0.3MP. The human eye is counted at minimum around 10MP near the center and ranges to 576 MP if you count peripheral vision and how the eyes can move (unlike a fixed camera). My calculation is only applicable to the assumption that a similar camera is used in both cases (do note, there is a limiter on adopting very high resolution sensors, given the processing power required goes up proportion to the resolution).

Main point is, if Tesla intended their set up to be a stereoscopic camera, they would put them much further apart (Subaru deliberately chose that distance so that their system can match a bare minimum radar setup at 80m). You have also completely ignored my point that Mobileye already said the other two front cameras will not be used as a stereoscopic system, but rather to allow front cameras with different fields of view.

 

[EaglesPDX]

Subaru set their parameter such that a maximum closing speed of 20mph will still have the system activate.

Subaru braking will activate at 60 mph in dynamic cruise mode. It can bring the vehicle to a stop in 126 feet from 60 mph and will if you are cruising at 60mph and traffic dead stops 127 feet in front of you.

Auto braking has different parameters when dynamic cruise is off and that is 30 mph max speed per the manual to bring vehicle to safe stop. The Insurance Institute for Highway Safety uses 25 mph for its test speed for auto-braking.

Both, auto braking and dynamic cruise, use the stereoscopic camera as the sole information source.

Main point is, if Tesla intended their set up to be a stereoscopic camera, they would put them much further apart (Subaru deliberately chose that distance so that their system can match a bare minimum radar setup at 80m).

Subaru has no radar, just the stereo cameras. Tesla will be able to use 3 in. camera separation (same as human eye) just as well, just takes a faster computer since the information has less differential to use. Tesla may still use the radar sensor along with the new stereoscopic cameras since is uses radar for other functions like auto-park where the camera could not see things at bumper level up close. Subaru warns about objects the camera's can miss for cruise and auto braking.

 

[raysspl]

This is very very good to hear! Can't wait to see real world results on this upgrade.

 

[stopcrazypp]

Subaru braking will activate at 60 mph in dynamic cruise mode. It can bring the vehicle to a stop in 126 feet from 60 mph and will if you are cruising at 60mph and traffic dead stops 127 feet in front of you.

Auto braking has different parameters when dynamic cruise is off and that is 30 mph max speed per the manual to bring vehicle to safe stop. The Insurance Institute for Highway Safety uses 25 mph for its test speed for auto-braking.

Both, auto braking and dynamic cruise, use the stereoscopic camera as the sole information source.

The first version had a 19mph closing speed, but 2nd version updated to 30mph. The system can still react at a higher speed, but might not stop in time.
http://media.subaru.com/pressrelease/562/123/subaru-debuts-next-generation-eyesight-system

But the specifics of the exact mph doesn't matter. My main point is that the fact that the Volvo system had a different and lower 9 mph setting that made sure it won't get the top score and that such a speed differential setting would make a huge difference in the test results regardless of the detection system being used. Basically your point ignores there are many independent variables to a vehicle getting a top score and the detection system is only one of them.

However, the speed differential of 20 mph limits the system’s ability to react in many dangerous situations. For instance, an EyeSight-equipped vehicle traveling at 70 mph (113 km/h) won’t recognize until the last second that it is approaching another vehicle in the same lane going much slower, say, 30 mph (48 km/h).

“The system will probably initiate braking, but it won’t avoid a collision,” he says.

What sounds like a fundamental flaw actually is the norm for these types of systems, which are intended to supplement – rather than supplant – the abilities of the driver.

In fairness, the City Safety system on the more expensive Volvo has an even more restrictive speed differential of 9 mph (15 km/h).

Subaru Gives EyeSight to the Distracted

Subaru has no radar, just the stereo cameras. Tesla will be able to use 3 in. camera separation (same as human eye) just as well, just takes a faster computer since the information has less differential to use. Tesla may still use the radar sensor along with the new stereoscopic cameras since is uses radar for other functions like auto-park where the camera could not see things at bumper level up close. Subaru warns about objects the camera's can miss for cruise and auto braking.

Actually more processing power would not solve the problem. The computer can't process something the sensors didn't detect. Tesla would need to use two higher resolution sensors that is 4.7x the horizontal resolution of Subaru's (given sensors typically increase in both horizontal and vertical resolution at the same time, in practice, that might mean a sensor with 22x the MP). That's going to be very expensive (plus it also ignores optics: sensor size and lens size matters also, so you can't just throw a high resolution sensor in). It is far cheaper to just make the separation distance further (or accept that the system will have a poor depth perception range).

But the known evidence points against this speculation regardless. I didn't look very carefully the first time around, but digging around, I looked more closely at the block diagram that was recently leaked about the updated autopilot hardware and guess what it says:
Triple Cam - Main
Triple Cam - Narrow/Mono
Triple Cam - Fisheye

That matches exactly the same illustration I posted before about the EyeQ4 system:

 

[palmer_md]

Pretty cool video explaining some of the 3d effects with technology.

 

[EaglesPDX]

@stopcrazypp "Actually more processing power would not solve the problem. "

The closer the two cameras are together the less diiferentiation so the computer would need to be able to make up the loss.

@stopcrazypp "The computer can't process something the sensors didn't detect. Tesla would need to use two higher resolution sensors that is 4.7x the horizontal resolution of Subaru's (given sensors typically increase in both horizontal and vertical resolution at the same time, in practice, that might mean a sensor with 22x the MP). That's going to be very expensive."

Well...we wouldn't want a Tesla to get as expensivel as a Subaru.