The Tesla / Mobileye story

[lunitiks]

So this is not exactly hot news... More like funeral memorial words

I never really understood what happened between with Mobileye and Tesla. I mean truthfully: When and why did they actually decide to break up? Who took the initiative? Why would Mobileye want to leave Tesla, with Model 3 coming up (or even worse; after the hundreds of thousands of reservations)? Which company told the truth in their press statements in September 2016? What did Tesla and Mobileye plan to do before the breakup? And what's the deal with that weird "one-eyed" dual camera in AP1 Model X...?

Anyways I was bored so I researched a little and made this timeline Of course I've probably included a lot of stuff that's irrelevant, and omitted data that's important. Hence this thread Hoping you guys can help me see this more clearly.

Thanks!

2014​
February​	Rumor: Tesla and Mobileye teams up for Driverless Cars	Haaretz, Streetinsider
May​	Model S wiring diagrams updated: "Blind Zone and Forward Camera, Front Radar" (future option)	Tesla Motors Service
August​	Mobileye raises ⁓ $1 billion IPO	Timesofisrael, Crunchbase
September​	Autopilot HW spotted on new production vehicles	TMC, Wikipedia
2015​
March​	Mobileye introduces EyeQ4, production to begin in 2018	Prnewswire
March, 17​	Elon Musk and Jen-Hsun Huang declare self-driving cars “solved” (GTC 2015)	Fortune
March, 20​	Model S wiring diagram updated: Three cameras, rear radars, DAS ECU	Tesla Motors Service
May​	Model S test vehicle spotted with Mobileye patented triple-camera rig	Electrek, Google Patents
July​	Wiring diagram updated: Three cameras, front and rear radars, DAS "HW2" ECU, more harness details	Tesla Motors Service
September​	Mobileye is preparing for "first vehicle based on 8 cameras, one radar and ultrasonic around the vehicle."	Electrek, Mobileye, TMC
October​	Software version 7.0 with 'Autosteer' begins rollout	Tesla
November​	Sterling Anderson appointed Director of Autopilot Programs	Autonews
"​	Tesla: Looking for "hardcore software engineers" to bring Autopilot to "full autonomy". Project is "super high priority"	Twitter, Electrek
December​	Musk offers “multimillion-dollar bonus” for Geohot to build a “Mobileye crushing” Autopilot system for Tesla. MBLE down 7 %	Electrek
"​	Tesla: "We're Sticking With Mobileye Car Tech". Their part "is the best in the world at what it does and that is why we use it"	Fortune, Tesla
2016​
January​	Nvidia annouces Drive PX2	Theverge, Anandtech
"​	Elon Musk: "We are working on a new hardware suite, but it will be some time before it enters production"	Electrek
January - February​	Jim Keller joins Tesla to lead Autopilot Hardware Engineering team. Tesla poaches several "high-profile chip architects and executives from AMD to develop the next generation Autopilot"	Electrek
March​	Elon Musk reportedly visits Mobileye in Israel	Globes, Electrek
March, 31​	Model 3 unveiling	Theverge, Electrek
April, 1​	Elon Musk: Unveil "Part 2" will be "super next level"	Twitter, Marketwatch
May​	Joshua Brown accident	NHTSA
July, 15​	Elon Musk: "Working on using existing Tesla radar by itself (decoupled from camera)", with "temporal smoothing to create a coarse point cloud"	Twitter
July, 26​	Official break-up between Mobileye and Tesla. Collaboration will not extend beyond EyeQ3	Fortune, Techcrunch
July, 26​	Sterling Anderson: Tesla needs “tight integration” of hardware and software	Electrek
August​	Elon Musk says "what we’ve got will blow people’s minds, it blows my mind… it’ll come sooner than people think"	Electrek
September, 11​	Official blog post: "Seeing the world in radar"	Tesla
September, 12​	Elon Musk: "Tesla is reaching the limit of Autopilot hardware, but fleet learning will improve for years"	Electrek
September, 14​	Mobileye claims Tesla was "pushing the envelope in terms of safety"	Reuters
September, 15​	Tesla confirms developing in-house 'Tesla Vision' program 'for some time'. Claims Mobileye disparaged AP safety after learning about AP Vision in late July. Also claims Mobileye attempted to force Tesla to discontinue and pay more.	Electrek, Fortune
September, 16​	"Mobileye responds to false allegations"	Mobileye
October, 10​	Rumor: 'Tesla Vision' based on CUDA, requiring Nvidia GPU hardware	Electrek
October, 19​	AP 2.0 unveil	Tesla

 

[banned-66611]

They split because MobileEye didn't want its name attached to people dying.

This set Tesla back years, but might have done everyone a favour by keeping the roads a bit safer.

 

[DaveT]

@lunitiks I can't divulge how I know what I know. But here's a clue for you since you seem very interested.

The MobileEye chip has a raw data stream that is closed off to Tesla. Mobileye takes the raw data stream from their chip and then applies algorithms and such to output a much more simplified data stream from which actions can be taken from. However, Mobileye's system is far from perfect... or rather I should say the Mobileye's algorithms tend to ignore some important data (i.e, consider Joshua's Brown accident). Tesla only has access to the simplified data stream... or basically a data stream that's been stripped of important raw data. But with access to the raw data stream, Tesla could do so much more and improve the safety of the system (which to Tesla's mission is essential). Mobileye saw it a different way... very different.

 

[AnxietyRanger]

Nice post, thanks @lunitiks.

Some relevant history also here:

Is Model X (or was?) getting an "AP 1.5" before full-autonomous "AP 2.0" suite upgrade?

For example, what was never properly discussed was the rear-view camera shown on the roof of the original PR image set for Model X on production launch in Q3/2015. It may have been a Gentex rear-view mirror camera, though, not necessarily a MobilEye/AP feature.

 

[lunitiks]

Thanks guys . @DaveT I always had a feeling that was the case. In contrast to the ap2 setup, eyeq3 is integrated with the camera in the mirror housing, with no ethernet ports or nothing, just CANBUS talking with bosch and gw/cid... So my question is; is it virtually impossible to access the raw data stream, or is the problem IP?

 

[AnxietyRanger]

@lunitiks If something, your great summary is a bit light on the Tesla/Nvidia story, which of course started way back when for the displays/computers in the Model S, but was cozy even in the autonomous space long before the MobilEye split. The Nvidia story could be related/relevant to the MobilEye story?

This thread collected a lot of the early Drive PX speculation, links and quotes at the time starting in April, 2015:

Model X mule(s) show signs of nVidia Tegra X1 Drive PX platform - no rear mirror!

For example, what is relevant here is that the B pillar cameras (though much higher up) were thought to have been seen on these cars. It is possible they weren't, but nevertheless the thread contains links and quotes of the time to Tesla's (and Nvidia's) autonomous prototypes and conference commentary.

 

[lunitiks]

Yes @AnxietyRanger I agree, Tesla's relationship with nvidia began long before mobileye.

What I found particularly interesting while researching this, was the stuff about Mobileye about to launch "first vehicle based on 8 cameras, one radar and ultrasonic around the vehicle" ... I wonder if today's setup is a continuation of concrete plans between tesla and mobileye

 

[AnxietyRanger]

Yes @AnxietyRanger I agree, Tesla's relationship with nvidia began long before mobileye.

What I found particularly interesting while researching this, was the stuff about Mobileye about to launch "first vehicle based on 8 cameras, one radar and ultrasonic around the vehicle" ... I wonder if today's setup is a continuation of concrete plans between tesla and mobileye

Well, we know even the first AP2 board had the space for a MobilEye EyeQ3, the empty area. We also know that Audi zFAS has both MobilEye and Nvidia chips on it...

But something went south with Tesla and MobilEye...

 

[AnxietyRanger]

What happened to the two forward-facing cameras and one rearward-facing roof-camera that were in Model X in the Q3/2015 PR image set, but were not in the car that launched?

A feature that close to being launched that it was included in the PR shots, but not in the first cars shipped?

 

[ItsNotAboutTheMoney]

You missed:
July 1, 2016: BMW Group, Intel and Mobileye announce a partnership to develop full autonomy by 2021.
Mar 13, 2017: Intel announces acquisition of Mobileye.

Tesla became a competitor.

 

[AnxietyRanger]

You missed:
July 1, 2016: BMW Group, Intel and Mobileye announce a partnership to develop full autonomy by 2021.
Mar 13, 2017: Intel announces acquisition of Mobileye.

Tesla became a competitor.

But how is this different from. e.g. Volkswagen group using MobilEye and Nvidia chips on their Level 3+ autonomous vehicle computers side by side? They announced full autonomy much earlier than that.

zFAS is Audi's "AP2" and features both Drive PX and EyeQ3, they are the two grey chips on the left/center. Curiously a setup Tesla was originally supposed to use, but did not.

Audi runs their custom software on this setup, which also includes further chips from other suppliers. Just like Tesla was supposed to, before going all-Nvidia.

Nvidia’s processor is the “brain” of Audi’s zFAS system, which is the computer that handles driver assistance onboard the A8, and that takes sensor data gathered from the vehicle’s radar, camera, laser scanning and ultrasound sensors to create a fused picture of the road with a range of different types of data. The zFAS decides how the car behaves when traffic jam pilot is engaged, processing data at a rate of 2.5 billion inputs per second.

Nvidia is powering the world’s first Level 3 self-driving production car

 

[cantanko]

Brain is telling me there was an incongruency between Mobileye's roadmap and the, shall we say, ambitious timeline Tesla wished to pursue. Can't find the citation immediately, but I think Mobileye were citing safety concerns as their product to date was very deterministic whereas Autopilot's approach appeared to be a lot more heuristic than they wanted. Could be wrong - need to find the citation...

 

[AnxietyRanger]

There is something more to this than the mere idea of competing races to Level 5.

@DaveT's information about Tesla's wish to gain access to the raw data stream rings true, because it jives with some other comments at the time: MobilEye wanting access to Tesla's telemetry in exchange for deeper access to their system, and MobilEye's claim that it was better to let go of Tesla than to let them be reckless with their chip (to paraphrase).

It seems plausible MobilEye has been able to negotiate better access to Aud's use of EyeQ3, for example, or was more at ease with how Audi was using their product in the first place.

 

[ItsNotAboutTheMoney]

But how is this different from. e.g. Volkswagen group using MobilEye and Nvidia chips on their Level 3+ autonomous vehicle computers side by side? They announced full autonomy much earlier than that.

zFAS is Audi's "AP2" and features both Drive PX and EyeQ3, they are the two grey chips on the left/center. Curiously a setup Tesla was originally supposed to use, but did not.

Audi runs their custom software on this setup, which also includes further chips from other suppliers. Just like Tesla was supposed to, before going all-Nvidia.

Nvidia is powering the world’s first Level 3 self-driving production car

As I see it:

MobilEye's reputation was helped by its use in Tesla's aggressive AutoPilot deployment.
Tesla benefits significantly from its AutoPilot.
But Tesla was working to eliminate its use of MobilEye.
With the big money from Intel coming, MobilEye could use the death as a reason to cut Tesla off before Tesla had advanced its in-house system.
It could have killed Tesla, but Tesla BSed enough to buy time and now has reasonable capability in AP2.

In 2017 MobilEye hasn't been discussed much. It's all been about Cruise Automation.

 

[AnxietyRanger]

Well, that's a theory at least. No comment really...

Tesla BSed enough to buy time

I agree they did. I don't agree the BS was necessary, though.

 

[AnxietyRanger]

Brain is telling me there was an incongruency between Mobileye's roadmap and the, shall we say, ambitious timeline Tesla wished to pursue. Can't find the citation immediately, but I think Mobileye were citing safety concerns as their product to date was very deterministic whereas Autopilot's approach appeared to be a lot more heuristic than they wanted. Could be wrong - need to find the citation...

Yes, they did cite safety reasons:

“Long term this is going to hurt the interests of the company and hurt the interests of an entire industry, if a company of our reputation will continue to be associated with this type of pushing the envelope in terms of safety,”

Mobileye says Tesla was 'pushing the envelope in terms of safety'

On thing not mentioned (not sure if in @lunitiks' links, maybe it was) was Elon mentioning that MobilEye was too slow for them:

“This was expected and will not have any material effect on our plans. MobilEye’s ability to evolve its technology is unfortunately negatively affected by having to support hundreds of models from legacy auto companies, resulting in a very high engineering drag coefficient. Tesla is laser-focused on achieving full self-driving capability on one integrated platform with an order of magnitude greater safety than the average manually driven car.”

Of course we now know it had a material effect to their plans of at least one year's delay.

There are several sides to this story, to me it reeks of inability to come to a negotiated end-result. Both sides wanted something the other was not willing to give.

 

[banned-66611]

Tesla simply wanted FSD and MobileEye didn't think it was safe with current tech.

What seems odd is that Tesla then downgraded the hardware to AP2 level. Not only did they then have to reproduce all the work MobileEye had done on sensor processing, but with inferior hardware as well.

 

[Shock-On-T]

So this is not exactly hot news... More like funeral memorial words

I never really understood what happened between with Mobileye and Tesla. I mean truthfully: When and why did they actually decide to break up? Who took the initiative? Why would Mobileye want to leave Tesla, with Model 3 coming up (or even worse; after the hundreds of thousands of reservations)? Which company told the truth in their press statements in September 2016? What did Tesla and Mobileye plan to do before the breakup? And what's the deal with that weird "one-eyed" dual camera in AP1 Model X...?

Anyways I was bored so I researched a little and made this timeline Of course I've probably included a lot of stuff that's irrelevant, and omitted data that's important. Hence this thread Hoping you guys can help me see this more clearly.

Thanks!

2014​
February​	Rumor: Tesla and Mobileye teams up for Driverless Cars	Haaretz, Streetinsider
May​	Model S wiring diagrams updated: "Blind Zone and Forward Camera, Front Radar" (future option)	Tesla Motors Service
August​	Mobileye raises ⁓ $1 billion IPO	Timesofisrael, Crunchbase
September​	Autopilot HW spotted on new production vehicles	TMC, Wikipedia
2015​
March​	Mobileye introduces EyeQ4, production to begin in 2018	Prnewswire
March, 17​	Elon Musk and Jen-Hsun Huang declare self-driving cars “solved” (GTC 2015)	Fortune
March, 20​	Model S wiring diagram updated: Three cameras, rear radars, DAS ECU	Tesla Motors Service
May​	Model S test vehicle spotted with Mobileye patented triple-camera rig	Electrek, Google Patents
July​	Wiring diagram updated: Three cameras, front and rear radars, DAS "HW2" ECU, more harness details	Tesla Motors Service
September​	Mobileye is preparing for "first vehicle based on 8 cameras, one radar and ultrasonic around the vehicle."	Electrek, Mobileye, TMC
October​	Software version 7.0 with 'Autosteer' begins rollout	Tesla
November​	Sterling Anderson appointed Director of Autopilot Programs	Autonews
"​	Tesla: Looking for "hardcore software engineers" to bring Autopilot to "full autonomy". Project is "super high priority"	Twitter, Electrek
December​	Musk offers “multimillion-dollar bonus” for Geohot to build a “Mobileye crushing” Autopilot system for Tesla. MBLE down 7 %	Electrek
"​	Tesla: "We're Sticking With Mobileye Car Tech". Their part "is the best in the world at what it does and that is why we use it"	Fortune, Tesla
2016​
January​	Nvidia annouces Drive PX2	Theverge, Anandtech
"​	Elon Musk: "We are working on a new hardware suite, but it will be some time before it enters production"	Electrek
January - February​	Jim Keller joins Tesla to lead Autopilot Hardware Engineering team. Tesla poaches several "high-profile chip architects and executives from AMD to develop the next generation Autopilot"	Electrek
March​	Elon Musk reportedly visits Mobileye in Israel	Globes, Electrek
March, 31​	Model 3 unveiling	Theverge, Electrek
April, 1​	Elon Musk: Unveil "Part 2" will be "super next level"	Twitter, Marketwatch
May​	Joshua Brown accident	NHTSA
July, 15​	Elon Musk: "Working on using existing Tesla radar by itself (decoupled from camera)", with "temporal smoothing to create a coarse point cloud"	Twitter
July, 26​	Official break-up between Mobileye and Tesla. Collaboration will not extend beyond EyeQ3	Fortune, Techcrunch
July, 26​	Sterling Anderson: Tesla needs “tight integration” of hardware and software	Electrek
August​	Elon Musk says "what we’ve got will blow people’s minds, it blows my mind… it’ll come sooner than people think"	Electrek
September, 11​	Official blog post: "Seeing the world in radar"	Tesla
September, 12​	Elon Musk: "Tesla is reaching the limit of Autopilot hardware, but fleet learning will improve for years"	Electrek
September, 14​	Mobileye claims Tesla was "pushing the envelope in terms of safety"	Reuters
September, 15​	Tesla confirms developing in-house 'Tesla Vision' program 'for some time'. Claims Mobileye disparaged AP safety after learning about AP Vision in late July. Also claims Mobileye attempted to force Tesla to discontinue and pay more.	Electrek, Fortune
September, 16​	"Mobileye responds to false allegations"	Mobileye
October, 10​	Rumor: 'Tesla Vision' based on CUDA, requiring Nvidia GPU hardware	Electrek
October, 19​	AP 2.0 unveil	Tesla

This is a wonderful and interesting history but I’m too busy reading this

 

[Pale_Rider]

Tesla simply wanted FSD and MobileEye didn't think it was safe with current tech.

What seems odd is that Tesla then downgraded the hardware to AP2 level. Not only did they then have to reproduce all the work MobileEye had done on sensor processing, but with inferior hardware as well.

AP2 hardware is vastly superior to the hardware found in AP1. The programming is what they are having to play catch up on.

 

[AnxietyRanger]

Just came across this January 2016 quote (perhaps from some time earlier?) from Elon saying that Tesla's autonomous cross-country trip would need "more radars" than AP1 has.

"We do need more sensors than the car currently has to achieve that goal. You need a lot of redundancy … more cameras, more radars, redundant power buses … so that any system can fail in the car and it doesn’t need to backup to a driver."

Elon Musk predicts a Tesla will be able to drive itself across the country in 2018

Obviously AP2 shipped with the same number of radars as AP1, but this at least speaks a bit more of the (known) plan to have more radars at some stage. I doubt it was just the slip of a tongue, we of course saw those schematics and even mule shots showing rear radars...