Software Update 2018.39 4a3910f (plus other v9.0 early access builds)

[CK_Stuggi]

What resolution of camera would be needed to deliver this 10cm 3D map at a distance of say 100m?

Depends on the focal length. speaking in 35mm ("full frame") equivalents, on a 24mm wideangle it would need to be super hires, on a 400mm I'd say HD (1080p) should be enough. Now, I dont know the 35mm focal length equivalent of the 3 front cameras or their resolution.
The question is what do you need 10cm 3D resolution for at 100m?
3D not necessarily needed for object detection, 2D can do that.
Judging from these collages from the cameras, and assuming the tele is just as high a resolution as the midrange and wide, it should be possible with the installed cameras to identify objects of 10cm size at 100m or more. But that's just eyeballing.
Lidar has the same issues plus significantly higher processing requirements and most of the same drawbacks.
All Tesla is talking about in terms of HW3 is an increase in processing power and running the Tesla Vision NN "on bare metal".
What can be done with just cameras was recently demonstrated by Mobileye with a very similar setup to Teslas, just cameras.
They also use a Roadbook, which is essentially a localized landmarks, to localize the vehicle on the road down to a precision of 2-3 cm. It's not as intense as a classic SLAM, because far fewer tracking points are needed.

Not really on topic though. My SLAM remark was intended as a reply to the original "one-eye-vision" post and why depth perception worked once in motion.

Another big problem with SLAM based on vision alone is moving objects. It works well for mapping stationary objects but if the things you're mapping are moving while you're moving you can't really separate their movement from your movement. It also requires a fair amount of processing horsepower, which Tesla is also stretched thin on.

Basically Tesla is completely screwed from doing any of the sophisticated things that the big boys (the ones with real sensor suites and processing power) can do.

SLAM is not even necessary. You can already calculate a pretty decent image difference map from simply comparing two subsequent frames from the camera to get outlines and calculate movement vectors from those. Combine that with the existing image recognition and you dont need full-blown SLAM.
And considering what kind of quality was available two years ago on decent cell phone cameras as well as that you are nowadays able to run pre-trained image recognition in realtime on said cellphones, I remain optimistic.

I still fail to see the "you need centimeter resolution at kilometer distance" Lidar argument, especially considering the massive cost/processing power demands and near identical issues with environmental influences.
But this is offtopic here as it's not really V9 related but belongs in the camera capabilities thread. So sorry to everyone who read this entire post and didnt learn anything new about V9.

 

[MarcusMaximus]

Take this with a grain of salt:
I believe the current v9 is a reduced form of the FSD code base. Actual FSD functionality requires AP3.
Previously, the FSD development was separate from the EAP due to restarting the base NN/ algorithms. EAP's code was being improved to provide more/ better functionality without destroying what was there...

That sounds plausible, but @Bladerskb reported it as fact, suggesting he either has an inside source or is making stuff up.

 

[rnortman]

Source? From what they’ve said to date, their FSD software is AP3 only.

They've said AP3 is required for FSD release, which is different from what you say here, which sort of implies that FSD is currently running on AP3. I suspect their FSD prototype software -- to the extent that it really exists at all -- is running on powerful off the shelf GPUs (probably Nvidia); more powerful than what's in HW2(.5). They are using this (hypothetical) R&D code running on the more powerful GPUs to set the requirements for HW3, if I had to guess. I doubt HW3 exists in much of a real form today, maybe some hacked up prototypes or proofs of concept, but maybe they have something good enough to run their models on. I bet the day to day development work though is on beefy OTS Nvidia GPUs.

 

[gearchruncher]

I believe the current v9 is a reduced form of the FSD code base. Actual FSD functionality requires AP3.

So Tesla's FSD code base is incapable of reading speed limit signs, stopping at stop signs, or changing lanes by itself? Two of those things can be done by Mobileye in 2015 in a small ASIC that fits in the camera housing. Plus everything else EAP does now too.

But yeah, FSD is a solved problem, and Tesla will be uploading this to your car any day, right after we put in that new computer.

 

[BigD0g]

They've said AP3 is required for FSD release, which is different from what you say here, which sort of implies that FSD is currently running on AP3. I suspect their FSD prototype software -- to the extent that it really exists at all -- is running on powerful off the shelf GPUs (probably Nvidia); more powerful than what's in HW2(.5). They are using this (hypothetical) R&D code running on the more powerful GPUs to set the requirements for HW3, if I had to guess. I doubt HW3 exists in much of a real form today, maybe some hacked up prototypes or proofs of concept, but maybe they have something good enough to run their models on. I bet the day to day development work though is on beefy OTS Nvidia GPUs.

Elon if we believe him states aph3 is already in dev cars today.

 

[mongo]

That sounds plausible, but @Bladerskb reported it as fact, suggesting he either has an inside source or is making stuff up.

v9.0 will start bringing in FSD features, per Elon, so it needs to come from that code base.

The camera feeds may be cropped or down sampled to work with the AP2.x HW.

So Tesla's FSD code base is incapable of reading speed limit signs, stopping at stop signs, or changing lanes by itself?

But yeah, FSD is a solved problem.

Huh? when did I say it was solved?

Elon if we believe him states aph3 is already in dev cars today.

Do you think the 100+ employee test drivers will be running AP3?

 

[rnortman]

Elon if we believe him states aph3 is already in dev cars today.

This could be true if they had two hacked up cars with prototypes rigged up in them. My post is only an educated guess but I stand by it -- day to day development work is happening right now on OTS Nvidia GPUs.

 

[MarcusMaximus]

This could be true if they had two hacked up cars with prototypes rigged up in them. My post is only an educated guess but I stand by it -- day to day development work is happening right now on OTS Nvidia GPUs.

Development work likely is and will continue to be on Nvidia GPUs. It wouldn’t make sense to do network training on anything else, especially if, as is highly likely, their runtime networks for AP3 are going to be quantized. Gradient calculations don’t work well without full floating point ops.

You generally don’t train networks on the same hardware used to run them...

 

[Bladerskb]

Source? From what they’ve said to date, their FSD software is AP3 only.

its really not. AP3 simply has more compute. but there is nothing stopping their software from running on 2.0 & 2.5 hardware. Their current FSD software is not as advanced as people think it is. Its mostly currently geared around highway autonomy. Alpha V10 is supposed to perform CC drive not tackle mainly street driving (per 1,000 miles rating).

Although Tesla’s NN architecture is probably in-efficient as hell based on the fact its using a public 2012 NN like googleNet.

Eyeq4 (2.5 TFLOPS, 3W) powers 8 cameras, multiple radar, lidar and sensing only takes up 10% of the chip processing if i recall correctly. I might have to look it up again and double check. Plus its running a lot more NN models than the sets that Tesla is running according to what @verygreen has shown us.

Their FSD cars uses 8 cameras and 4 eyeq4 chips (10 TFLOPS total). Their highway autonomy BMW cars uses 2 or was it 3 eyeq4 chips. Obviously Mobileye architecture must be a lot more efficient.

This is compared to AP2's (8-10 FP16 TFLOPS, 125W).

Obviously there's also hardware efficiency in eyeq4's ASIC with its deep learning and computer vision accelerators, so forth. But my point is that Tesla's' current FSD software should be able to run on 2.0 hardware even with its inefficient NN architecture because it’s not that advanced.

https://cdn.mobileye.com/wp-content/uploads/2017/11/mobileye_eyeq.png

 

[BigD0g]

v9.0 will start bringing in FSD features, per Elon, so it needs to come from that code base.

The camera feeds may be cropped or down sampled to work with the AP2.x HW.



Huh? when did I say it was solved?



Do you think the 100+ employee test drivers will be running AP3?

Eventually yes, today no

 

[MarcusMaximus]

its really not. AP3 simply has more compute. but there is nothing stopping their software from running on 2.0 & 2.5 hardware. Their current FSD software is not as advanced as people think it is. Its mostly currently geared around highway autonomy. Alpha V10 is supposed to perform CC drive not tackle mainly street driving (per 1,000 miles rating).

Although Tesla’s NN architecture is probably in-efficient as hell based on the fact its using a public 2012 NN like googleNet.

Eyeq4 (2.5 TFLOPS, 3W) powers 8 cameras, multiple radar, lidar and sensing only takes up 10% of the chip processing if i recall correctly. I might have to look it up again and double check. Plus its running a lot more NN models than the sets that Tesla is running according to what @verygreen has shown us.

Their FSD cars uses 8 cameras and 4 eyeq4 chips (10 TFLOPS total). Their highway autonomy BMW cars uses 2 or was it 3 eyeq4 chips. Obviously Mobileye architecture must be a lot more efficient.

This is compared to AP2's (8-10 FP16 TFLOPS, 125W).

Obviously there's also hardware efficiency in eyeq4's ASIC with its deep learning and computer vision accelerators, so forth. But my point is that Tesla's' current FSD software should be able to run on 2.0 hardware even with its inefficient NN architecture because it’s not that advanced.

https://cdn.mobileye.com/wp-content/uploads/2017/11/mobileye_eyeq.png

Again, you’re stating as fact a whole lot of not-publicly-visible(and otherwise unreported) information about Tesla’s FSD development codebase. So do you have a source for this?

 

[rnortman]

Development work likely is and will continue to be on Nvidia GPUs. It wouldn’t make sense to do network training on anything else, especially if, as is highly likely, their runtime networks for AP3 are going to be quantized. Gradient calculations don’t work well without full floating point ops.

You generally don’t train networks on the same hardware used to run them...

I meant that the inference models are probably also running on Nvidia GPUs for development work -- beefier GPUs than what's in the car for the advanced R&D, and then they probably work to squeeze the models down to fit on HW2(.5) GPUs.

 

[mongo]

Eventually yes, today no

whopse. I shouldn't have included the .0 good catch

 

[dhanson865]

I would wager 3/4 of the people on this forum do not know who Verstappen is. Americans like the race cars that can only turn in one direction.

Stock car racing is the most popular in the US. The tracks might only be an oval that curves to the left but exiting pit row is much easier if you turn to the right and the cars literally are street legal.

So modify that to "Americans like the race cars that mostly only turn in one direction" and you are right, but they sure can turn right.

 

[rnortman]

Although Tesla’s NN architecture is probably in-efficient as hell based on the fact its using a public 2012 NN like googleNet.

I believe this is no longer true, I would guess as of the 10.4 firmware update they are running on a new NN architecture. What you say applied to the first ~18 months of AP2 firmwares, and it's embarrassing as heck for Tesla that this is what was running on their fleet.

 

[bigdaddie]

I removed the firmware image from my server upon Tesla's (kind) request.
I understand they are starting to push a newer version anyway.

Does the car have to be rooted prior to uploading via your method?

 

[conman]

From @verygreen or others who've taken a closer look at the new firmware, does V9 include a new neural network?

 

[verygreen]

Does the car have to be rooted prior to uploading via your method?

no. don't need to root the car

 

[verygreen]

From @verygreen or others who've taken a closer look at the new firmware, does V9 include a new neural network?

it does

 

[CK_Stuggi]

The whole "solved" quote referred to Elons remark. Personally I understood it as "we know the method, so now it is just a question of time and training to achieve the solution". This is the meaning of the "Vision is essentially solved" quote to me. Now it certainly does take more time and training of the networks than Elon originally anticipated (surprise), but the fundamental methodical principle is not being changed. Essentially Mobileyes chips for their vision solution are doing the same thing, they may have more TOPs/Watt, but it's the same essential approach.
And the examples I've seen from Amnon Shashua’s keynote presentation at the 2018 Intel Capital Global Summit dont use 8 radars and 10 lidars. They use 8 cameras.