Tesla begins HD map development (collecting short camera videos) as i predicted

[Sir Guacamolaf]

Breaking news: Tesla autopilot suddenly gains ability to recognize scooby doo.
In other unrelated news, Sir Guacamolaf really needs to stop carrying garbage out in his scooby doo underwear.

 

[Cosmacelf]

Geez guys, why all the blowback against Bladerskb?

I actually hadn't read anything he had posted about this before, but I agree with him 100%. I was mystified that Mobileye was touting in public technical presentations about how they did real HD mapping 1.5 or 2 years ago, yet it appeared that Tesla wasn't doing this. Oh, Elon talked about HD maps, but they were GPS only as far as I could tell.

As Bladerskb points out, even "high res" GPS isn't that accurate. It is often several meters off. That's fairly useless for autopilot uses.

What Mobileye did years ago was to use a small fleet of OEM test cars to grab video of things that are stationary along routes. So think traffic signs primarily. Even along rural highways, you are going to have a deer crossing sign, a speed indicator, a street sign, a curvy road warning, etc. every few hundred meters at most. If you know your position within a few meters from GPS and then see an expected curvy road sign 1 meter away from the expected position, you have just localized your position exactly. You can do dead reckoning for 200 meters until you hit another sign. This way you know exactly where you are within centimeters at all times.

I am actually more than a little bit pissed off at Tesla right now because I had ASSUMED that Tesla had been gathering this data for the last two years. I mean, Elon almost stated as much when he talked about high resolution maps, and then also talked about fleet learning. But if they are just grabbing video clips NOW, then WTF?

Anyways, I'm glad they are finally doing it, even if it is 2 years later than they should have done it. From talking to other people in silicon valley, it is getting really, really hard to attract and keep minimally competent engineers without paying them enormous sums of money, and even then, good luck.

 

[Duke-U]

It is not the message, it is the way it is presented that makes people oppose him....

 

[Hugh Mannity]

Breaking news: Tesla autopilot suddenly gains ability to recognize scooby doo.
In other unrelated news, Sir Guacamolaf really needs to stop carrying garbage out in his scooby doo underwear.

I normally take the garbage out in my "Y fronts" and rubber boots. Scooby gitch would be a step up; where did you score those ? My neighbours may appreciate that.

 

[MP3Mike]

Now their RSD will include lane lines, road signs and traffic light positions, etc just like Mobileye.
The only reason you would want the position of traffic lights for example is because you want to build a map of traffic lights and add control semantics to them.

I'm not sure that they are using this data to create HD maps, as I think HD maps are a dead end and waste of money/resources. Because just about any physical thing you map today could be moved, removed, changed tomorrow. You can't count on it at all. (Or maybe it is just in my area that people seem to plow down road signs all the time, and the replacement signs aren't always put back in the same location, if they are even replaced at all.)

Sure they are capturing the video so they can analyze where traffic lights are positioned but that is so that they can have their DNN trained to know where to look for them, not to build a HD map that will be out of date the day after it is created. (Since it seems each city/state/country uses different styles of traffic lights in different positons.)

it is like GM's Supercruise that relies on HD maps of the supported freeways that are created by special LIDAR vehicles mapping them. What happens tomorrow when they start construction in the area and move lanes of traffic? Does Supercruise see that what is in front of it doesn't match the HD map and just disable itself? That will never work for FSD. You can't rely on any pre-existing data/map for FSD, everything has to be able to be handled in real time by the sensors on the car itself.

 

[Cosmacelf]

I'm not sure that they are using this data to create HD maps, as I think HD maps are a dead end and waste of money/resources. Because just about any physical thing you map today could be moved, removed, changed tomorrow. You can't count on it at all. (Or maybe it is just in my area that people seem to plow down road signs all the time, and the replacement signs aren't always put back in the same location, if they are even replaced at all.)

Sure they are capturing the video so they can analyze where traffic lights are positioned but that is so that they can have their DNN trained to know where to look for them, not to build a HD map that will be out of date the day after it is created. (Since it seems each city/state/country uses different styles of traffic lights in different positons.)

it is like GM's Supercruise that relies on HD maps of the supported freeways that are created by special LIDAR vehicles mapping them. What happens tomorrow when they start construction in the area and move lanes of traffic? Does Supercruise see that what is in front of it doesn't match the HD map and just disable itself? That will never work for FSD. You can't rely on any pre-existing data/map for FSD, everything has to be able to be handled in real time by the sensors on the car itself.

HD maps are only one piece of the puzzle. No one is going to rely entirely on maps for autopilot. Obviously you need real time sensor data for new obstacles, stopped cars, debris, etc. But they are an important part of the ability of cars to confidently drive at high speeds.

Look at how we humans drive a new mountain road we have never driven before. You can always tell a local driver because they drive the curvy road a lot faster than a driver who is driving it for the first time. A local who has driven the same stretch of road 50 times before literally remembers (unconsciously) where every little curve and road slope change occurs. Based on current location cues, his brain knows that the road will curve down to the left in 50 feet, for example.

HD maps do the same thing. And they really help when sensor data is degraded either through snow on the road, heavy rain, a blinding misaligned on-coming headlight, or the sun at exactly the wrong angle in the sky. Having multiple inputs to making a correct decision, often called sensor fusion, allows an AI to drive like a human.

As far as road signs being moved, well so what. They aren't ALL going to be moved. Next time you take a drive, look at how many road signs you see as you drive. There are a LOT of them. The occasional moved sign will be noted as an exception by a properly coded piece of software.

 

[boaterva]

That data sharing prompt I've never seen before. Is it being rolled out with the new software update or can we navigate to it somehow already?

Did you get 17.17.4? It came with that. If so, perhaps it's a USA thing only, which would explain why you didn't get it.

 

[trils0n]

I am actually more than a little bit pissed off at Tesla right now because I had ASSUMED that Tesla had been gathering this data for the last two years. I mean, Elon almost stated as much when he talked about high resolution maps, and then also talked about fleet learning. But if they are just grabbing video clips NOW, then WTF?

Could also be they were collecting it before, but are now finally asking for permission (perhaps after the lawyers said "holy crap you didn't ask for permission to record??").

I thought they disclosed early on they were taking still-images whenever the driver intervened when autopilot was active, and when driver actions were different than shadow mode's anticipated actions. Video would be a next step?

 

[jayman]

Did you get 17.17.4? It came with that. If so, perhaps it's a USA thing only, which would explain why you didn't get it.

I'm in the US, got 17.17.4 this am and never got that prompt

 

[boaterva]

I'm in the US, got 17.17.4 this am and never got that prompt

So, now, we have a new theory.... Certain selected vehicles are picked for special surveillance duty!

 

[Bladerskb]

I'm not sure that they are using this data to create HD maps, as I think HD maps are a dead end and waste of money/resources. Because just about any physical thing you map today could be moved, removed, changed tomorrow. You can't count on it at all. (Or maybe it is just in my area that people seem to plow down road signs all the time, and the replacement signs aren't always put back in the same location, if they are even replaced at all.)

Sure they are capturing the video so they can analyze where traffic lights are positioned but that is so that they can have their DNN trained to know where to look for them, not to build a HD map that will be out of date the day after it is created. (Since it seems each city/state/country uses different styles of traffic lights in different positons.)

it is like GM's Supercruise that relies on HD maps of the supported freeways that are created by special LIDAR vehicles mapping them. What happens tomorrow when they start construction in the area and move lanes of traffic? Does Supercruise see that what is in front of it doesn't match the HD map and just disable itself? That will never work for FSD. You can't rely on any pre-existing data/map for FSD, everything has to be able to be handled in real time by the sensors on the car itself.

"create HD maps, as I think HD maps are a dead end and waste of money/resources."

You are contradicting yourself since they already went out their way to create a high precision map using only GPS which is in-accurate.

These maps are real time and crowdsourced and always up to date, unlike the GM supercruise maps.

End-to-end HD Mapping for Self-Driving Cars | NVIDIA

NVIDIA's open mapping platform is built on the NVIDIA DriveWorks software toolkit for autonomous driving, and designed for carmakers, map companies and startups to accelerate development. When just using cameras, the system incorporates deep learning algorithms to detect lanes, signs and other landmarks, and can be used to both create maps and determine when the environment has changed. - See more at: End-to-end HD Mapping for Self-Driving Cars | NVIDIA

 

[Bladerskb]

I'm in the US, got 17.17.4 this am and never got that prompt

I believe U.S is auto opted in. Check your privacy settings.

 

[MP3Mike]

I'm in the US, got 17.17.4 this am and never got that prompt

I don't think you are prompted automatically. You have to go into the privacy settings and turn it on if you want.

 

[Snuffysasa]

This is exactly as I predicted. To the point that it needs its own thread.
Its funny how I have been right about everything I have said concerning the tesla's autopilot yet I have received unprecedented backlash.

EAP HW2 - my experience thus far... [DJ Harry] (only one camera active)

Enhanced Autopilot 2 may never happen before Full Self Driving is ready! (my accurate predicted timeline)

Whether its the delay for parity or number of cameras currently being used, or how neural networks/deep learning works and the way Tesla leverages it. All the way down to the requirement of High Definition maps that are accurate to 5-10 CM unlike Tesla's current high precision map that uses GPS logging and is only accurate to a-couple meters.





All the way down to and didn't have / couldn't process the raw video data from Mobileye's AP1. I also said when they start their HD map development we will all know about it through announcement from Elon Musk himself or through data uploads from Wi-Fi.



Infact in my very first post in this forum I listed exactly how this video collection will happen and what it will be used for.
I have also said in the past that they will only need short video clip or even just a picture in some cases and processed meta data in most cases. The short video clips are then annotated by a human and collated to be used in training the specific models.



Tesla are doing two things. They are creating a 5-10 cm HD map for the car to localize in (requires only metadata processed by the on-board dnn model).

The HD Map also includes exact location of traffic lights. How traffic light works in SDCs is that they look for the exact position of a traffic light. They don't look at the entire picture. since they know the exact position of what traffic light they want to examine from the HD map, they focus on it (requires only metadata).

HD Maps also includes what traffic light exactly corresponds to a lane or road. At an intersection there could be 10 traffic lights facing you and you need to know exactly which one you should be paying attention to in relation to where you want to go (*requires maybe video)

Detection NN Models are not perfect and you can get accuracy of 80% easily but to improve on that, you need millions of more pictures to train the network with. Improving a traffic light or stop sign detection model will (require video clip/image). It can also be improved to only take a picture of intersections that are not mapped yet / intersections that it fail to properly recognize.

When Raw Video clip ( which are just small number of picture frames) are uploaded, they are annotated by a human and collated.

Why would you need video clips and not just metadata

For example if your car were to come to a stop at an intersection with no car directly in-front of it and that intersection haven't been mapped and it doesn't detect a traffic/stop sign. The car will take a picture with the assumption that its traffic sign/light detection model failed and there must be a stop light, or stop sign somewhere. The picture gets sent back to HQ, which is then annotated by a human and collated.

"Enhanced Autopilot 2 may never happen before Full Self Driving is ready! "

this doesn't make sense explain this to me.

In your other post you say:

"It will take another 3-4 months for an update for EAP to bring it within parity with AP1. Then it will take another 2-3 for another update that will introduce some samples of the true EAP features listed above. Then another 2-3 for a update will bring the cars close to feature-complete of the promises of EAP."

source, I do not remember Elon saying this at the October conference. When was this said??

but even if that was true, then that means we are like 6 months away from EAP being finished, and EAP would be finished in November.
If that is the case why would you think fully self driving will be ready before that? that makes no sense. what may make sense is EAP gets finished at the end of this year, and then after that in 2018 FSD features begin gradually rolling out.



"the fact that Tesla hadn't started their HD map development"
Tesla hasn't started HD map development??

pretty sure Tesla has been building HD map since Autopilot 7.0
Tesla Building Next Gen Maps through its Autopilot Drivers




I agree with you that Tesla is now collecting raw video data to and sending it to their servers to be annotated and then used as part of a training set for Tesla Vision CNN or CNNs.
And I think it is very possible they will also use this now for creating HD precision maps, however, there is no hard evidence that they are doing this.
Up until now I kind of assumed tesla cars were already working on making HD precision maps. And It's possible they have and will continue to without sending the raw sensor data, but sending the processed data, detected object data, either form mobileye on AP1 or from Drive PX on AP2.

 

[Snuffysasa]

Things like:

• How far away is the traffic lights / intersection lines /stop signs (video clip frames), feed the frames with the distance information (get distance information from car exact speed/IMU) into a deep neural network to develop a model that can predict the distance of traffic lights, stop signs and intersection lines (needed for FSD)

etc.

I agree with you if they had the distance information they could use that to train better neural network..

But How would they get accurate distance info from traffic lights, intersections, stop signs, etc

 

[Snuffysasa]

I'm not sure that they are using this data to create HD maps, as I think HD maps are a dead end and waste of money/resources. Because just about any physical thing you map today could be moved, removed, changed tomorrow. You can't count on it at all. (Or maybe it is just in my area that people seem to plow down road signs all the time, and the replacement signs aren't always put back in the same location, if they are even replaced at all.)

Sure they are capturing the video so they can analyze where traffic lights are positioned but that is so that they can have their DNN trained to know where to look for them, not to build a HD map that will be out of date the day after it is created. (Since it seems each city/state/country uses different styles of traffic lights in different positons.)

it is like GM's Supercruise that relies on HD maps of the supported freeways that are created by special LIDAR vehicles mapping them. What happens tomorrow when they start construction in the area and move lanes of traffic? Does Supercruise see that what is in front of it doesn't match the HD map and just disable itself? That will never work for FSD. You can't rely on any pre-existing data/map for FSD, everything has to be able to be handled in real time by the sensors on the car itself.

HD Maps will be updated daily, in the next 5 -10 years we will many many companies with level 4 fully autonomous cars in many many parts of the world. Every single one of them will require an HD map.

they will not run exclusively on HD maps but also use the sensors.

It will be expensive and costly maintain these HD maps daily, but they will be paid for by the cost of self driving cars.

 

[Bladerskb]

"Enhanced Autopilot 2 may never happen before Full Self Driving is ready! "

this doesn't make sense explain this to me.

In your other post you say:

"It will take another 3-4 months for an update for EAP to bring it within parity with AP1. Then it will take another 2-3 for another update that will introduce some samples of the true EAP features listed above. Then another 2-3 for a update will bring the cars close to feature-complete of the promises of EAP."

source, I do not remember Elon saying this at the October conference. When was this said??

That was my prediction based on my statistical observation. I said that in April we will get an update that brings us "within parity" but not full parity. That happened at the end of April. Then I said in June/July we will finally see some of the true EAP features like autosteer+, on/off ramp, self lane change, self park, smart summon. Then by sept/oct we will be eap feature/performance complete.

"
but even if that was true, then that means we are like 6 months away from EAP being finished, and EAP would be finished in November.
If that is the case why would you think fully self driving will be ready before that? that makes no sense. what may make sense is EAP gets finished at the end of this year, and then after that in 2018 FSD features begin gradually rolling out.

The reason why I said EAP won't be here before FSD has heavily progressed is because of HD mapping which I talked about in first post of the thread. Not that the actual FSD software will be ready, but the underlining like the crowd-sourced real time HD map. Which is ofcourse now happening.

"the fact that Tesla hadn't started their HD map development"
Tesla hasn't started HD map development??

pretty sure Tesla has been building HD map since Autopilot 7.0
Tesla Building Next Gen Maps through its Autopilot Drivers

That is Tesla high precision map which I have mentioned 1,000 times. It uses gps logging and isn't accurate enough and isn't a hd map. Its good for level 2 ADAS though.
Tesla is mapping out every lane on Earth to guide self-driving cars

"
I agree with you that Tesla is now collecting raw video data to and sending it to their servers to be annotated and then used as part of a training set for Tesla Vision CNN or CNNs.
And I think it is very possible they will also use this now for creating HD precision maps, however, there is no hard evidence that they are doing this.
Up until now I kind of assumed tesla cars were already working on making HD precision maps. And It's possible they have and will continue to without sending the raw sensor data, but sending the processed data, detected object data, either form mobileye on AP1 or from Drive PX on AP2.

There are thousands of configuration of traffic lights in different positions and locations, representing different drivable path.
They will need HD map to do any kind of good semantic traffic light detection without false positives and false negatives.

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37259.pdf

 

[mdevp]

What will happen when a Tesla owner is in an accident and the other guy sues for the video? Apple vs FBI all over again?

 

[GasKilla]

I got the new update but didn't get a data sharing prompt...

My guess is us AP1 folks (aka hardware 1) aren't participating in the data sharing due to the lack of additional cameras on our cars. Anyone with AP1 get the prompt?

 

[Snuffysasa]

That was my prediction based on my statistical observation. I said that in April we will get an update that brings us "within parity" but not full parity. That happened at the end of April. Then I said in June/July we will finally see some of the true EAP features like autosteer+, on/off ramp, self lane change, self park, smart summon. Then by sept/oct we will be eap feature/performance complete.

Okay thank you clarifying. So do you still believe that June/July is when we would see some Enhanced Autopilot features, like autosteer+. on/off ramp, auto overtaking, smart summon.

I am curious what kind of features do you think would come first? and what would later like in sept/oct??

So listen carefully to this part, the EAP features such as on/off ramp, automatic lane change, transition between freeway... to me these features seem impossible without an HD map. because, for things like freeway transitioning and exiting, the vehicle would have to know exactly what lane it is supposed to be in, and to automatic lane change, the vehicle would have to know it is not going to merge into a lane that is about to end, or is an exit lane. I suppose technically possible it could use camera to read signs about lanes, but seems unreliable. Therefore, I feel the HD map is needed to enable these EAP features. thoughts on this?


Elon said in January that users who got fully self driving would notice a difference in 3-6 months. so late April - late July..
Any thoughts on this? Oblivious, Elon is overly optimistic and says things that may mislead people. However, I doubt he would fabricate something like that if it was based off no truth. surely he could not be talking about collecting map data.. the user doesn't notice a difference. Do you think there is something to this, and in a few months or a little longer if its delayed. There will be some sort of FSD feature, even if its a simple feature? any idea would it could be and its timing, or if its happening at all?

The reason why I said EAP won't be here before FSD has heavily progressed is because of HD mapping which I talked about in first post of the thread. Not that the actual FSD software will be ready, but the underlining like the crowd-sourced real time HD map. Which is ofcourse now happening.

Ok I understand what you meant.

That is Tesla high precision map which I have mentioned 1,000 times. It uses gps logging and isn't accurate enough and isn't a hd map. Its good for level 2 ADAS though.
Tesla is mapping out every lane on Earth to guide self-driving cars

okay thank you, I didn't know this.

There are thousands of configuration of traffic lights in different positions and locations, representing different drivable path.
They will need HD map to do any kind of good semantic traffic light detection without false positives and false negatives.

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37259.pdf

Yea I agree with you here.



Thanks,
finally do you have any thoughts this:
Why is Tesla sharing camera data?