Tesla begins HD map development (collecting short camera videos) as i predicted

[cdub]

I am actually more than a little bit pissed off at Tesla right now because I had ASSUMED that Tesla had been gathering this data for the last two years. I mean, Elon almost stated as much when he talked about high resolution maps, and then also talked about fleet learning. But if they are just grabbing video clips NOW, then WTF?

MobleEye doesn't allow ANYONE access to the raw video feed of their cameras when you use their chip. Probably main reason why Tesla switched to their own.

 

[Snuffysasa]

So know that Tesla AP2 cars starting now will begin creating HD maps for fully self driving. How long would it take to say... accurately map out the major freeways and highways in a place like Cali where there are lots of Teslas?

 

[MP3Mike]

Anyone with AP1 get the prompt?

It isn't a proactive prompt, you have to go to the privacy settings and turn it on if you want to opt-in.

From ev-fw I don't see that any AP1 car has received the new version yet, so I don't think we know if they can opt-in or not yet.

 

[cdub]

It isn't a proactive prompt, you have to go to the privacy settings and turn it on if you want to opt-in.

From ev-fw I don't see that any AP1 car has received the new version yet, so I don't think we know if they can opt-in or not yet.

Tesla cannot access the AP1 mobleye video data as far as I can tell.

 

[GasKilla]

Tesla cannot access the AP1 mobleye video data as far as I can tell.

It isn't a proactive prompt, you have to go to the privacy settings and turn it on if you want to opt-in.

I'll check the next time I'm in my car

From ev-fw I don't see that any AP1 car has received the new version yet, so I don't think we know if they can opt-in or not yet.

Which is the new firmware version? I recently went to a service center where they upgraded my firmware.

 

[stopcrazypp]

...
Infact in my very first post in this forum I listed exactly how this video collection will happen and what it will be used for.
I have also said in the past that they will only need short video clip or even just a picture in some cases and processed meta data in most cases. The short video clips are then annotated by a human and collated to be used in training the specific models.
...
Why would you need video clips and not just metadata

For example if your car were to come to a stop at an intersection with no car directly in-front of it and that intersection haven't been mapped and it doesn't detect a traffic/stop sign. The car will take a picture with the assumption that its traffic sign/light detection model failed and there must be a stop light, or stop sign somewhere. The picture gets sent back to HQ, which is then annotated by a human and collated.

Lol lots of horn tooting for yourself, but you seem to selectively quote yourself to try to match new facts, when you said a statement that directly contradicted yourself (you now say "Why would you need video clips and not just metadata", but back then you said "Tesla would only need the meta data").

From same quote you quoted above, you left out the part where you said they only need meta data and that they don't have to send the raw video:

The car with 8 cameras can creates a holistic 3d view of its world in real world with annotations.

IF an interesting situation were to take place. Or an interesting intersection or stretch of road were marked on the map for recording. All that needs to happen is for the car to record the last 30 seconds of the cars encounter and send it over to Tesla HQ.

But wait, you don't have to send raw video over. You people keep forgetting that the Nvidia PX2 runs the DNN in real time and already processes the raw video data.

Tesla would only need the meta data. What do i mean by meta data? everything already tagged and processed by the DNN and includes numerical and temporal representation of the space around the car, the coordinates of the objects(other cars/obstacles/pedestrain/traffic signs/lanes/road edges/etc) around the car and their velocity, and more.

This immense data could be less than 1MB and can then be loaded into Tesla's simulator.

They will have metadata of the exact position of every traffic light in every city. They will have metadata of position of traffic signs, stop signs, speed limits, lane markings, etc

They will have data of every parking place and spot a tesla car ever parked in during manual mode.

So if a tesla drives to and parks in McDonald or any business in manual mode. The car will save data of the parking structure and its spots and how exactly to navigate in and out of it and beams it up to HQ.

if the car were to encounter a place it fails at during shadow mode. it simply does what xbox does. Records the last 30 seconds and beams it up to HQ.

In fact, I believe you took repeatedly took issue with @Reciprocity claims about Tesla using raw video footage (which this new update proves they are doing). In fact, I can see @Reciprocity is much closer to predicting this, as he explicitly says that Tesla would collect moments of video at key points (you seem to have continually misinterpreted his claim, into something like sending all video data to Tesla HQ which would take TBs of data).

See below the exchange from this thread:
Experts ONLY: comment on whether vision+forward radar is sufficient for Level 5?

P.S. All that stuff looks scary to humans, but it is trivial to track a few objects for a super computer.
P.S.S. do you think Intel might be exaggerating a bit. Or do they not know how compress video? Tesla has no Lidar or that's 1/2 the 4000GB a day. Compressing the video gets it down to a couple of Netflex HD movies for the average driver per day. Lastly, you dont need every moment of data, only the moments where the car deviates from what the human did in a measurable way. In coding, this is like only viewing the differentials in the code vs the entire code base to find out what changed. The smarter the system gets, the less of these situations that need to be analyzed by more powerful machines or worst case, people.

The TB of data I'm referring to is the one required for @Reciprocity wishful scenario.
He claims that Tesla is doing end to end and sending raw pixels (billions of miles of pictures according to him).

However, what you are talking about is meta-data, which is good and solves the data size problem.
I have mentioned this many times. Check one of my first posts in this forum.

The problem is that those data still need to be human labeled and collated and algorithm must be created and fine-tuned to each specific problem.
This is exactly what for example google does.

You still can't feed that information (raw pixel or meta data) to a network and then get FSD back.

And Tesla does/have used driveworks. For your information that end to end method is not driveworks.

Infact driveworks was released October 2016 to all Nvidia's OEM

 

[croman]

It isn't a proactive prompt, you have to go to the privacy settings and turn it on if you want to opt-in.

From ev-fw I don't see that any AP1 car has received the new version yet, so I don't think we know if they can opt-in or not yet.

Actually you must opt out, not in. It defaults as approved sharing. No prompt.

 

[Bladerskb]

Okay thank you clarifying. So do you still believe that June/July is when we would see some Enhanced Autopilot features, like autosteer+. on/off ramp, auto overtaking, smart summon.

I am curious what kind of features do you think would come first? and what would later like in sept/oct??

Yes I still believe June/July will be when we see any kind of EAP specific features. Remember the last big AP update before this was in the last week of April. So that's 5 weeks. This fits right in my observation of June/July.

Coming first would either be on/off ramp and/or self auto-lane change.
In-fact they might actually start off with auto-lane change using camera instead of ultrasonic first but still requires driver turning the signal then before releasing another update that does it by itself.

However, I can say with high confidence that smart summon will be last.

So listen carefully to this part, the EAP features such as on/off ramp, automatic lane change, transition between freeway... to me these features seem impossible without an HD map. because, for things like freeway transitioning and exiting, the vehicle would have to know exactly what lane it is supposed to be in, and to automatic lane change, the vehicle would have to know it is not going to merge into a lane that is about to end, or is an exit lane. I suppose technically possible it could use camera to read signs about lanes, but seems unreliable. Therefore, I feel the HD map is needed to enable these EAP features. thoughts on this?

I agree with you and that's why I believe we won't see EAP performing to optimal standard till the ending of the year.
But we know Elon, he has a record of releasing Pre-alpha/Beta software. So I won't be surprised if he did release something without HD map being completely ready.

Elon said in January that users who got fully self driving would notice a difference in 3-6 months. so late April - late July..
Any thoughts on this? Oblivious, Elon is overly optimistic and says things that may mislead people. However, I doubt he would fabricate something like that if it was based off no truth. surely he could not be talking about collecting map data.. the user doesn't notice a difference. Do you think there is something to this, and in a few months or a little longer if its delayed. There will be some sort of FSD feature, even if its a simple feature? any idea would it could be and its timing, or if its happening at all?

I doubt we will see any thing by July, but more like Sept/Oct. Subsets of FSD features like stopping at traffic light, stop signs, handling roundabouts.

 

[croman]

@Bladerskb

Why did Tesla claim to gather this information in 2015? This article says Tesla is mapping the location of lights signs etc. Isn't that your HD map or is there more granularity in other types of map data?

Tesla Building Next Gen Maps through its Autopilot Drivers

 

[Bladerskb]

Lol lots of horn tooting for yourself, but you seem to selectively quote yourself to try to match new facts, when you said a statement that directly contradicted yourself (you now say "Why would you need video clips and not just metadata", but back then you said "Tesla would only need the meta data").

From same quote you quoted above, you left out the part where you said they only need meta data and that they don't have to send the raw video:

This is incorrect because I have said many times than I can remember in both this forum, reddit and electrek.co that tesla can take and upload video clips/pictures to train and improve their DNN but those clips MUST BE HUMAN ANNOTATED.

In that quote I specifically deal with the situation where you have great detection neural network model so they only need to record metadata

In fact, I believe you took repeatedly took issue with @Reciprocity claims about Tesla using raw video footage (which this new update proves they are doing). In fact, I can see @Reciprocity is much closer to predicting this, as he explicitly says that Tesla would collect moments of video at key points (you seem to have continually misinterpreted his claim, into something like sending all video data to Tesla HQ which would take TBs of data).

See below the exchange from this thread:
Experts ONLY: comment on whether vision+forward radar is sufficient for Level 5?

First of all this update PROVES they HAVENT been doing it, which is direct contradiction from what you have been posting the last couple months.

This is completely incorrect and displays your lack of knowledge. @Reciprocity
is talking about end to end self driving which is mapping images to steering angle, brakes and accelerator for control.
This is completely different from what Tesla is doing which is collecting video and pictures and doing human annotation on signs, traffic light, lanes, etc and training their models with it and also doing hd mapping.

@Reciprocity also said Tesla has billions of miles of video already from AP1. The same thing that you said which I proved to be false.

What @Reciprocity has been saying has nothing to do with what tesla is doing. He and you are 100% wrong.

 

[Snuffysasa]

Yes I still believe June/July will be when we see any kind of EAP specific features. Remember the last big AP update before this was in the last week of April. So that's 5 weeks. This fits right in my observation of June/July.

Okay cool, that's what I was thinking too, really hoping for some cool EAP feature to kick off Model 3 launch.

last big AP update end of April... are you referring to the update at end of April, where they added low speed AEB... that's defiantly an important update, but most people won't see it as big update.. maybe you referring to the update at the end or march that added full speed auto steer, auto lane change, and auto summon?

Coming first would either be on/off ramp and/or self auto-lane change.
In-fact they might actually start off with auto-lane change using camera instead of ultrasonic first but still requires driver turning the signal then before releasing another update that does it by itself.

However, I can say with high confidence that smart summon will be last.

hmm about the switching to camera for lane change...
so they have to already be using the camera for centering in new lane. to do auto lane change, the system would detect slow moving traffic, i mean nothing that it doesn't do already, just some new code that initiates lane change when front car is moving slow...

however there is one important piece to this, where it has to detect cars in the side lane to make sure it doesn't hit them obviously. but I imagine EAP can continue to do this fine with ultrasonic, I see no reason to not keep using ultrasonic for the cars on its side...

However it will need to detect cars in that lane that are approaching, in the side lane, that are a long ways back, like out of range of the ultrasonic sensors...

I imagine it would have to use the 2 rear facing cameras to do this.... So I would guess at somepoint soon there will be an OTA update, where someone realizes that those back cameras are now being used. (right now only 2 front cameras are being used). And then an update some point after that which would enable the automatic lane changing. Does this logic make sense to you?

I agree with you and that's why I believe we won't see EAP performing to optimal standard till the ending of the year.
But we know Elon, he has a record of releasing Pre-alpha/Beta software. So I won't be surprised if he did release something without HD map being completely ready.

though I suppose those GPS maps would be sufficient to know if a lane is an exit lane or is about to close... maybe?

I doubt we will see any thing by July, but more like Sept/Oct. Subsets of FSD features like stopping at traffic light, stop signs, handling roundabouts.

I was hoping to see some FSD feature around like august/September... Yea I would have maybe guessed stopping at stop lights, stop signs at first, and the user still needs to decide when to go through the intersection...

Though I was hoping that all this time, Tesla had been making progress on training Tesla vision to detect stop signs, and stop lights, since they are only collecting data for it now is discouraging it.... However I'm sure they must have still been doing some training on their own with their own data.

 

[Reciprocity]

Lol lots of horn tooting for yourself, but you seem to selectively quote yourself to try to match new facts, when you said a statement that directly contradicted yourself (you now say "Why would you need video clips and not just metadata", but back then you said "Tesla would only need the meta data").

From same quote you quoted above, you left out the part where you said they only need meta data and that they don't have to send the raw video:


In fact, I believe you took repeatedly took issue with @Reciprocity claims about Tesla using raw video footage (which this new update proves they are doing). In fact, I can see @Reciprocity is much closer to predicting this, as he explicitly says that Tesla would collect moments of video at key points (you seem to have continually misinterpreted his claim, into something like sending all video data to Tesla HQ which would take TBs of data).

See below the exchange from this thread:
Experts ONLY: comment on whether vision+forward radar is sufficient for Level 5?

I agree with ever nice thing said about me and disagree with every mean thing.

But seriously, none of us really knows and we are all just speculating and frankly it's very fun whether you are right or wrong. Both bladerskb and I have had some positives and negatives. At the end of the I think Tesla will win the race, he doesn't. We will all see soon enough.

 

[robby]

@Bladerskb You can save keystrokes by changing your post signature to ", as I predicted."

 

[TexasEV]

My guess is us AP1 folks (aka hardware 1) aren't participating in the data sharing due to the lack of additional cameras on our cars. Anyone with AP1 get the prompt?

No, I think it's because you have Mobileye hardware. Tesla can't use the Mobileye hardware for this. They're having to start from scratch with AP2.

 

[Reciprocity]

Okay cool, that's what I was thinking too, really hoping for some cool EAP feature to kick off Model 3 launch.

last bug AP update end of Apri... are you referring to the update at end of April, where they added low speed AEB... that's defiantly an important update, but most people won't see it as big update.. maybe you referring to the update at the end or march that added full speed auto steer, auto lane change, and auto summon?



hmm about the switching to camera for lane change...
so they have to already be using the camera for centering in new lane. to do auto lane change, the system would detect slow moving traffic, i mean nothing that it doesn't do already, just some new code that initiates lane change when front car is moving slow...

however there is one important piece to this, where it has to detect cars in the side lane to make sure it doesn't hit them obviously. but I imagine EAP can continue to do this fine with ultrasonic, I see no reason to not keep using ultrasonic for the cars on its side...

However it will need to detect cars in that lane that are approaching, in the side lane, that are a long ways back, like out of range of the ultrasonic sensors...

I imagine it would have to use the 2 rear facing cameras to do this.... So I would guess at somepoint soon there will be an OTA update, where someone realizes that those back cameras are now being used. (right now only 2 front cameras are being used). And then an update some point after that which would enable the automatic lane changing. Does this logic make sense to you?



though I suppose those GPS maps would be sufficient to know if a lane is an exit lane or is about to close... maybe?



I was hoping to see some FSD feature around like august/September... Yea I would have maybe guessed stopping at stop lights, stop signs at first, and the user still needs to decide when to go through the intersection...

Though I was hoping that all this time, Tesla had been making progress on training Tesla vision to detect stop signs, and stop lights, since they are only collecting data for it now is discouraging it.... However I'm sure they must have still been doing some training on their own with their own data.

You won't see and FSD features until after the public demo in Nov-Dec time frame Elon had given for the NY to LA demo.

They will more then likely release those features as safety features first as they won't need any or very little regulator approval to stop at a stoplight automatically if someone is about to blow through it. This will be part of the process for gaining full regulatory approval. Imagine sitting in front of the NHSTA and showing them video of the car stopping people from going through a red light or a stop sign and causing a major accident. Now take that times 1000 videos captured over a few months. Regulators will be tripping over themselves to get that approved.

 

[immolated]

I'm in the US, got 17.17.4 this am and never got that prompt

U.S. citizens don't need to consent to increased surveillance.

 

[GasKilla]

No, I think it's because you have Mobileye hardware. Tesla can't use the Mobileye hardware for this. They're having to start from scratch with AP2.

My point was AP1 cars have a single camera vs AP2 cars that have 3 forward facing cameras, even if they can access the Mobileye camera data it's probably less effective to sort data from one forward camera vs 3 especially if you are looking for stop signs and red lights.

 

[MP3Mike]

Which is the new firmware version? I recently went to a service center where they upgraded my firmware.

17.17.4

 

[GasKilla]

17.17.4

Thanks! I'm still on 17.14.35

 

[theslimshadyist]

This is exactly as I predicted. To the point that it needs its own thread.
Its funny how I have been right about everything I have said concerning the tesla's autopilot yet I have received unprecedented backlash.

EAP HW2 - my experience thus far... [DJ Harry] (only one camera active)

Enhanced Autopilot 2 may never happen before Full Self Driving is ready! (my accurate predicted timeline)

Whether its the delay for parity or number of cameras currently being used, or how neural networks/deep learning works and the way Tesla leverages it. All the way down to the requirement of High Definition maps that are accurate to 5-10 CM unlike Tesla's current high precision map that uses GPS logging and is only accurate to a-couple meters.





All the way down to the fact that Tesla hadn't started their HD map development and didn't have / couldn't process the raw video data from Mobileye's AP1. I also said when they start their HD map development we will all know about it through announcement from Elon Musk himself or through data uploads from Wi-Fi.



Infact in my very first post in this forum I listed exactly how this video collection will happen and what it will be used for.
I have also said in the past that they will only need short video clip or even just a picture in some cases and processed meta data in most cases. The short video clips are then annotated by a human and collated to be used in training the specific models.



Tesla are doing two things. They are creating a 5-10 cm HD map for the car to localize in (requires only metadata processed by the on-board dnn model).

The HD Map also includes exact location of traffic lights. How traffic light works in SDCs is that they look for the exact position of a traffic light. They don't look at the entire picture. since they know the exact position of what traffic light they want to examine from the HD map, they focus on it (requires only metadata).

HD Maps also includes what traffic light exactly corresponds to a lane or road. At an intersection there could be 10 traffic lights facing you and you need to know exactly which one you should be paying attention to in relation to where you want to go (*requires maybe video)

Detection NN Models are not perfect and you can get accuracy of 80% easily but to improve on that, you need millions of more pictures to train the network with. Improving a traffic light or stop sign detection model will (require video clip/image). It can also be improved to only take a picture of intersections that are not mapped yet / intersections that it fail to properly recognize.

When Raw Video clip ( which are just small number of picture frames) are uploaded, they are annotated by a human and collated.

Why would you need video clips and not just metadata

For example if your car were to come to a stop at an intersection with no car directly in-front of it and that intersection haven't been mapped and it doesn't detect a traffic/stop sign. The car will take a picture with the assumption that its traffic sign/light detection model failed and there must be a stop light, or stop sign somewhere. The picture gets sent back to HQ, which is then annotated by a human and collated.

Do you want a medal or a chest to pin it on???