Navigate On Autopilot: automatic lane change results

[Enginerd]

Greetings. I'd like to thank everyone for their support and interest in this experiment. Due to unplanned secondary effects of the pandemic, it would appear that drive #62 from mid-March will be the last official drive for this data set.

This experiment centered around my "unique commute" between my office in Savannah Georgia and my family in Greenboro North Carolina (330 miles each way). I had planned to continue that inconvenient lifestyle for a total of about 2.5 years, until my son graduated from high school with his peers in June of this year. However, as you probably know, the formal educational structure in the USA kind of dissolved in March when the widespread precautionary closures hit. My family took that opportunity to wrap things up in North Carolina, and accelerate the move to Georgia. Now, with in-person graduation effectively cancelled, the house sold, and all of our family members and worldly possessions in Georgia, my long distance commute is a thing of the past.

Like many of you, my driving is very limited nowadays. I've had precious little experience with the new FSD stop lights and stop signs features. The future of FSD will be very interesting to say the least. I've been thinking about how to continue quantifying the successes, shortcomings, and failures as the feature set becomes more comprehensive. However, if/when my commute resumes, it is (thankfully) only 4 miles each way, Monday through Friday, so it would be a very tiny (but repeatable) data set. If I decide to do any formal tracking at all, I would lean towards a very simple system with 3 basic categories for each event/maneuver. I'd appreciate any feedback you have on this.

• successful (green)
  
• marginal/stressful (yellow)
  
• driver intervention required for safety (red)

Looking back, I am pleased that I was able to:

1. quantify the success and reliability of NOA features, and
   
2. observe a small amount of general improvement over time

I am disappointed that:

1. the rate of improvement was *really* slow
2. my experiment was not fully representative of the varied conditions that exist in other more urban areas
   
3. I couldn't see the experiment through until NOA reached the magic 100% success rate

I attribute NOA's slow apparent progress to Tesla focusing on other required FSD features, and their presumed realization that a core rewrite was necessary to make long-term headway towards that goal. I'm OK with that, although I do wish I could summarize it all on a chart.

Drive safe y'all, and be well.

 

[thimel]

I’ve enjoyed the reports of you r systematic testing. Story to see it stop, but great that you now have a short commute.
Thanks for doing the study.

 

[Didgie]

Greetings. I'd like to thank everyone for their support and interest in this experiment. Due to unplanned secondary effects of the pandemic, it would appear that drive #62 from mid-March will be the last official drive for this data set.

This experiment centered around my "unique commute" between my office in Savannah Georgia and my family in Greenboro North Carolina (330 miles each way). I had planned to continue that inconvenient lifestyle for a total of about 2.5 years, until my son graduated from high school with his peers in June of this year. However, as you probably know, the formal educational structure in the USA kind of dissolved in March when the widespread precautionary closures hit. My family took that opportunity to wrap things up in North Carolina, and accelerate the move to Georgia. Now, with in-person graduation effectively cancelled, the house sold, and all of our family members and worldly possessions in Georgia, my long distance commute is a thing of the past.

Like many of you, my driving is very limited nowadays. I've had precious little experience with the new FSD stop lights and stop signs features. The future of FSD will be very interesting to say the least. I've been thinking about how to continue quantifying the successes, shortcomings, and failures as the feature set becomes more comprehensive. However, if/when my commute resumes, it is (thankfully) only 4 miles each way, Monday through Friday, so it would be a very tiny (but repeatable) data set. If I decide to do any formal tracking at all, I would lean towards a very simple system with 3 basic categories for each event/maneuver. I'd appreciate any feedback you have on this.

• successful (green)
  
• marginal/stressful (yellow)
  
• driver intervention required for safety (red)

Looking back, I am pleased that I was able to:

1. quantify the success and reliability of NOA features, and
   
2. observe a small amount of general improvement over time

I am disappointed that:

1. the rate of improvement was *really* slow
2. my experiment was not fully representative of the varied conditions that exist in other more urban areas
   
3. I couldn't see the experiment through until NOA reached the magic 100% success rate

I attribute NOA's slow apparent progress to Tesla focusing on other required FSD features, and their presumed realization that a core rewrite was necessary to make long-term headway towards that goal. I'm OK with that, although I do wish I could summarize it all on a chart.

Drive safe y'all, and be well.

Nooooo, say it ain't so! How am I gonna get all the lovely data for my video's now?

So, enough of the selfish rant, really happy that your life has become more convenient and family are well. Best wishes to you Enginerd.

 

[Fernand]

Drat, just go back to North Carolina .... not.
Thanks for all the dedication & tabulations!

 

[BridgeMojo]

@Enginerd what you did was a Big Deal. Thank you so much.

 

[Didgie]

This is the final video with Enginerd's lovely data...Or is it???

I made this vid using that last drop of Enginerd's data, it's a bit basic so considering making a dedicated reel showing the progress of Enginerds data over the past 12 months, as a bit of a thank you and testimonial. What do y'all think?

 

[drtimhill]

I am disappointed that:

1. the rate of improvement was *really* slow
2. my experiment was not fully representative of the varied conditions that exist in other more urban areas
   
3. I couldn't see the experiment through until NOA reached the magic 100% success rate

I attribute NOA's slow apparent progress to Tesla focusing on other required FSD features, and their presumed realization that a core rewrite was necessary to make long-term headway towards that goal. I'm OK with that, although I do wish I could summarize it all on a chart.

Obviously Tesla are the only ones who can accurately answer your speculation. However, what rate of improvement did you expect? Upon what was this expectation based? What is "100% success rate" anyway? I'm not being an apologist here, but realistic expectations can only be based upon precedent, and for the kind of things Tesla is doing, there is no precedent. You might as well be critical of Einstein for taking 10 years to develop General Relativity.

Comparisons to rocket science and SpaceX come to mind, and phrases such as "this is [or is not] rocket science". But rocket science is actually a mature engineering discipline compared to some of the things being done by Tesla (and others, of course) with self-driving cars. SpaceX have been truly brilliant is achieving something:. a dramatic reduction in the cost of reliably delivering payloads to space. But this is, at the end of the day, an economic achievement; they have not done anything technically that Apollo etc did not do decades before them.

But Tesla is doing something truly new; they are pushing at the very edges of AI with the various autopilot/self-driving efforts. And guess what? It's hard. Hard as in "it's nearly impossible to predict how long it will take or how far we can get". And let's just ignore Elons "it will be ready on XXX", that's just investor talk.

Tonight I watched my car, by itself, drive along a road with better accuracy than many human drivers, warn me that it was going to stop at an upcoming traffic signal, and then cleanly slow and stop exactly at the signal, before driving on after I tapped the accelerator. Yes, measuring how well the car does, and looking for flaws, and reporting and discussing them, are all good. Very good. But let's not lose sight of how breathtaking the achievements have been, and continue to be.

 

[Didgie]

Obviously Tesla are the only ones who can accurately answer your speculation. However, what rate of improvement did you expect? Upon what was this expectation based? What is "100% success rate" anyway? I'm not being an apologist here, but realistic expectations can only be based upon precedent, and for the kind of things Tesla is doing, there is no precedent. You might as well be critical of Einstein for taking 10 years to develop General Relativity.

Comparisons to rocket science and SpaceX come to mind, and phrases such as "this is [or is not] rocket science". But rocket science is actually a mature engineering discipline compared to some of the things being done by Tesla (and others, of course) with self-driving cars. SpaceX have been truly brilliant is achieving something:. a dramatic reduction in the cost of reliably delivering payloads to space. But this is, at the end of the day, an economic achievement; they have not done anything technically that Apollo etc did not do decades before them.

But Tesla is doing something truly new; they are pushing at the very edges of AI with the various autopilot/self-driving efforts. And guess what? It's hard. Hard as in "it's nearly impossible to predict how long it will take or how far we can get". And let's just ignore Elons "it will be ready on XXX", that's just investor talk.

Tonight I watched my car, by itself, drive along a road with better accuracy than many human drivers, warn me that it was going to stop at an upcoming traffic signal, and then cleanly slow and stop exactly at the signal, before driving on after I tapped the accelerator. Yes, measuring how well the car does, and looking for flaws, and reporting and discussing them, are all good. Very good. But let's not lose sight of how breathtaking the achievements have been, and continue to be.

We absolutely need people like Enginerd recording and reporting AP performance results for the masses, he is one of the good ones

You make a great point too, the other day I did a 1hour/48mile drive, over 90% was on AP and apart from junctions and roundabouts there were only 3 occasions where my Tesla didn't do quite what I wanted it to do (cyclist, joining the dual carriageway, speed matching). In anyone's book that is pretty awesome

 

[Enginerd]

Obviously Tesla are the only ones who can accurately answer your speculation. However, what rate of improvement did you expect? Upon what was this expectation based? What is "100% success rate" anyway? I'm not being an apologist here, but realistic expectations can only be based upon precedent, and for the kind of things Tesla is doing, there is no precedent. You might as well be critical of Einstein for taking 10 years to develop General Relativity.

I get the thrust of what you're trying to say, and I appreciate it. Based on FSD demos, promises of robo-taxis, and claims of feature complete, I was hoping to see some level of NOA improvement (with a good signal to noise ratio) over about a year time span. There was some improvement in NOA, but you have to really squint and tilt your head to see it. For sure there are other improvements in many other areas that Tesla is not showing us yet, and a small handful that have been released.

And let's just ignore Elons "it will be ready on XXX", that's just investor talk.

Fair, he's not good with target dates, but he has a good handle on the technical content. The reliability for FSD has to be really high when it is released to the masses, which is why you hear Karpathy talk about the march of 9's (99%, 99.9%, 99.99%, etc.).

But this is, at the end of the day, an economic achievement; they have not done anything technically that Apollo etc did not do decades before them.

I think that re-entering and landing a rocket ass-end first at the center of the pad counts a new technical achievement.

But Tesla is doing something truly new; they are pushing at the very edges of AI with the various autopilot/self-driving efforts. And guess what? It's hard. Hard as in "it's nearly impossible to predict how long it will take or how far we can get"... let's not lose sight of how breathtaking the achievements have been, and continue to be.

Peace. It's hard, they're working it, and it will take time. In the mean time, I'm enjoying it, and I can't wait to see their future results.

 

[drtimhill]

I think that re-entering and landing a rocket ass-end first at the center of the pad counts a new technical achievement.

Agreed, and I think its spectacular every time I watch one of them land. But the rationale remains economic; re-use the rocket (or as much as possible) to reduce costs.

And to be clear, I wasn't being critical of your work, which I for one appreciate very much. It's just easy for us all to forget, when the car doesnt meet our (high) expectations, just how amazing it really is, even at this early stage in its development.

 

[DopeGhoti]

But the rationale remains economic; re-use the rocket

The rationale may well be economic, but that does not mean that the task was not extremely technical. In the modern era, the rationale for nearly anything any company does - particularly publicly traded ones - is economic.

 

[Leloup]

I like stops!
I hate lights (red)
Why?
Because the stop sign is intelligent: if there is no one, I move on.
I dream that the cars respect all the stops.
Thus, we will save money because a fire is much more expensive than a stop sign.

 

[ggreen4077]

Scheduled for service appointment for my 2020 Model 3 next week to try to iron out some issues with NOA. Sent the following events log for the service techs to analyze prior to the appointment. Created this log at the request of the service techs. Some of the issues may be my learning curve, but it appears NOA does not pass slower cars, does not enter or exit freeway, does not show vehicles approaching from rear soon enough for safety and does not warn when changing lanes in front of vehicles approaching from rear.

May 5, 2020
Report of NOA issues with my Model 3.

Traveling US 59/I69 North through Porter, TX to exit FM 1485 at New Caney. Navigating to Montgomery County tax office which requires an exit and U-turn to get to the west side of Highway 59. Car is in center lane of access road two miles from entrance to HWY 59 . Traveling at 73 miles per hour on navigation autopilot.

10:09 in center lane of access road approaching US 59 on-ramp north of Northpark. Car does not move into left lane to merge onto 59, tries to pass entrance, have to take control to enter US 59. Reengaged NOA.

10:09 Pick-up approached from rear left becomes visible on display about six feet off my bumper. Car approaching from left becomes visible as it is beside my rear bumper.

10:11 pick-up approaching from right rear visible probably six feet off my bumper.

Sped up to 73 miles per hour following a semi truck going about 65 approaching exit in 1 mile.

10:13 I'm running in the center lane. No traffic on either side approaching sign for exit. Pick up approaching from the right rear visible now. Missed exit 159A. Car did not move over to exit lane and did not exit.

10:17 truck just merged in front of me from on-ramp. I had to manually slow down because the car would not move over and did not slow down until I was right on his bumper. Reengaged NOA.

10:18 Navigation says Exit, car did not exit 59.

10:18 tried to move over into the center lane with turn signal, the car started moving and then swerved back to the right.

10:19 had to manually take control to get the car off the interstate, U turned and headed south on 59 access road. Restarted NOA. Plan to merge onto 59 from the feeder road. Traveling in the center lane .3 miles before on-ramp. Navigation says to get in left lane car does not move over to the left lane until I physically make it move and get on the access ramp for 59.

10:23 pick-up passed on the left, visible about 5 feet off my bumper.

10:24 truck coming by on left not visible until he's beside me.

10:25 pick-up right behind that truck just now visible. He is almost even with my bumper before I see him on the screen.

Accelerating to 73 miles per hour coming up on the truck that's running a little slower.

10:25 my car is slowing but will not pass the truck. Speed down to 68 miles per hour. Truck approaching in the left rear he's probably 20 feet behind me and he's not visible. Becomes visible as he is even with my bumper.

10:26 Both lanes are now clearer behind me right and left. Car has slowed to 63 miles an hour in 65 mph zone behind the same truck and the car will not pass it.

10:26. Down to 61 miles an hour in a 65 Zone behind the truck with my cruise set at 80 and it will not pass.

10:27 manually change lanes to pass the truck. Car picking up speed. 2 miles to my exit running in the center lane.

10:28 1 mile to the exit at Community Drive that Navigation says I should use to exit HWY 59 South bound. Car is again slowing behind slower traffic at 64 miles per hour and will not pass and will not move over into the right lane to exit. Speed now down to 59 miles an hour behind slower traffic cruise is set at 80 running in navigate on autopilot. Car will not pass the cars in front. 1 miles to exit.

10:29, just passed my exit again. Car made no effort to move into the right lane or to exit.

5:18 pm, on Northpark Dr. Moved to change lanes in front of slow moving car. Car behind me accelerated, causing me to swerve back into my lane. No collision warning, car not visible in display.

 

[navguy12]

Scheduled for service appointment for my 2020 Model 3 next week to try to iron out some issues with NOA. Sent the following events log for the service techs to analyze prior to the appointment. Created this log at the request of the service techs. Some of the issues may be my learning curve, but it appears NOA does not pass slower cars, does not enter or exit freeway, does not show vehicles approaching from rear soon enough for safety and does not warn when changing lanes in front of vehicles approaching from rear.

May 5, 2020
Report of NOA issues with my Model 3.

Traveling US 59/I69 North through Porter, TX to exit FM 1485 at New Caney. Navigating to Montgomery County tax office which requires an exit and U-turn to get to the west side of Highway 59. Car is in center lane of access road two miles from entrance to HWY 59 . Traveling at 73 miles per hour on navigation autopilot.

10:09 in center lane of access road approaching US 59 on-ramp north of Northpark. Car does not move into left lane to merge onto 59, tries to pass entrance, have to take control to enter US 59. Reengaged NOA.

10:09 Pick-up approached from rear left becomes visible on display about six feet off my bumper. Car approaching from left becomes visible as it is beside my rear bumper.

10:11 pick-up approaching from right rear visible probably six feet off my bumper.

Sped up to 73 miles per hour following a semi truck going about 65 approaching exit in 1 mile.

10:13 I'm running in the center lane. No traffic on either side approaching sign for exit. Pick up approaching from the right rear visible now. Missed exit 159A. Car did not move over to exit lane and did not exit.

10:17 truck just merged in front of me from on-ramp. I had to manually slow down because the car would not move over and did not slow down until I was right on his bumper. Reengaged NOA.

10:18 Navigation says Exit, car did not exit 59.

10:18 tried to move over into the center lane with turn signal, the car started moving and then swerved back to the right.

10:19 had to manually take control to get the car off the interstate, U turned and headed south on 59 access road. Restarted NOA. Plan to merge onto 59 from the feeder road. Traveling in the center lane .3 miles before on-ramp. Navigation says to get in left lane car does not move over to the left lane until I physically make it move and get on the access ramp for 59.

10:23 pick-up passed on the left, visible about 5 feet off my bumper.

10:24 truck coming by on left not visible until he's beside me.

10:25 pick-up right behind that truck just now visible. He is almost even with my bumper before I see him on the screen.

Accelerating to 73 miles per hour coming up on the truck that's running a little slower.

10:25 my car is slowing but will not pass the truck. Speed down to 68 miles per hour. Truck approaching in the left rear he's probably 20 feet behind me and he's not visible. Becomes visible as he is even with my bumper.

10:26 Both lanes are now clearer behind me right and left. Car has slowed to 63 miles an hour in 65 mph zone behind the same truck and the car will not pass it.

10:26. Down to 61 miles an hour in a 65 Zone behind the truck with my cruise set at 80 and it will not pass.

10:27 manually change lanes to pass the truck. Car picking up speed. 2 miles to my exit running in the center lane.

10:28 1 mile to the exit at Community Drive that Navigation says I should use to exit HWY 59 South bound. Car is again slowing behind slower traffic at 64 miles per hour and will not pass and will not move over into the right lane to exit. Speed now down to 59 miles an hour behind slower traffic cruise is set at 80 running in navigate on autopilot. Car will not pass the cars in front. 1 miles to exit.

10:29, just passed my exit again. Car made no effort to move into the right lane or to exit.

5:18 pm, on Northpark Dr. Moved to change lanes in front of slow moving car. Car behind me accelerated, causing me to swerve back into my lane. No collision warning, car not visible in display.

Sure sounds like some sort of hardware alignment issue.

Buddy of mine had his Bolt rear ended last December and the blind spot warning sensor alignment and focus material (think old fashioned hearing aid, bull horn feeding an old persons ear) is still not sorted out, giving all sorts of false positives.

 

[drtimhill]

I like stops!
I hate lights (red)
Why?
Because the stop sign is intelligent: if there is no one, I move on.
I dream that the cars respect all the stops.
Thus, we will save money because a fire is much more expensive than a stop sign.

Roundabout. You don’t have to stop at all when traffic is low, which increases efficiency (CO2 etc) as well saving time and wear. They also self0adjust to changes in traffic flow patterns. Disadvantage is they are big and more expensive to deploy.

 

[drtimhill]

The rationale may well be economic, but that does not mean that the task was not extremely technical. In the modern era, the rationale for nearly anything any company does - particularly publicly traded ones - is economic.

I think I already said that.

 

[pongajim]

Obviously Tesla are the only ones who can accurately answer your speculation. However, what rate of improvement did you expect? Upon what was this expectation based? What is "100% success rate" anyway? I'm not being an apologist here, but realistic expectations can only be based upon precedent, and for the kind of things Tesla is doing, there is no precedent. You might as well be critical of Einstein for taking 10 years to develop General Relativity.

Comparisons to rocket science and SpaceX come to mind, and phrases such as "this is [or is not] rocket science". But rocket science is actually a mature engineering discipline compared to some of the things being done by Tesla (and others, of course) with self-driving cars. SpaceX have been truly brilliant is achieving something:. a dramatic reduction in the cost of reliably delivering payloads to space. But this is, at the end of the day, an economic achievement; they have not done anything technically that Apollo etc did not do decades before them.

But Tesla is doing something truly new; they are pushing at the very edges of AI with the various autopilot/self-driving efforts. And guess what? It's hard. Hard as in "it's nearly impossible to predict how long it will take or how far we can get". And let's just ignore Elons "it will be ready on XXX", that's just investor talk.

Tonight I watched my car, by itself, drive along a road with better accuracy than many human drivers, warn me that it was going to stop at an upcoming traffic signal, and then cleanly slow and stop exactly at the signal, before driving on after I tapped the accelerator. Yes, measuring how well the car does, and looking for flaws, and reporting and discussing them, are all good. Very good. But let's not lose sight of how breathtaking the achievements have been, and continue to be.

Excellent comment..!

 

[pongajim]

I get the thrust of what you're trying to say, and I appreciate it. Based on FSD demos, promises of robo-taxis, and claims of feature complete, I was hoping to see some level of NOA improvement (with a good signal to noise ratio) over about a year time span. There was some improvement in NOA, but you have to really squint and tilt your head to see it. For sure there are other improvements in many other areas that Tesla is not showing us yet, and a small handful that have been released.


Fair, he's not good with target dates, but he has a good handle on the technical content. The reliability for FSD has to be really high when it is released to the masses, which is why you hear Karpathy talk about the march of 9's (99%, 99.9%, 99.99%, etc.).


I think that re-entering and landing a rocket ass-end first at the center of the pad counts a new technical achievement.


Peace. It's hard, they're working it, and it will take time. In the mean time, I'm enjoying it, and I can't wait to see their future results.

Landing rear end first has been done before....Reentering + Landing rear end first is new.

 

[Sans-gas]

This is the final video with Enginerd's lovely data...Or is it???

I made this vid using that last drop of Enginerd's data, it's a bit basic so considering making a dedicated reel showing the progress of Enginerds data over the past 12 months, as a bit of a thank you and testimonial. What do y'all think?

I loved this little ride-along. Thanks for sharing. After 2 months of lockdown, it felt like I almost got a little travel in. My big escape these days IS my Tesla M3 drives around the county. I would be insane without my lil’ trips.

 

[Fernand]

FWIW, @ggreen4077 , I can't fully interpret your list, but before your appt, what have you got to lose, try the following to do a proper reset:

1- Dust all the cameras off with a dry microfiber towel, no point in getting the lenses wet unless they are caked. Wipe all the secondary sensors on the car body with a damp microfiber towel and dry.
2- Reset the display computer by holding both scroll wheels until a few seconds after the Tesla logo re-appears.
3- Reset the main computer by setting your phone to airplane mode, tap the Power Off in Security menu, sit in car 5 minutes without moving out of your seat. Car will gradually power down to dead. Wake phone out of airplane mode so the car can be unlocked and ready to drive. Then tap brake to awaken car.

4- Go for a drive and see if things are better.

No matter all the naysayers who think it's "unnecessary", after seeing that it clears many anomalies, I now also do this after every update. It only takes a few minutes, and I've had far FAR fewer issues than most.

Except the one time I forgot to wake the phone and I couldn't silence the alarm until I did
.