Wiki Tracking FSD Feature Complete

[diplomat33]

It seems they have only one main branch where they are developing FSD features - but they are turned off unless you are in dev mode. That is what @verygreen is essentially saying.

The fact that all the features shown in the autonomy demo can be recreated by hackers, I think proves the point.

So you are saying that the software that we have now in our cars is exactly the same as the software in development, just with some features turned off? In other words, Tesla has nothing in development right now that is not also in our cars as well? Somehow I doubt that.

And I watched the video again. AP is turned off at the intersection whereas the FSD demo did show NOA handling the intersection. So the hacker was not able to completely recreate the entire FSD demo. That's because the FSD demo had additional "intersection" software that our cars don't have yet. Unless I misread the video?

 

[EVNow]

So you are saying that the software that we have now in our cars is exactly the same as the software in development, just with some features turned off? In other words, Tesla has nothing in development right now that is not also in our cars as well? Somehow I doubt that.

I think they are merging back the dev branch frequently with the production branch. Otherwise we shouldn't see all the features we saw in the demo, in the hacked software as well.

And I watched the video again. AP is turned off at the intersection whereas the FSD demo did show NOA handling the intersection. So the hacker was not able to completely recreate the entire FSD demo. That's because the FSD demo had additional "intersection" software that our cars don't have yet. Unless I misread the video?

Apparently the turning was not good (either too sharp or not sharp enough) - so he was turning it off at intersections. There are a couple of videos where he did let it turn on its own.

 

[diplomat33]

I think they are merging back the dev branch frequently with the production branch. Otherwise we shouldn't see all the features we saw in the demo, in the hacked software as well.


Apparently the turning was not good (either too sharp or not sharp enough) - so he was turning it off at intersections. There are a couple of videos where he did let it turn on its own.

Thanks. Very helpful.

 

[EVNow]

Thanks. Very helpful.

Another explanation for having all the FSD dev code fleetwide but disabled is to collect data in the "shadow mode". While shadow mode potential is huge, not sure what they actually do.

 

[JusRelax]

 

[diplomat33]

Note sure if this has been mentioned already but here is the list of self-driving behaviors that Waymo cars are already competent in (taken from Waymo Safety Report 2018, Appendix A on page 36-37). I feel like this would be a good list to use to judge Tesla's FSD progress:

Set of Behavioral Competencies Recommended by NHTSA
1 Detect and Respond to Speed Limit Changes and Speed Advisories
2 Perform High-Speed Merge (e.g., Freeway)
3 Perform Low-Speed Merge
4 Move Out of the Travel Lane and Park (e.g., to the Shoulder for Minimal Risk)
5 Detect and Respond to Encroaching Oncoming Vehicles
6 Detect Passing and No Passing Zones and Perform Passing Maneuvers
7 Perform Car Following (Including Stop and Go)
8 Detect and Respond to Stopped Vehicles
9 Detect and Respond to Lane Changes
10 Detect and Respond to Static Obstacles in the Path of the Vehicle
11 Detect Traffic Signals and Stop/Yield Signs
12 Respond to Traffic Signals and Stop/Yield Signs
13 Navigate Intersections and Perform Turns
14 Navigate Roundabouts
15 Navigate a Parking Lot and Locate Spaces
16 Detect and Respond to Access Restrictions (One-Way, No Turn, Ramps, etc.)
17 Detect and Respond to Work Zones and People Directing Traffic in Unplanned or Planned Events
18 Make Appropriate Right-of-Way Decisions
19 Follow Local and State Driving Laws
20 Follow Police/First Responder Controlling Traffic (Overriding or Acting as Traffic Control Device)
21 Follow Construction Zone Workers Controlling Traffic Patterns (Slow/Stop Sign Holders)
22 Respond to Citizens Directing Traffic After a Crash
23 Detect and Respond to Temporary Traffic Control Devices
24 Detect and Respond to Emergency Vehicles
25 Yield for Law Enforcement, EMT, Fire, and Other Emergency Vehicles at Intersections, Junctions, and Other Traffic Controlled Situations
26 Yield to Pedestrians and Bicyclists at Intersections and Crosswalks
27 Provide Safe Distance From Vehicles, Pedestrians, Bicyclists on Side of the Road
28 Detect/Respond to Detours and/or Other Temporary Changes in Traffic Patterns

Examples of Additional Behavioral Competencies Tested by Waymo
29 Moving to a Minimum Risk Condition When Exiting the Travel Lane is Not Possible
30 Perform Lane Changes
31 Detect and Respond to Lead Vehicle
32 Detect and Respond to a Merging Vehicle
33 Detect and Respond to Pedestrians in Road (Not Walking Through Intersection or Crosswalk)
34 Provide Safe Distance from Bicyclists Traveling on Road (With or Without Bike Lane)
35 Detect and Respond to Animals
36 Detect and Respond to Motorcyclists
37 Detect and Respond to School Buses
38 Navigate Around Unexpected Road Closures (e.g. Lane, Intersection, etc.) 39 Navigate Railroad Crossings
40 Make Appropriate Reversing Maneuvers
41 Detect and Respond to Vehicle Control Loss (e.g. reduced road friction)
42 Detect and Respond to Conditions Involving Vehicle, System, or Component-Level Failures or Faults (e.g. power failure, sensing failure, sensing obstruction, computing failure, fault handling or response)
43 Detect and Respond to Unanticipated Weather or Lighting Conditions Outside of Vehicle’s Capability (e.g. rainstorm)
44 Detect and Respond to Unanticipated Lighting Conditions (e.g. power outages)
45 Detect and Respond to Non-Collision Safety Situations (e.g. vehicle doors ajar)
46 Detect and Respond to Faded or Missing Roadway Markings or Signage 47 Detect and Respond to Vehicles Parking in the Roadway

This would be a pretty good list for "feature complete". At a minimum, I think Tesla needs to do 1-28 since they are recommended by the NHTSA. Hopefully, Tesla is paying attention to these recommendations by the NHTSA to help guide their development goals.

 

[willow_hiller]

That's a great list, @diplomat33 .

It seems like quite a mountain to climb, but at the same time I'm realizing that in addition to the "what" of feature complete, the industry seems to be missing a good sense of "how well." For example, NOA might check the box for "High -Speed merge" but at the moment we don't seem to have a measure by which to quantify how well an autonomous system needs to perform a merge before that feature is considered complete.

 

[diplomat33]

That's a great list, @diplomat33 .

It seems like quite a mountain to climb, but at the same time I'm realizing that in addition to the "what" of feature complete, the industry seems to be missing a good sense of "how well." For example, NOA might check the box for "High -Speed merge" but at the moment we don't seem to have a measure by which to quantify how well an autonomous system needs to perform a merge before that feature is considered complete.

Thanks. And yes, you make a great point about "how well". It's like that TV ad "just ok is not good enough". Certainly, any autonomous vehicle needs to be better than "just ok" in these features.

In terms of measuring the "how well", I think that is where disengagement reports and Tesla's fleet data come in. For a feature like "high speed merges", Tesla can collect and analyse thousands of real world examples per day from the fleet to measure how good NOA is at that feature and then use that data to improve the feature. And I don't know what the exact number might be, but presumably we could quantify the "how well" by measuring disengagements during that feature. For example, "high speed merges interrupted by disengagement" divided by "total number of high speed merges".

Yes, the list does seem quite daunting. In fact, it might explain why Waymo has not released their cars in full robotaxi mode yet. As impressive as this list might seem, Waymo probably knows that it is not a complete list and there are probably still behaviors that their cars lack. Plus, Waymo cars may still want to get even better at some of the features on their list. Two reasons why Waymo would not consider their cars ready yet for full deployment without a driver.

 

[willow_hiller]

Thanks. And yes, you make a great point about "how well". It's like that TV ad "just ok is not good enough". Certainly, any autonomous vehicle needs to be better than "just ok" in these features.

In terms of measuring the "how well", I think that is where disengagement reports and Tesla's fleet data come in. For a feature like "high speed merges", Tesla can collect and analyse thousands of real world examples per day from the fleet to measure how good NOA is at that feature and then use that data to improve the feature. And I don't know what the exact number might be, but presumably we could quantify the "how well" by measuring disengagements during that feature. For example, "high speed merges interrupted by disengagement" divided by "total number of high speed merges".

Yes, the list does seem quite daunting. In fact, it might explain why Waymo has not released their cars in full robotaxi mode yet. As impressive as this list might seem, Waymo probably knows that it is not a complete list and there are probably still behaviors that their cars lack. Plus, Waymo cars may still want to get even better at some of the features on their list. Two reasons why Waymo would not consider their cars ready yet for full deployment without a driver.

I agree there would be one denominator (total number of maneuvers attempted). I would be interested in two different numerators, however. First, as you suggested "maneuvers interrupted by disengagement," but maybe secondly "maneuvers resulting in a collision." I think regulators would be willing to stomach a high number of disengagements in general if the vehicle always safely pulled over to the side of the road; if the data show the technology is causing injury or damage to property, on the other hand, that's a different story.

 

[EVNow]

Note sure if this has been mentioned already but here is the list of self-driving behaviors that Waymo cars are already competent in (taken from Waymo Safety Report 2018, Appendix A on page 36-37). I feel like this would be a good list to use to judge Tesla's FSD progress:

This is great. I was thinking about asking whether anyone knows the list of features Waymo has implemented !

I'll split these into specific & platform features and "edge" cases and updated the first post.

I think a generic way to track the features would be to turn this thread into a Feature parity with Waymo/Cruise thread (may be once Tesla announces FC).

 

[EVNow]

That's a great list, @diplomat33 .

It seems like quite a mountain to climb, but at the same time I'm realizing that in addition to the "what" of feature complete, the industry seems to be missing a good sense of "how well." For example, NOA might check the box for "High -Speed merge" but at the moment we don't seem to have a measure by which to quantify how well an autonomous system needs to perform a merge before that feature is considered complete.

2x better than humans ?

For eg., Tesla is claiming they have done x # of lane changes (in millions) without a crash. I think in lane changes (may be merges too) cars have an inherent advantage, since they see all sides all the time. Their reaction time can be better too.

ps : Interesting thing about the Waymo report is they are saying they test all these competencies - but don't say how well they do in them. They could have said how many such situations happened in 2018, for eg. The problem is Waymo, Cruise and others have such puny numbers compared to Tesla. Waymo says they completed 10 M miles of Av testing, Tesla says the fleet has done 1B miles of AP !

 

[diplomat33]

This is great. I was thinking about asking whether anyone knows the list of features Waymo has implemented !

I'll split these into specific & platform features and "edge" cases and updated the first post.

I think a generic way to handle this would be to turn this thread into a Feature parity with Waymo thread (may be once Tesla announces FC).

Glad I could help.

We could do a table like this to keep track of Tesla's progress:

 

[EVNow]

Glad I could help.

We could do a table like this to keep track of Tesla's progress:

View attachment 423208

I don't like the way those features / capabilities are … not organized. So, I'll do something similar to what I have now, but expand the capabilities to include all those listed features.

 

[EVNow]

@diplomat33 I've now added all the NHTSA and Waymo extra features, under appropriate headings. Some of them are repetitions of what I already had in more detail, but I added them for completeness.

Does anyone know if Cruise also publishes something similar ? I guess they are doing all the things NHTSA is recommending and may be some more of their own.

 

[EinSV]

@EVNow great idea for a thread. One suggestion: the color scheme is not intuitive at all, which makes the chart unnecessarily hard to understand.

Maybe adopt the automotive "standard" of red-yellow-green with "green" meaning "feature complete," red meaning "clearly not complete" and yellow somewhere in between.

Readers could then quickly and easily process the info in the chart.

 

[EVNow]

@EVNow great idea for a thread. One suggestion: the color scheme is not intuitive at all, which makes the chart unnecessarily hard to understand.

Maybe adopt the automotive "standard" of red-yellow-green with "green" meaning "feature complete," red meaning "clearly not complete" and yellow somewhere in between.

Readers could then quickly and easily process the info in the chart.

Thanks, something to consider. I've 4 levels of "done" - so used 4 green saturations. Not done is light orange. May be I'll include a legend or turn 3 levels of pre-wide release "done" to yellow.

 

[diplomat33]

@diplomat33 I've now added all the NHTSA and Waymo extra features, under appropriate headings. Some of them are repetitions of what I already had in more detail, but I added them for completeness.

Does anyone know if Cruise also publishes something similar ? I guess they are doing all the things NHTSA is recommending and may be some more of their own.

Thanks for adding the NHTSA and Waymo features.

 

[diplomat33]

@diplomat33 I've now added all the NHTSA and Waymo extra features, under appropriate headings. Some of them are repetitions of what I already had in more detail, but I added them for completeness.

Isn't feature 2.3 "perform car following (including stop and go)" the same as what TACC does? So, I would think AP already does that now.

 

[EVNow]

@EinSV I've updated the color scheme.

@diplomat33 That's correct. I've updated 2.3. Also, I was wondering whether "merge" (2.6) for city roads is done. I use AP in one city road where there is a merge, but not often enough to figure out whether it is aware of other cars and handles them.

 

[diplomat33]

@diplomat33 That's correct. I've updated 2.3. Also, I was wondering whether "merge" (2.6) for city roads is done. I use AP in one city road where there is a merge, but not often enough to figure out whether it is aware of other cars and handles them.

I am not sure about merging on city streets.