If this is being implemented, it'll help a lot. (No evidence of implementation yet.) I know people who, facing weird situations, have simply decided "time to pull over and call the cops for a rescue". If this is implemented, they could get to pseudo-full self-driving much quicker. (Again, no signs that they're doing this yet.)
Yep. I look forward to the announcement where they say they're doing that. They didn't mention it at autonomy day, therefore they're not doing it yet.
Based on Karpathy's talk? I'd say they're just starting to descend from Mt. Stupid. It'll take a while. (Everyone else is still CLIMBING Mt. Stupid, so Tesla's still ahead.)
I do think there is evidence that they are taking this sort of approach. I've seen it with AP1. Whenever AP1 gets confused about where the road is it disengages and cedes to control of the driver. Moreover, as time progressed, the frequency of these disengagements decreased. Now this is in the context of having a driver to hand over control to, but it does demonstrate the principle. Clearly Tesla is thinking about all sorts of redundancy to assure high reliability. So I've got to believe they have some sort of fall back responses when the NN is highly uncertain about how to proceed. With FSD they can't cede control to a driver, but they can slow to a stop.
Also I think the example of the bike on the back of a car is an example of this sort of strategy. They recognized that the NN was being confused about bicycles appearing to cross the road when in fact they were strapped onto a car. What alerted them to this problem? I should hope that it was discovered by diagnostic routines. Hopefully, the system was correctly alert to the potential for a bike or pedestrian to cross the lane of travel. That would be a true positive. But then the NN notices that drivers routinely drive toward "bikes crossing the road" in some cases. These would be the false positives where the bikes are actually strapped on to the car ahead. They'd also have visual confusion about an object that looks like it could be a car or a bike. If they are were doing this right, the cars would be taking a defensive response to these false positives. Then there would be diagnostic routines to review these defensive responses. Once the problem has thus been identified, the NN can be trained to accurately observe that a bike is attached to a vehicle and thus resolve the false positive. So in this instance, I think Tesla was rising on the Slope of Enlightenment.
Now you've pointed out that a bike strapped on to a car can in fact fall off. So yes, there does still exist some really risk in following such a vehicle. So this goes to setting a prudent following distances in such a case. If the case is well identified, i.e., the vehicle can perceive when a bike is attached to a vehicle. Then they are also in a position to tell when it in not attached to a vehicle. So bikes falling off a vehicles are probably already identified events which the NN can learn from. Ideally the NN is recalibrating to allow a suitable distance or to consider changing lanes. But all this is part of the general problems of how much following distance to allow and when to change lanes. The more gritty work was just to train the net to see when bikes are attached and when they are not.
So I am much more optimistic about where Tesla is on FSD. There have already been years where AP was just not making much apparent progress. Remember when a car was supposed to solo across the US? That did not happen. Why? I think that was when research was in the Valley of Despair. They had had enough experience with AP to know that it was really hard to avoid having to cede control to the driver or to have the driver actively disengage. I think that's how they climbed back down Mt. Stupidity. So I think AP was Mt. Stupidity, but that was okay because there was still a driver at the wheel to handle the stupidity. I know I did my part. There was one power pole in particular that AP just wanted to drive right into. So I had to take control every time. I sure hope that somewhere in Tesla this data was being analyzed to understand just what weird thing was going on. Even so, I was doing my part to tell the car, "Don't be stupid. That's not the way to go." I'm also encouraged by shadow mode. Every time a driver takes an evasive action, this should trigger data. The NN should be trained to predict when a driver is about to take an evasive action. This is one way to detect false negatives, the situation where the driver is correctly responding to a hazard, but the NN has heretofore failed to detect that hazard. By simultaneously comparing driver behavior to the action that the NN would have taken in that situation, the system can detect a kind of cognitive dissidence. Specifically there is a substantial disconnect between how the driver and NN interpret a situation. So this should signal for some sort of diagnostic work. Of course, sometimes drivers do stupid things where the NN would not have done the same stupid thing. But where there is substantial disagreement between driver and NN, there is the potential for the NN to learn something important. When the driver takes evasive action, either the NN has a false negative, the driver has a false positive or perhaps both. Eventually, the NN will be able to critique the driver and point out where the driver had false positives or false negatives. (Imaging the NN reporting this analysis of driver behavior back to one's insurance carrier!) At any rate, I think there is huge potential to identify edge cases, especially false negatives by having NN analyze human driver behavior. I think a lot of this problem detection can be done analytically. I suspect that hand work is needed more to resolve the false negatives or false positives where feature extraction is underdeveloped. So long as the system can respond with caution where problems have been detected or where drivers and the NN are likely to diverge, you go along way toward eliminating the possibilities for false negatives. This still can leave you with an abundance of false positives to resolve, but you have more time to work that out.
You've got to climb down Mt. Stupidity (detecting false negatives) very fast, but can take more time climbing the Slope of Enlightenment (resolving false positives). I think they've got multiple lines of attack on both fronts. It a big tool box. What is harder to assess is just how effectively they are using these tools. No doubt they are still on their own learning curve around how to best to cultivate the NN. But I do suspect they are well down the path where FSD needs very little supervision from drivers. To wit, if the NN can predict when drivers are about to take evasive action and act faster than human response, there should be no need for the driver to take that action, as the NN can beat them too it. You actually get to a moral problem where in shadow mode FSD passively allows drivers to make mistakes. No doubt some drivers will make lethal mistakes that NN would have calculate to be such even before the driver commits them.
There is an analogy here to NN that have been trained to play chess by estimating the probability of winning with each play made. Such a system would be able to watch a human player and know when they have elected to make a move that sends their probably to winning to zero. The NN could tell the player, "Don't be stupid. That play will cost you the game." It's all well and good for a NN to watch a human player lose a game of chess, and it may even learn something in the process. But what about when a NN knows that a human driver will imperil lives? So as NN descend Mt. Stupid, it gains the ability to critique its own driving and that of human drivers. At some point we look back, and NN will show us how much time even the best human drivers stumbling about on Mt. Stupidity. We'll even have little black boxes that bear witness to the final mistakes that a human driver makes. For me, I think this lends some insight into why Musk is so serious about how people will eventually want to ban driving without computer assistance. Shadow mode probably gives you a very dim view of human drivers even as it illuminates how FSD can avoid making the same mistakes over and over again.
