I found this to be one of the best explanations of the problems of autonomous driving: Hotz describes the 3 problems of autonomous driving: 1) Static Driving This is the basic level where the car just navigates a static map with no other moving objects. At this stage, you are developing path finding and developing the controls for steering and braking. Hotz says this stage is now "easy" if you have sensor fusion that includes LIDAR and HD maps. 2) Dynamic Driving This is stage 2 where you add moving objects (cars, pedestrians). Now, your car needs to navigate a path like in static driving but also avoid hitting anything. This stage is more difficult. You need to track and predict where objects will be in real-time and make sure you are not on a collision course. When you finish this stage, your autonomous vehicle can drive on public roads but it will be very robotic and may not react to other driver's actions correctly. 3) Counterfactual driving This is the final stage where you factor in the sometimes unpredictable nature of human driving. it is the most difficult stage because it requires intelligent thinking. You have to anticipate what other drivers might do. Will that car try to cut me off? You can't solve it with hard coded rules. A rule that works in one driving scenario may not work in another. But, if you can solve this stage of autonomous driving, then you would have a truly L5 autonomous vehicle that could pass for a human driver. Hope this is helpful.