OK, you've convinced me to make one more point which I had in the back of my head.
I think until the how-to-drive problem is properly specified, which it has not been yet, we won't know what the neural network needs to be able to identify, so we won't know how to start working on the NN training.
Example already brought up: it needs to know when it is approaching a blind curve. This is a pretty complex 3d modeling problem -- not just about modeling the things you can see, but the implications of empty space which you can't see. Example from the other end: identifying that there's someone else on the other side of the blind curve requires listening for their horn. (I don't think Tesla's system even has the microphone pickup yet.) Another example: identifying the type of road surface (dirt, gravel, grass) is actually quite important for certain driving tasks.
Until the specification of the how-to-drive problem is nailed down, we don't actually know what the neural network needs to be able to identify. I fully expect that when the how-to-drive problem is being seriously worked on, it will end up requiring a basically-from-scratch retraining of the neural network because it'll have to pick out data which they didn't previously realize was necessary.
This is the problem with not understanding what problem you're working on.
(For this reason, I think Tesla's existing data collection isn't worth much of anything. Having the largest fleet of cars on the road to collect data with will, however, be worth quite a lot sometime in the future when they actually have a self-driving problem specification -- they'll be able to do that training stage much quicker than anyone who doesn't have millions of instrumented cars on the road. They will, however, be starting from scratch at that point.)
