Estimate by Lex Fridman at MIT:
Accelerating doublings
As Tesla deliveries accumulate and the rate of Model 3 deliveries accelerates, the number of Enhanced Autopilot miles driven every month will increase exponentially.
Lex estimates that Tesla has delivered about 280,000 HW2 cars. At 312,400 HW2 deliveries, Tesla will hit a run rate of 902 million Enhanced Autopilot miles a year — a doubling since October 2016, when HW2 was launched. At 624,800 HW2 deliveries (which Tesla will probably hit in about 9 months, give or take 3 months), the run rate will double again to 902 million miles every 6 months.
And in another year after that — assuming a steady rate of 10,000 Model 3 deliveries per week — the run rate will double to 902 million miles every 3 months. In 2021, with production starting at the Shanghai factory, the pace of deliveries will increase and the time between doublings will start to shrink.
If the Shanghai factory produces as many cars as the Fremont factory is expected to, then by the end of 2021 the run rate will be 902 million miles every 3 weeks. This is before accounting for any potential increase in usage from new features like Navigate on Autopilot, or general improvements in lane keeping.
Data determinism: considerations for
My assumption is that Tesla will be able to use data collected from cars while Enhanced Autopilot is engaged to improve perception, path planning, and control, as well as to compile HD maps. Perception can be improved by collecting camera images and hand labelling them to train neural networks. (Is radar also trainable? What about ultrasonics?) Path planning and control can be improved by snapshotting disengagements and aborts and using them to either train neural networks or inform human software engineers, as the case may be.
By snapshotting useful moments for billions of miles of driving on Enhanced Autopilot, Tesla can improve Enhanced Autopilot and the perception, path planning, and control subsystems for Full Self-Driving Capability.
Research and expert opinion on deep supervised learning and deep reinforcement learning suggests to me that the primary differentiator or competitive advantage in the performance of neural networks in commercial applications like vehicle autonomy is the size and quality of cleaned, curated, well-labelled data sets.
Data determinism: considerations against
I am unclear on the degree to which open source or academic neural network architectures differ in performance from proprietary architectures on the same data sets. It could be that advantages in AI talent and/or computing resources enables a company like Google to secretly develop proprietary neural network architectures that significantly outperform anything else. But isn’t the design of neural networks increasingly automated? Or are humans still better at it? If you are automating neural network design, doesn’t your algorithm still try to find the optimal architecture for the data set you have? In which case it seems like data sets still might matter most, regardless of how much compute you have.
If neural network architecture isn’t a key differentiator, or if its importance pales in comparison to that of data sets, then Tesla is in good shape. The amount of data that Tesla can choose to sample from is increasing exponentially. Tesla can use the cash flow it’s generating from Model 3 sales to pay for the human labour needed to label that data. Previously, I did some rough back-of-the-envelope math and, depending on a few variables, it looks like manually annotating 1 or 2 billion miles’ worth of footage per year might not be prohibitively expensive. There are probably ways for Tesla to intelligently narrow down the data that would be useful to annotate — I reckon it’s a small fraction of miles driven on Enhanced Autopilot.
One rejoinder is that regardless of how much labelled data you have, you still need the AI talent and good ol’ fashioned software engineering talent to develop an actual, working autonomy backend and product. This is true. Tesla is fortunate in that, according to one survey of people who work in tech, it was the 2nd most attractive tech company for prospective employees in the Bay Area, behind only Google. Tesla was also 4th in the world, behind SpaceX at #1 (which shares personnel and IP with Tesla), Google, and Shopify.
Tesla’s ability to attract Chris Lattner to lead the Autopilot software team is a sign of its ability to pull top talent, even though it didn’t work out. Lattner went to Google Brain after Tesla. Andrej Karpathy seems like a pretty well-respected figure in AI, although perhaps he is respected more as a teacher than a researcher. I’m not sure.
R&D funding is another area where Google has Tesla beat (along with AI talent and compute). Funding is relevant only insofar as what it’s used to pay for: AI talent, software 1.0 engineering talent, compute, manual data labelling, and real world vehicle testing. Labelling is only relevant if Waymo is collecting a higher quantity of useful data than Tesla. The scale of Tesla’s in-house vehicle testing is unknown. I already discussed compute in the context of neural network architecture search; simulation is also relevant, but useful simulation scenarios seem constrained by real world data.
People seem to agree that more software 1.0 engineers don’t make software development faster. The optimum for software 2.0/AI might also be small teams when it comes to designing neural networks or designing neural network search algorithms. Although Google might have an advantage in having several different research teams working on proprietary neural network architectures and network architecture search.
Lidar is its own, long discussion. I’m hoping @verygreen will test Tesla’s camera-based depth estimation.
The cash value
Because I’m an investor, at this point in my thought process I resolve the uncertainty by incorporating actionable financial information. Suppose the global autonomy market is worth $1.5 trillion. Suppose Waymo captures a 50% market share, worth $750 billion. Since Alphabet’s market cap is already $800 billion, that’s an increase of 94%. Suppose Tesla only captures a 7.5% market share, worth $112.5 billion, and all the value of its current businesses is wiped away. That’s a 125% increase. Alphabet, by being so valuable, has a huge financial handicap. As long as Tesla still succeeds in the autonomy market, Waymo can win by a lot in terms of technology and still be a worse investment.
That said, I still lean toward data determinism. The importance of training data for neural network performance is shown by publicly available research and it’s something deep learning experts talk openly about. The superiority of proprietary neural network architectures is just conjecture — a possibility, rather than a known fact.
As the years go by and HW2 Teslas drive billions of miles on Enhanced Autopilot — and Tesla hopefully tries to get maximum usefulness out of the data — it should become increasingly clear what’s true. We can watch Tesla’s progress and Waymo’s progress and see what happens. Elon keeps saying that Tesla’s progress on autonomy will happen exponentially. Well, one of the inputs to progress — driving data available for collection — will certainly increase exponentially. In other contexts like AlphaGo, neural networks have rapidly moved from subhuman performance to superhuman performance — with AlphaGo it happened in three years. Why couldn’t autonomous driving be the same?
Accelerating doublings
As Tesla deliveries accumulate and the rate of Model 3 deliveries accelerates, the number of Enhanced Autopilot miles driven every month will increase exponentially.
Lex estimates that Tesla has delivered about 280,000 HW2 cars. At 312,400 HW2 deliveries, Tesla will hit a run rate of 902 million Enhanced Autopilot miles a year — a doubling since October 2016, when HW2 was launched. At 624,800 HW2 deliveries (which Tesla will probably hit in about 9 months, give or take 3 months), the run rate will double again to 902 million miles every 6 months.
And in another year after that — assuming a steady rate of 10,000 Model 3 deliveries per week — the run rate will double to 902 million miles every 3 months. In 2021, with production starting at the Shanghai factory, the pace of deliveries will increase and the time between doublings will start to shrink.
If the Shanghai factory produces as many cars as the Fremont factory is expected to, then by the end of 2021 the run rate will be 902 million miles every 3 weeks. This is before accounting for any potential increase in usage from new features like Navigate on Autopilot, or general improvements in lane keeping.
Data determinism: considerations for
My assumption is that Tesla will be able to use data collected from cars while Enhanced Autopilot is engaged to improve perception, path planning, and control, as well as to compile HD maps. Perception can be improved by collecting camera images and hand labelling them to train neural networks. (Is radar also trainable? What about ultrasonics?) Path planning and control can be improved by snapshotting disengagements and aborts and using them to either train neural networks or inform human software engineers, as the case may be.
By snapshotting useful moments for billions of miles of driving on Enhanced Autopilot, Tesla can improve Enhanced Autopilot and the perception, path planning, and control subsystems for Full Self-Driving Capability.
Research and expert opinion on deep supervised learning and deep reinforcement learning suggests to me that the primary differentiator or competitive advantage in the performance of neural networks in commercial applications like vehicle autonomy is the size and quality of cleaned, curated, well-labelled data sets.
Data determinism: considerations against
I am unclear on the degree to which open source or academic neural network architectures differ in performance from proprietary architectures on the same data sets. It could be that advantages in AI talent and/or computing resources enables a company like Google to secretly develop proprietary neural network architectures that significantly outperform anything else. But isn’t the design of neural networks increasingly automated? Or are humans still better at it? If you are automating neural network design, doesn’t your algorithm still try to find the optimal architecture for the data set you have? In which case it seems like data sets still might matter most, regardless of how much compute you have.
If neural network architecture isn’t a key differentiator, or if its importance pales in comparison to that of data sets, then Tesla is in good shape. The amount of data that Tesla can choose to sample from is increasing exponentially. Tesla can use the cash flow it’s generating from Model 3 sales to pay for the human labour needed to label that data. Previously, I did some rough back-of-the-envelope math and, depending on a few variables, it looks like manually annotating 1 or 2 billion miles’ worth of footage per year might not be prohibitively expensive. There are probably ways for Tesla to intelligently narrow down the data that would be useful to annotate — I reckon it’s a small fraction of miles driven on Enhanced Autopilot.
One rejoinder is that regardless of how much labelled data you have, you still need the AI talent and good ol’ fashioned software engineering talent to develop an actual, working autonomy backend and product. This is true. Tesla is fortunate in that, according to one survey of people who work in tech, it was the 2nd most attractive tech company for prospective employees in the Bay Area, behind only Google. Tesla was also 4th in the world, behind SpaceX at #1 (which shares personnel and IP with Tesla), Google, and Shopify.
Tesla’s ability to attract Chris Lattner to lead the Autopilot software team is a sign of its ability to pull top talent, even though it didn’t work out. Lattner went to Google Brain after Tesla. Andrej Karpathy seems like a pretty well-respected figure in AI, although perhaps he is respected more as a teacher than a researcher. I’m not sure.
R&D funding is another area where Google has Tesla beat (along with AI talent and compute). Funding is relevant only insofar as what it’s used to pay for: AI talent, software 1.0 engineering talent, compute, manual data labelling, and real world vehicle testing. Labelling is only relevant if Waymo is collecting a higher quantity of useful data than Tesla. The scale of Tesla’s in-house vehicle testing is unknown. I already discussed compute in the context of neural network architecture search; simulation is also relevant, but useful simulation scenarios seem constrained by real world data.
People seem to agree that more software 1.0 engineers don’t make software development faster. The optimum for software 2.0/AI might also be small teams when it comes to designing neural networks or designing neural network search algorithms. Although Google might have an advantage in having several different research teams working on proprietary neural network architectures and network architecture search.
Lidar is its own, long discussion. I’m hoping @verygreen will test Tesla’s camera-based depth estimation.
The cash value
Because I’m an investor, at this point in my thought process I resolve the uncertainty by incorporating actionable financial information. Suppose the global autonomy market is worth $1.5 trillion. Suppose Waymo captures a 50% market share, worth $750 billion. Since Alphabet’s market cap is already $800 billion, that’s an increase of 94%. Suppose Tesla only captures a 7.5% market share, worth $112.5 billion, and all the value of its current businesses is wiped away. That’s a 125% increase. Alphabet, by being so valuable, has a huge financial handicap. As long as Tesla still succeeds in the autonomy market, Waymo can win by a lot in terms of technology and still be a worse investment.
That said, I still lean toward data determinism. The importance of training data for neural network performance is shown by publicly available research and it’s something deep learning experts talk openly about. The superiority of proprietary neural network architectures is just conjecture — a possibility, rather than a known fact.
As the years go by and HW2 Teslas drive billions of miles on Enhanced Autopilot — and Tesla hopefully tries to get maximum usefulness out of the data — it should become increasingly clear what’s true. We can watch Tesla’s progress and Waymo’s progress and see what happens. Elon keeps saying that Tesla’s progress on autonomy will happen exponentially. Well, one of the inputs to progress — driving data available for collection — will certainly increase exponentially. In other contexts like AlphaGo, neural networks have rapidly moved from subhuman performance to superhuman performance — with AlphaGo it happened in three years. Why couldn’t autonomous driving be the same?
