There are now two published comparison tests which pit Tesla's Autopilot against the best of competing systems - and seem to point to rapid improvement in Tesla's "fleet learning" just like Musk promised the world when it was first released in October.
Significance - these two tests pit an evolving Autopilot against the same generation of Mercedes' Drive Pilot
Car and Driver ran their comparison test using Tesla's Autopilot 7.0 back in December. Two days ago Motor Trend published a new comparison test using 7.1. Yet both comparisons use the same 2016 version of Mercedes' Drive Pilot. This may be the only chance we will have to see how/whether Musk's promise of fleet learning has come true - because there are no other published tests trying to measure Autopilot error rates - especially none comparing against a competing automobile which has not changed over time. And remember - Mobileye has said in several presentations that EyeQ3 - the SOC Tesla currently uses - is the first deployment of a system which has the capability to train neural networks on the fly in actual fleet units. Nobody knows if that's marketing hogwash - but here at least we have some comparison snapshots at different points in time.
Now - I realize these were done with different testers, on different routes, using different methodology. Yes it's a highly flawed data set here but it's all we've got and it's fun to look at.
The punchline - Tesla appears to have dropped its error rate against Mercedes significantly
Disclaimer: I know this is not scientific. I too have taken enough lab courses to know this comparison is perhaps meaningless - enough to generate a hypothesis about improvement but nothing more. I wish Car and Driver would repeat their earlier test on the same route with 7.1 - or perhaps 8.0 next month. Having said that - it seems that Tesla's early build of 7.0 appeared to make 50% fewer mistakes than Mercedes' system last December. This July's Motor Trend test shows Autopilot 7.1 making 90% fewer mistakes than the same Mercedes system.
Mercedes degrades five times more than Tesla on poorly marked two lane rural roads - Autopilot 7.1 July 2016
Motor Trend gave us results split between well marked highways and poorly marked two lane rural roads, whereas Car and Driver's earlier test is a combined route and they did not publish data showing performance by "road type."
What Motor Trend's results seem to show is that while Tesla is always far more accurate than Mercedes - Tesla REALLY has Benz beat on the more difficult roads. Mercedes' error rate per mile increased 66% on Santiago Canyon vs Highway 241 - whereas Tesla's error rate increased only 12%. The following screenshot is my calculation
Test 1 - Car and Driver February 2016 Issue
The February issue of Car and Driver published the first autopilot comparison test done by any car magazine - I think that a February publish date means the test itself was conducted in December - remember that 7.1 was not released until January.
The test pitted Tesla vs MBZ vs BMW vs Infiniti on a 50 mile loop in Michigan - a mix of rural, city and freeway driving with plenty of "interrupted lane markings and patched pavement."
In C&D's words: "Using a 50-mile mix of freeway stretches, rural two-lanes, and city streets, we tabulated exactly how many guidance interruptions were caused by broken lane marks, inconsistent pavement patches, intersections, and exit and entrance ramps. We also noted when a car lost the lane-keeping sense for no apparent reason. Then we ranked the four contenders according to the number of control lapses each test car experienced."
Test 2 - Motor Trend July 2016
Six months later, Motor Trend ran a similar comparison test on a mix of roads in Southern California - which they also claimed included a tough rural road comparison section with shadows, poor lane markings etc. The results in this test were even more dramatic than Car and Driver's - instead of having half as many "wheel touch" incidents as Mercedes, the Tesla showed over 90% fewer:
Significance - these two tests pit an evolving Autopilot against the same generation of Mercedes' Drive Pilot
Car and Driver ran their comparison test using Tesla's Autopilot 7.0 back in December. Two days ago Motor Trend published a new comparison test using 7.1. Yet both comparisons use the same 2016 version of Mercedes' Drive Pilot. This may be the only chance we will have to see how/whether Musk's promise of fleet learning has come true - because there are no other published tests trying to measure Autopilot error rates - especially none comparing against a competing automobile which has not changed over time. And remember - Mobileye has said in several presentations that EyeQ3 - the SOC Tesla currently uses - is the first deployment of a system which has the capability to train neural networks on the fly in actual fleet units. Nobody knows if that's marketing hogwash - but here at least we have some comparison snapshots at different points in time.
Now - I realize these were done with different testers, on different routes, using different methodology. Yes it's a highly flawed data set here but it's all we've got and it's fun to look at.
The punchline - Tesla appears to have dropped its error rate against Mercedes significantly
Disclaimer: I know this is not scientific. I too have taken enough lab courses to know this comparison is perhaps meaningless - enough to generate a hypothesis about improvement but nothing more. I wish Car and Driver would repeat their earlier test on the same route with 7.1 - or perhaps 8.0 next month. Having said that - it seems that Tesla's early build of 7.0 appeared to make 50% fewer mistakes than Mercedes' system last December. This July's Motor Trend test shows Autopilot 7.1 making 90% fewer mistakes than the same Mercedes system.
Mercedes degrades five times more than Tesla on poorly marked two lane rural roads - Autopilot 7.1 July 2016
Motor Trend gave us results split between well marked highways and poorly marked two lane rural roads, whereas Car and Driver's earlier test is a combined route and they did not publish data showing performance by "road type."
What Motor Trend's results seem to show is that while Tesla is always far more accurate than Mercedes - Tesla REALLY has Benz beat on the more difficult roads. Mercedes' error rate per mile increased 66% on Santiago Canyon vs Highway 241 - whereas Tesla's error rate increased only 12%. The following screenshot is my calculation
Test 1 - Car and Driver February 2016 Issue
The February issue of Car and Driver published the first autopilot comparison test done by any car magazine - I think that a February publish date means the test itself was conducted in December - remember that 7.1 was not released until January.
The test pitted Tesla vs MBZ vs BMW vs Infiniti on a 50 mile loop in Michigan - a mix of rural, city and freeway driving with plenty of "interrupted lane markings and patched pavement."
In C&D's words: "Using a 50-mile mix of freeway stretches, rural two-lanes, and city streets, we tabulated exactly how many guidance interruptions were caused by broken lane marks, inconsistent pavement patches, intersections, and exit and entrance ramps. We also noted when a car lost the lane-keeping sense for no apparent reason. Then we ranked the four contenders according to the number of control lapses each test car experienced."
Test 2 - Motor Trend July 2016
Six months later, Motor Trend ran a similar comparison test on a mix of roads in Southern California - which they also claimed included a tough rural road comparison section with shadows, poor lane markings etc. The results in this test were even more dramatic than Car and Driver's - instead of having half as many "wheel touch" incidents as Mercedes, the Tesla showed over 90% fewer:
Attachments
Last edited:
