It really just depends on the roads. With clear lane markings and minimal hills-in-curves, AP2 does pretty well on rural roads. Around here, our back roads have a lot of elevation change, and a lot of those in curves. AP2 just either gives up or makes really dumb decisions (my best example is a place where AP2 still comes out of the curve in and locked on to the lane of oncoming traffic, if I let it). AP1 usually makes the effort, and at reasonable speeds for the curve will usually make it happen. Definitely a YMMV situation.
@wk057 To your point about the visualization of the car 'swimming' or knocking into each other, please realize that this is just Tesla trying to dumb down the image processing algorithm's data for our perusal. The actual data being generated and reacted upon is exceptionally accurate to the last inch. Since a lot of sensor fusion is happening (ultrasound/camera), it is impossible to show all that data accurately on the screen for the driver. Considering all the variables here, to me it's a surprise Tesla even bother with the visual representation. Because buyers normally dont look past what is directly in front of them and misconstrue that as the software not working.
Just few months ago AP1 used to false positive brake on an overpass -- every single time. Now it is doesn't do it, ever. I don't know if it was a whitelist improvement, or an actual design improvement, but it is an improvement. I also think it takes sharper turns with more precision and likelihood at staying within lane lines on sharp curves. I also think it has gotten better at dealing with going over hills -- hopefully it will catch up to AP2 which doesn't have those problems. That is really bad. All driver assistance tech should build in -- stay out of the left lanes except to pass. Each post should stand on its own terms. And you can count on new readers coming to this forum and we can't reasonably expect them to do dui diilgence on each poster. Your post seemed unnecessarily negative. You could have described the exact same experience of NavOnAP even pointing out its faults and areas for further improvement, without all that negativity. it would have been more effective at conveying the information both good and bad and it would have been more constructive. but not really a big deal. I do the same thing all the time but try to be better.
It's so weird because I would say our experience is the opposite on those examples too. With limited paint striping, our AP2.5 car shines in comparison to our AP1. It will engage in tougher situations (less paint) and stay engaged longer. And on roads with rolling hills and curves, the AP2.5 car is superior also. Far less ping-ponging going over rises. Seems a lot more confident. AP2.5 for sure centers up way better on roads that only have a center stripe. I can't even use our AP1 car on such roads as it hugs the right side of the road too much and ends up hitting breaks (potholes) in the asphalt on the side of the road. Learned this the hard way with AP1.
Did you ever tried initiating one/confirming it? It has always done this but if you do enable the lance change/confirm it it will wait for the vehicle to move (sometimes it will break to get behind him or sometimes it will get in front) the lane will only change color once you initiate the lane change confirmation. in around 500 miles it has never moved me into a lane with a vehicle. I had the opposite refuse to switch lane once because it saw a large semi in my left lane but it was 2 lanes to the side a second or two later realized it was not really there and then it started the lane change. This is pulled from either the map database speed signs(which as you mentioned may be stale) or map database road type or history of the driver. If you always set it to 53 on a 55 road it seems to remember for that road type and next time you engage it on a 55 zone it goes back to the limit you last had configured. (I have the suspicion that is also remembering it based on specific geographic road areas) but haven't measured it correctly enough to be able to confirm it. I hope this and/or stop and traffic lights would be next unless they ran out of processing power and need their new asic/tpu ap3. This seems to be based on what's rendered and not what it's actually acted upon soo hard to know for sure. Maybe you can get into the firmware and print the real readings from all the vehicles around it instead of the rendered ones?
Thanks for the report Jason. Makes me a bit worried that they'd ship something like this to customers. Hopefully soon remedied? Could it be an issue with some specific combination of your particular hardware in combination with some specific iteration of code?
Overall, I disagree. While I believe visualizing the data may prove moderately difficult, it's not *that* difficult. Keep in mind, AP1 does this just fine, and AP2 has had the same issue even before bringing more cameras online. So, I don't buy this particular logic. I have copies of firmwares from initial AP1 release to present. I can assure you, nothing has changed with the core of AP1 in nearly 2 years. The overpass false positive improvement is in fact a whitelist situation, as Tesla is still processing "ADAS tiles" for both systems. The other "improvements" you note are, frankly, your imagination... most likely other circumstances impacting the performance of the system (shadows/sun position, windshield cleanliness, etc). I actually found that AP works significantly better in areas when nearby trees have leaves vs when they don't, likely due to the huge variations in shadowing. Agreed, which is why this was a bit confusing to me to find this behavior. I'll give this the benefit of appearing to be fair criticism. Unfortunately there's just no way to spin this in a positive manner. The post is about the failings of Tesla, their AP teams, and NavOnAP in general. Not much neutral or positive spin to those things.
huh, even on the model 3? I thought their different setup makes it immune, but I guess not? anyway I mostly agree with your experience to the point that I am not even bothering using this feature lately.
did you try tail of the dragon on ap1? I did yesterday and was super underwhelmed by all the errors so fast I needed to abandon the idea entirely (ap footage of that forthcoming shortly), but might be interesting to compare to ap1
Could you point me to the post @wk057 was referencing regarding this rear camera issue you discovered? Thanks, Jeff
This likely has the most detailed logs (and other posts in this thread), but I've been saying about this in several other threads lately. Autopilot in 360 Degrees
First, thanks for your informative post and accounting of issues you found. I'm not nearly as knowledgeable about the inner workings of AP V42 nor Tesla SW in general and I find your posts very useful so, please continue informing us here. I just completed 3 trips with AP V42.2 on my '17X (V2.0 HW) and '17 3 (V2.5 HW). So my recent experience is with ~200 miles of auto-navigate on I-5 in CA with the X. And about 150 miles of travel on various freeways in SoCal the Model 3. Relative to "bad" points 1-11 in your list: 1) I assumed detection of stationary objects in the field of travel is no better with this version and saw no more or fewer situations in which AP might kill me (meaning 1 situation in a construction zone in the Model 3). 2) I did not experience this. 3) I had the opposite experience and tested just for this scenario because I'm concerned with how much trust my spouse puts in AP. Examples: A)Merging onto freeway traffic with entrance below freeway grade and semi-truck on collision course with my merge. Model X detected truck, slowed, and just waited for the truck to pass before flawlessly merging into freeway lane. This was impressive to me because of the poor visibility due to bad geometry for seeing the traffic. B) I had exactly the scenario you described with a large truck merging from lane 2 into lane 3 just as I confirmed to AP, my merge from lane 4 into lane 3. AP detected the truck as my Model 3 was just over the lane line and then quickly, but not abruptly came back into lane 4, avoiding the truck. Then, asked to re-confirm lane merge after truck passed. It was good driving. 4) Totally agree. Also doesn't detect disabled or emergency vehicles on shoulder which, by fed law require slowing or changing lanes (pref, latter). 5) have not experienced this. 6) I have seen this a couple times. It's disturbing. I think both times were in construction zones that had no actual construction. I wonder if it's a DOT data base issue? Anyway, I concur. 7) Yes! AP, please use posted speeds. 8) Around here, I haven't come across the cones scenario, I'll take your word for it. In general, I avoid AP use in construction zones and advised my spouse not to use it there either. 9) Mad Max mode fixed that issue for me. It was my greatest frustration during initial test before I RTFM. 10) Agreed. I've seen this too. In fact in general I get frustrated with just not comprehending what the hell AP is "thinking". I do note that sometimes it seems to want to get over to lane 1 quickly I suspect because I have HOV set as a pref. But the lane change freakout has happened twice in thick traffic and I'm sure it PO'd drivers around me. (just bad driving, AP). 11) I haven't noticed this. Manual lane changes always worked for me. It's not useless for me based on a few days of testing. I found it pleasant, with Mad Max, on my long drive down I-5 dealing with truck traffic with only 2-lanes to work with. Over the winding Grape-vine section, AP seemed no better or worse than before, meaning I had 2 interventions due to scary AP driving over a 40 mile section. But, I was prepared for that. I also don't know why auto-nav engages turn signal to exit freeway but not to enter. Is that not a thing? I always do. I'm going to wait a version or 2 before I ok auto-nav for general family use but I'll keep using it if for no other reason than to provide Tesla with stats. TLDR; Two-steps forward, one back. Inverse for you I guess.
First of all, amazing thread, big fan of your work. Second, I truly hope the post I quoted happens for the benefit of all Tesla customers. I want you to embarrass the f*** out of their AP team. Please please please make a video with your mods and get them to say “s*** why is this one guy beating us?” Embarrassing progress in 2 years. I would love to tweet your video at Elon, and say quit ****ing around on twitter and DO THIS.
I'm sensing a theme that a bunch of the proponents that aren't haven't as many issues are driving California routes. The country is more than California, despite what many Californians may believe. This is probably also why rain sensing on AP2+ absolutely sucks.
Well, if they can't get the fused sensor data stable/accurate for the display, what does that say about what the car is using as inputs to their driving algorithms? The display is a whole lot easier to control than the car. When I was doing that kind of stuff, the display showed what the system's view of what was going on, for debugging if nothing else. Since it's not physically possible for vehicles to bounce around the way the display is showing, that says there's something seriously wrong with how they are modeling the environment around the Tesla. If Tesla is doing some kind of physical model, then it's pretty bizarre that they display the surrounding traffic so badly, if not, then IMHO they're going down a dead end. Also, anyone who isn't aware of the positions of cars around them when driving is going to have a pretty bad experience sooner or later.
Telsa's Motion planning and control algorithms is straight garbage among other things, in-addition to their terrible in-accurate and in-efficient network my friend. But be-careful, you can easily go from a Saint to a foe around these quarters now. Jimmy has this entire forum hypnotized! I have even felt strong resistance from @lunitiks and @S4WRXTTCS of all people. Look at the stuff coming out of the Tesla camp quoting him... Deep Dive Into Tesla's Autopilot & Self-Driving Architecture vs Lidar-Based Systems | CleanTechnica Its not safe making posts like yours though with the current climate, even i am taking precautions. So...
There would be a huge benefit in all aspects of the car for them to have engineers outside California. List of gripes is far too long.
Completely respect your work. From a highlevel it appears your input and then output is limited in scope. It appears you are only evaluating this at a point-in-time and technical aspect on a limited type and length of roads. Have you considered what limitations (restrictions) and cost Mobileye put on Tesla? Are you considering that Tesla can now accelerate since they have this in house now? Have you reviewed all of Jimmy_D post on this? Neural Networks How about listed to this podcast/interview: Interview: Neural Networks & Autopilot V9 With Jimmy_d (10.29.18) - TechCast Daily What about this: Good discussion he is having on why the cars in the display are not displaying accurately. HTH jimmy_d, Sunday at 6:21 PM Neural Networks
I have no doubt that EAP and FSD are monumental technical problems and being the leader in this field is the next closest thing to "bet the farm" that Model 3 is (was) to Tesla. Even Calif has a wealth of driving corner cases (even parts with 100 cm annual rainfall). I have it on good authority that in the cramped open-plan engineering floor at Deer Creek, Musk occupies a desk, no office, not far from center, surrounded immediately by the AP folks. Presumably they are the developers and alpha testers (Musk included) and only drive CA roads and mostly freeways. I've encountered testers at SC spots along CA North-South route including one doing exit to exit AP testing 2 years ago. Now, employees, including SpaceX are also AP beta testers. @wk057, have you tried to sign up as one? Seems like you'd be excellent. Maybe many others outside of our little CA weather-bubble should as well.
To be blunt, a bunch of the info in this quote is complete BS, but I unfortunately don't have the time to dive into the details to unequivocally disprove this right this second. Suffice it to say, while mapping the raw data to a visualization is challenging, it's not super difficult. I've already done it with AP1 for my mods and started on modding my own visualizer (top-down) for use with AP2. It's basically pre-pre-alpha that's already more usable than Tesla's.