HW2.5 capabilities

[daktari]

Back to poor vision over crests, imho I can see lane markers when the car veers off. My eyes are positioned even lower and further back than the camera. Car also behaves badly in a small highway bumps and undulations, where lane markers are visible all the time.
My interpretation is that the car's vision is tricked to believe the undulation is a turn. Because it has mono, not stereo vision.
Using the tele cam would identify the correct path behind the bump, but you probably need some smart code to identify "false positive turn" to "ignore turn" and keep going straight.

 

[jimmy_d]

Nvidia specifically said they chose curvy sections more than straight sections of road to keep the net from "learning" to go straight. All those cues you mention about trees, phone poles, etc. are some of the data I think give NN such an advantage in these situations. It's in many ways akin to the problems with natural language translation that NN solved. There's so many quirky things to deal with that you can't if...then...else or switch case your way through them all.

I totally agree with your interpretation. I was really surprised that AP2 was having more problems because I imagined that they would be evaluating more of the frame for clues about the upcoming road than would have been possible with the more limited AP1 compute capacity. And probably they are - I notice that AP2 is worlds better at identifying pavement boundaries as proper lane delineations, which is a lot harder than recognizing stripes.

Of course, salting your training data to equally represent output categories is basic training methodology so I presume everybody does that. But the fact that the world has a lot fewer 'cresting a curve' situations than 'continuing straight' situations leads me to wonder if it might not be hard to come up with enough training data for cresting curves.

But I'm sure Tesla has a good reason to have the cars veer dangerously, unpredictable and eagerly off on some angle when they lose the road. Seriously, if instead of "panic! Turn the wheel!" as the car does at the top of the hill, would it be better if it slowed down but continued straight? In your concrete example, is there enough time to slow down and have the pitch of the car level w the road enough to see the turn upon cresting the hill?

I wonder how often they would be slowing down if they did that. There's certainly time, but it might happen with enough frequency to annoy the people in the car. I took AP1 on a mountain road trip a month ago and was amazed at how well it was handling curvy sections in terms of slowing appropriately for tight turns and then accelerating for straights. In that situation it's varying speed a lot, but you can clearly understand what it's doing and of course it feels appropriate. But if it was doing it in situation where a human wouldn't find it necessary then maybe it becomes annoying? I haven't seen it slow for a cresting hill yet.

My experience is that the behavior of the car on AP2 improves rapidly as you lower the speed. At 50 MPH it's a whole different beast than it is at 75MPH. That got me to wondering if the problem might be a control lag issue, with AP1 having shorter lag time than the current AP2. There's an underdamped steering control oscillation at about 0.2Hz that I see from time to time. If that corresponds to AP2's nominal full data processing lag time then it could well be an update rate issue.

 

[rnortman]

He was claiming accuracy +/- 1 one foot or so for the maps, and said that combined with image recognition neural nets you do not need maps that are in the cm accuracy range.

It may be that Tesla is doing something like this. In that case I stand by my assertion that they're not even close to what Waymo is using. +/-1 foot on a visual landmark every 1/8 mile is not an HD map. It may be that they won't need an HD map and can do with a sparse landmark map instead, when combined with GPS averaging, but it's not an HD map.

I wonder if they have yet done more than that problematic section of the 405? I have not seen any evidence of anything like that in the East coast areas I've driven in. If the lane lines disappear here or are difficult to see, AP gives up (or veers into oncoming traffic). If they only fixed it for that section of road to appease Elon, that's far from a large scale roll-out -- that's a proof of concept. But it may well be that the increased uploads mean it's coming for everybody soon.

 

[jimmy_d]

What makes you think that (a) Tesla has HD maps of any sort deployed and being used in production, and (b) either AP1 or AP2 are using maps? I think they don't (yet) have HD maps (or anything close to it outside of their R&D labs) and if APx is using maps/GPS at all (beyond speed limit and road classification) I think it's a pretty primitive attempt to slow down in advance of a curve, and I think even that is speculation at this point.

I would be thrilled to be shown to be wrong on any of those points.

Just to avoid any terminology confusion: I'm using the term HD maps to describe digital maps that are higher resolution than the classic NavTeq style maps that describe roads in terms of curve parameters and centerlines. So any map that, for instance, has separate lane centers for each lane and which reports lane centerlines at a frequency higher than just what is needed for curve description would be an HD map in that definition. I might be using the term incorrectly, I'm just following the term use as I understand it. There's another, different, way the term gets used that seems to come from originally from Google where they mean a map that is generated from lidar records and which provides a centimeter resolution topological representation of the entire path. I have not heard anything to suggest that Tesla is doing this second activity and, in a sense, it doesn't work for their use case so I'd guess they won't go that route. As for the former, Tesla executives made representations regarding generating what they were calling high precision maps back when AP1 was first introduced. I just did a search to find an example and noted that I couldn't find examples of those statements more recently, so maybe there's something to the notion that they changed their approach.

You've probably seen this image, it's from a Tesla presentation:



This showed up in a couple of different autopilot presentations back in 2014/2015. The left frame is what I described as NavTeq style data, the one on the right shows much higher resolution including individual high resolution lane metrics.

It's possible that they don't actually do that I suppose, but they definitely use some kind of beyond NavTeq map data because I've often seen my car respond to things that can't be seen from the forward camera or radar - like anticipating and slowing appropriately for a blind curve on a road through a tunnel of trees. When I say appropriately I don't mean that it just slowed down - earlier versions of AP1 would do that, but later version of AP1 would match the entry speed of the curve to the unseen curvature of the upcoming bend. At first I thought maybe it was reading warning signs but I found that the behavior did not correlate to any signage. It slows to different speeds depending on the curve. It's not always right on, but it performs much better than chance would allow. I remember quite distinctly when this capability showed up because it dramatically changed it's behavior on some routes that I drive frequently.

Anyway, that's all I have to support my belief that they are using HD maps - in the past they've said they are, and I see behavior that I can't explain without using GPS/maps. Just like everyone else here, Tesla is mostly a black box to me so I could be wrong.

 

[jimmy_d]

Elon is on the record saying HD GPS lane mapping in AP1 is what allowed them to get the cars to stay in lane on poorly marked sections of the 405 in Los Angeles. I am too lazy to find it right now but it's in the record. The skeptics claim this wasn't / isn't possible because of technical limitations but there are theories of how Tesla could have used gps systems with confidence intervals wider than a lane to produce maps with single lane accuracy. For map-making If for example the confidence interval for position reports is plus-minus 20 feet, and the sampling errors are uniformly distributed across the range, then one could average the reported position numbers to arrive at a more precise range that maps out lanes. Then on the driving side, again if the samples are frequent enough the computer could average he samples to arrive at a more precise image of where it is in the lane and combine this with visual information to arrive at a more confident and accurate decision on whether it is in the lane. This kind of tempora sampling to remove noise is used in other fields also. For example software can now increase the effective resolution of 8mm film by comparing successive frames of an image to remove noise (because the noise/grain - analogous to a sampling error - is not in the same part of each frame) and construct a synthetic image that is more detailed than any individual film frame.

I have some data on this. I frequently drive the 5 between LA and SF through the central valley. That road was undergoing a lot of repaving work during 2016. This resulted in sections of road where all the lane markers would just disappear for a half mile or more because it was in a post paving but pre-marking stage. I'd drive the route every few weeks, always on autopilot, and it would be different every time. The first time I encountered one was January 2016 and the car just gradually wandered off the road. I was amazed that there wasn't any alarm other than that the steering would start to wander. Over the following months the behavior would gradually improve and I would look forward to encountering such unmarked patches to see how the system was coming along. After a couple of months it would wander, but not get too close to the edge of the pavement, then it would wander less, but I'd be far from the lane center when the marks returned, and finally in maybe 4Q 2016 the car could drive the unmarked length with very little wandering and pretty reliably be close to the lane center when the marks came back. As of mid 2017 the car could even manage unmarked sections with a slight curve, though not as well.

As an aside, I rarely had another vehicle to follow through these sections so the car was not relying on radar tracking a lead car. In the display the lane lines would disappear, but the light grey 'lane surface' would remain and the car would track the center of that grey strip. I can't explain this behavior except by assuming gradually improving maps / gps.

 

[AnxietyRanger]

Oh thank you - what poor timing on my part - I just criticized your youtube argument on another thread at the exact moment you were complimenting me. Wow I feel like an ass now. Sorry. heh.

Don't worry about it. We're all a bit a** at times - and all of us deserve criticism at times too. It is the whole that counts.

Let's keep the dialogue going...

 

[jimmy_d]

Continuing this - Elon, when talking about the 405 freeway problem, went into detail about how they solved it and claimed it happened during AP1 development, prior to the 7.0 release of October 2015. He specifically mentioned that part of the 405 (he lives near that freeway and would probably take it to work on occasion) was not usable with autopilot until Tesla had a test driver make multiple passes through each lane so the GPS could gather data to place the car in lane. In fact he said the system worked so well that the first time they constructed these maps at that freeway section, the test driver himself got the lanes wrong (the 405 had a lot of work going on and the markings were crazy bad) - and then the Tesla followed his bad driving precisely in the "fake lanes" his driving had constructed. So they had to "wipe" the memory and have him drive the section again, this time in the correct position for the lanes.

Living in So Cal, all I know personally is that the freeway markings are garbage (budget problems) - and over the months each release of AP1 handled poor markings and high glare situations better than the last until finally it performed admirably even driving directly into the sun on low-contrast sections of freeway that are challenging even for a human in those conditions. I found it quite shocking how good it got. It took months but it happened. Either this was evidence of HD mapping at work - or the visual neural network lane recognition was getting better - or both.

That was my experience exactly. The 405 sucked and it still does. Faded and worn white stripes painted on white concrete with most of the bots-dots missing, and lanes that split, merge, and curve all over the place. It wasn't until I tried driving it on autopilot that I realized that I myself was mostly just following the other cars and not really paying much attention to the lines at all.

 

[Bladerskb]

It may be that Tesla is doing something like this. In that case I stand by my assertion that they're not even close to what Waymo is using. +/-1 foot on a visual landmark every 1/8 mile is not an HD map. It may be that they won't need an HD map and can do with a sparse landmark map instead, when combined with GPS averaging, but it's not an HD map.

I wonder if they have yet done more than that problematic section of the 405? I have not seen any evidence of anything like that in the East coast areas I've driven in. If the lane lines disappear here or are difficult to see, AP gives up (or veers into oncoming traffic). If they only fixed it for that section of road to appease Elon, that's far from a large scale roll-out -- that's a proof of concept. But it may well be that the increased uploads mean it's coming for everybody soon.

Its not just visual landmarks. Mobil-eye is also doing lane and road boundary and drive-able path mapping with 10cm accuracy.

 

[BigD0g]

Its not just visual landmarks. Mobil-eye is also doing lane and road boundary and drive-able path mapping with 10cm accuracy.

I really wish Tesla would give us these type of videos, the silence is deafening.

 

[stopcrazypp]

Just to avoid any terminology confusion: I'm using the term HD maps to describe digital maps that are higher resolution than the classic NavTeq style maps that describe roads in terms of curve parameters and centerlines. So any map that, for instance, has separate lane centers for each lane and which reports lane centerlines at a frequency higher than just what is needed for curve description would be an HD map in that definition.

That's the definition I use too (and I believe Mobileye and other industry partners does too), so I think that is the source of confusion with @rnortman as he seems to be referring to something like Waymo's maps which are down to centimeter accuracy.

I think we may have to come up with new terms like in TV: 720i to 1080p are all considered HD, Ultra HD covers both 4K and 8K, but we have specific terms to refer to specific resolutions.

 

[calisnow]

My experience is that the behavior of the car on AP2 improves rapidly as you lower the speed. At 50 MPH it's a whole different beast than it is at 75MPH. That got me to wondering if the problem might be a control lag issue, with AP1 having shorter lag time than the current AP2. There's an underdamped steering control oscillation at about 0.2Hz that I see from time to time. If that corresponds to AP2's nominal full data processing lag time then it could well be an update rate issue.

Very interesting observation and theory - I'll have to drive some tricky sections at different speeds and observe.

...they definitely use some kind of beyond NavTeq map data because I've often seen my car respond to things that can't be seen from the forward camera or radar - like anticipating and slowing appropriately for a blind curve on a road through a tunnel of trees...later version of AP1 would match the entry speed of the curve to the unseen curvature of the upcoming bend. At first I thought maybe it was reading warning signs but I found that the behavior did not correlate to any signage. It slows to different speeds depending on the curve. It's not always right on, but it performs much better than chance would allow. I remember quite distinctly when this capability showed up because it dramatically changed it's behavior on some routes that I drive frequently.

Agreed - the first time AP1 did this for me I got goose bumps. It was the sharp 90 degree turn north from the 10 East to the 110 north in downtown Los Angeles. It is a blind curve but I know it very very well - and when the Tesla slowed down to the speed needed for the sharp left turn we were still completely "blind" to the curve itself and like you said - no signage.

 

[calisnow]

I have some data on this. I frequently drive the 5 between LA and SF through ... sections of road where all the lane markers would just disappear for a half mile or more ... The first time I encountered one was January 2016 and the car just gradually wandered off the road. ... Over the following months the behavior would gradually improve and I would look forward to encountering such unmarked patches to see how the system was coming along. After a couple of months it would wander, but not get too close to the edge of the pavement, then it would wander less, but I'd be far from the lane center when the marks returned, and finally in maybe 4Q 2016 the car could drive the unmarked length with very little wandering and pretty reliably be close to the lane center when the marks came back. As of mid 2017 the car could even manage unmarked sections with a slight curve, though not as well. ...I can't explain this behavior except by assuming gradually improving maps / gps.

Completely agree and I observed similar improvements further east on the 10 where the concrete is very low contrast to the lane markings - parts of Ontario, Redlands and Yucaipa. One section of the 10 west in Yucaipa was so bad I would at first cancel autopilot because I was sure I'd get called in for being drunk. But just like you said it gradually improved and now the AP1 car tracks these sections like a train on rails.

 

[rnortman]

OK, so I am pleased to say that I stand corrected. Tesla is clearly using their own lane-level maps combined with GPS smoothing of some sort. Not exactly HD but apparently effective enough. I haven't noticed this perhaps because I'm not in CA, the Land of Many Teslas. There are only a handful around here -- still enough of a novelty that people ask me "what kind of car is that?" -- so they probably don't have enough data to do anything useful (or perhaps they haven't even tried outside of CA yet...)

 

[bjornb]

Self Driving and Crowdsourcing at XKCD today :

xkcd: Self Driving

 

[boonedocks]

My experience is that the behavior of the car on AP2 improves rapidly as you lower the speed. At 50 MPH it's a whole different beast than it is at 75MPH. That got me to wondering if the problem might be a control lag issue, with AP1 having shorter lag time than the current AP2. There's an underdamped steering control oscillation at about 0.2Hz that I see from time to time. If that corresponds to AP2's nominal full data processing lag time then it could well be an update rate issue.

I am beginning to think this is not just possible but most likely probable. Some are reporting excellent results from AP2 while others like myself are completely disheartened by it. My daily commute is 90+mmiels R/T and 80 of it is expressway. The avg speed it about 75mph. If 50 is a whole different beast then that could explain my cars poor behavior.

 

[croman]

I am beginning to think this is not just possible but most likely probable. Some are reporting excellent results from AP2 while others like myself are completely disheartened by it. My daily commute is 90+mmiels R/T and 80 of it is expressway. The avg speed it about 75mph. If 50 is a whole different beast then that could explain my cars poor behavior.

I use AP at 70-80mph consistently in Chicago. Behaves, for the most part, really well. I also use it on local divided roads (but with intersections) and it behaves really well on most areas (it will occasionally dive for a left only turn lane but its, somehow, gotten better without an update at avoiding most of them).

I'm not sure what to make of the variability in experience. My AP2 was absolute garbage before 17.17.4 but people really loved the early iterations. It has substantially improved recently at handling curves (17.17.4 had the straight-line curve issue that was fixed but a bunch of new bugs jumped into the picture and those seem to have been quashed).

I'd strongly urge your local SC to look into why your AP2 is garbage. It might not be hardware related but its good to have that eliminated as a potential issue.

 

[chillaban]

I use AP at 70-80mph consistently in Chicago. Behaves, for the most part, really well. I also use it on local divided roads (but with intersections) and it behaves really well on most areas (it will occasionally dive for a left only turn lane but its, somehow, gotten better without an update at avoiding most of them).

I'm not sure what to make of the variability in experience. My AP2 was absolute garbage before 17.17.4 but people really loved the early iterations. It has substantially improved recently at handling curves (17.17.4 had the straight-line curve issue that was fixed but a bunch of new bugs jumped into the picture and those seem to have been quashed).

I'd strongly urge your local SC to look into why your AP2 is garbage. It might not be hardware related but its good to have that eliminated as a potential issue.

My experience matches yours, pretty much exactly. 17.17.4 and before were really just a novelty for me, because the herky-jerky 8-bit steering didn't instill much confidence around curves -- it was pretty much just a straight-line and low-speed Autopilot.

But then the first set of "silky smooth" releases, including 2017.28, made me really WANT to love it because it made high speed steering silky-smooth at times, but with it came a myriad of other quirks, like departing the lane for no reason at lower speeds, leaving huge gaps in stop-and-go, and even occasionally making sudden and incorrect adjustments at highway speeds despite good lane lines....

I do have to say, the recent updates like 2017.34 made a huge improvement there, and even with that release, you really have to either nitpick or find the exact wrong conditions on the highway to make a point that AP2 is worse than AP1 for highway use (local roads and intersection gaps are another story).... But some of the AP2 behaviors people are describing really make me think there's something wrong with their cars.

 

[jimmy_d]

I am beginning to think this is not just possible but most likely probable. Some are reporting excellent results from AP2 while others like myself are completely disheartened by it. My daily commute is 90+mmiels R/T and 80 of it is expressway. The avg speed it about 75mph. If 50 is a whole different beast then that could explain my cars poor behavior.

I just had a drive that matches your experience. Friday, after getting the latest AP2 update I did a run from SF down to LA. It's 80% straight road through agricultural land and 20% city freeway driving in significant traffic. I tried to do the whole thing on autopilot and I had a total of 6 forced disengagements - every single one of them was in city traffic. I had the speed set to 75 the whole way, though of course it was often traffic limited. On recent AP1 urban freeway traffic disengagements had completely disappeared for me, so AP2 seems to be lagging there.

I guess the good news is it's doing OK for road trip situations (at least it's good news for me personally), and I've had good experience in low speed traffic (under 40MPH or so), but high speeds on urban freeways are still not making the grade. Have only had AP2 for a couple of weeks now so I haven't personally seen what the rate of improvement looks like, but I'm hopeful. I remember that it took AP1 about 6 months to go from where AP2 is right now to being pretty good on urban freeways. Hopefully AP2 will come faster than that.

Sorry to hear that your use case seems to be right where AP2 is weak. I get the impression on these message boards that you are not alone in your dilemma. Here's to hoping it gets a lot better for you real soon.

 

[jimmy_d]

OK, so I am pleased to say that I stand corrected. Tesla is clearly using their own lane-level maps combined with GPS smoothing of some sort. Not exactly HD but apparently effective enough. I haven't noticed this perhaps because I'm not in CA, the Land of Many Teslas. There are only a handful around here -- still enough of a novelty that people ask me "what kind of car is that?" -- so they probably don't have enough data to do anything useful (or perhaps they haven't even tried outside of CA yet...)

I remember thinking that I was really lucky to be in a place where Tesla probably gets a lot of data. And I often wonder how much of the improvement I saw over the years on AP1 was because they had accumulated data for maps and how much of it was the software actually getting better.

Did you have an AP1 car? Your comment makes me wonder how much improvement AP1 users in Tesla-rare areas experienced.

 

[rnortman]

Did you have an AP1 car? Your comment makes me wonder how much improvement AP1 users in Tesla-rare areas experienced.

Nope, I never had AP1. Took delivery of my S75D AP2 with HW2.5 just a couple of weeks ago. AP has been trying to murder me that entire time and so far I'm winning that battle, but only by keeping my hands on the wheel and remaining vigilant.