HW2.5 capabilities

[verygreen]

That's literally 5 cameras rendered useless. @verygreen can confirm this

The night footage is here, feel free to process it (there's also a dashcam video for comparison): https://app.box.com/s/jah6ovf6y5eoq9faejeylt7fm2j2od4i

 

[Silenus136]

My apologies if covered in another post, but how do I tell if the 100D I have on order (late September delivery) is AP 2.0, 2.5, or something else?

 

[JeffK]

The reason radar isn't a main system (besides it will need surround radars to even be A system) is that unlike it, lidar system can differentiate and classify objects. It can classify a deer or pedestrian, a car, a cone, lanes, trees, barriers, road sign,cyclist, pedestrians, curbs, grass, road edges etc. Radar on the other hand will just return that it sees a big object and cant tell you that the object is a big rock or a deer.

Volvo used camera/radar combos to detect large animals. No one suggested using radar only even though it is possible to do some basic pedestrian, large animal, huge rock, or whatever detection. It doesn't really matter much. If there's an object in front of the car you don't want to hit it. It doesn't matter if it's an Alaskan moose, North-Western moose, or a big deer you aren't going to want to hit it. The best sensor for confirmation of what the object is will always be a camera which is better than both radar and lidar for such a task.

Lidar also works in heavy rain, snow and dust.

Lidar works with corrective algorithms in rain, snow, or dust, lidar does not see through snow whether on the sensor or on the ground, unlike radar. lidar will also fail if it gets dirty, you'd need something like Waymo's lidar wipers, you'll definitely need the wiper fluid if it gets oily spray from other drivers. In a really heavy rain where humans typically pull over, forget about it, it'll be next to useless (same would be true with cameras, but not for radar).

Unlike sunny California, I feel like people in regions with varying weather conditions are going to find themselves reaching to the top of their cars trying to clean the lidar when the wipers fail to do an adequate job. It'll be super fun when the wiper fluid nozzles freeze up and the wipers get frozen. Perhaps the lidar dome is heated as well? Winter is going to suck for people with lidar and without garages.

 

[croman]

My apologies if covered in another post, but how do I tell if the 100D I have on order (late September delivery) is AP 2.0, 2.5, or something else?

Check your mytesla image to see if it has APH3 code.

 

[aWalkingShadow]

lidar will also fail if it gets dirty, you'd need something like Waymo's lidar wipers, you'll definitely need the wiper fluid if it gets oily spray from other drivers. In a really heavy rain where humans typically pull over, forget about it, it'll be next to useless (same would be true with cameras, but not for radar).

Doesn't your argument imply that Tesla's current AP2 hardware won't be sufficient for level 5 FSD? The side and back cameras don't have any wipers, so all it would take is some unlucky mud spray from other drivers.

 

[verygreen]

Doesn't your argument imply that Tesla's current AP2 hardware won't be sufficient for level 5 FSD? The side and back cameras don't have any wipers, so all it would take is some unlucky mud spray from other drivers.

Side cameras worked great in the rain as I recently verified.
Backup camera - not so much: AP2.0 Cameras: Capabilities and Limitations?

Additionally nobody promised level 5 FSD on HW2 or any other hardware in a Tesla. They only promised "most conditions". Rain is rare in California, I think?

 

[JeffK]

Doesn't your argument imply that Tesla's current AP2 hardware won't be sufficient for level 5 FSD? The side and back cameras don't have any wipers, so all it would take is some unlucky mud spray from other drivers.

Just because any system doesn't work in the harshest conditions doesn't make it any less level 5. There are many weather conditions which prevent humans from driving or in the very least prevent humans from driving safely.

 

[aWalkingShadow]

Side cameras worked great in the rain as I recently verified.
Backup camera - not so much: AP2.0 Cameras: Capabilities and Limitations?

Additionally nobody promised level 5 FSD on HW2 or any other hardware in a Tesla. They only promised "most conditions". Rain is rare in California, I think?

Excuse my ignorance, is the backup camera the same as the rear facing camera used by autopilot?

Just because any system doesn't work in the harshest conditions doesn't make it any less level 5. There are many weather conditions which prevent humans from driving or in the very least prevent humans from driving safely.

A little bit of rain or mud on the ground isn't particularly harsh. A thin mist of water spray from neighbouring cars is quite common where I live. If you're unlucky, some of the droplets could get onto the camera.

 

[verygreen]

Excuse my ignorance, is the backup camera the same as the rear facing camera used by autopilot?

autopilot does not really use any other cameras than two front ones. There are 3 rear-facing cameras. the backup camera (also displays on CID when you put the car in reverse) and the two on the sides where turn signal repeaters would be otherwise (hence the name, repeater cameras?)

 

[jimmy_d]

It all depends on a lot of factors, though.

It's one thing to just know in general words about something happening, it's quite a bit different thing when you have 90% of firmware for this new thing.

Not to say there is not some amount of speculation, of course, but some facts about hardware in customers hands (hw2.5 in model 3 meets that to a significant degree) are now irrefutable (where as before whole hw2.5 was questioned).

Still would I love to have insider view on certain things? Of course I would!

Yes, with access to a lot of the code you certainly get more that just rumors. There's hard data embedded in there and it can tell you a lot. But just for perspective: in the aforementioned company experience we also had technically sophisticated customers guessing about our future direction based on careful teardowns of our stuff and review of our firmware (the binary of which could be read directly give the right equipment), and they were still wrong about 98% of the time. Our product was really different and they certainly had a lot less data to go on than you seem to have. But then Tesla is a much more complicated company in a much more complicated situation too, so you're in the situation of having to get more from your data.

I really appreciate your contributions BTW. I don't have the time or the chutzpah to take apart my beloved ride but I really like hearing about what can be gleaned to someone with the skills and resources.

 

[stopcrazypp]

Side cameras worked great in the rain as I recently verified.
Backup camera - not so much: AP2.0 Cameras: Capabilities and Limitations?

Additionally nobody promised level 5 FSD on HW2 or any other hardware in a Tesla. They only promised "most conditions". Rain is rare in California, I think?

The only thing that seems to be effective is using some plastic to create a kind of awing for the backup camera. Cheap and effective. Maybe Tesla can offer a better integrated version with a trim replacement or a stick-on or snap-on version.
Rain on rear view camera

 

[verygreen]

The only thing that seems to be effective is using some plastic to create a kind of awing for the backup camera. Cheap and effective. Maybe Tesla can offer a better integrated version with a trim replacement or a stick-on or snap-on version.
Rain on rear view camera

nope, those drops are not from above, note how the surface street still has the backup camera clear (even after some rain), but as you increase speed the mist from the road is sucked behind the car and gets on the backup camera.

 

[jimmy_d]

@lunitiks Why would you assume that? AP3 and AP4 hardware revisions sound perfectly plausible.



I don't think people are worried on the basis of wiring charts and log files alone.

The worry - if you can call it that - let's call it interest instead...

The interest in what Tesla can achieve with AP2 and the expectations that we have are based on the whole history and insight we have into Tesla. We know they overpromised on AP1 and some of those promises (e.g. summon meeting you on the curb or the traffic light promises) will probably never materialize themselves in the AP1. We know Tesla overpromised on the performance of P85D, again P85DL, P90DL V1 and P90DL V2 etc. Things that were never corrected and will likely never be corrected, outside of buying a new car. And of course things like the Model X launch and what we learned of it before the launch through leaks compared to what Tesla said in public, suggest that Tesla is not transparent when it comes to these things. If there was an issue, Tesla would likely not be forthcoming about it in advance.

Hence the conclusion - in general - that some of Tesla's stated, forward-looking goals may only be met by buying a completely new car is rooted in past experience. People are now applying that knowledge as well as technical findings into AP2 capabilities. Is this some certain, factual representation of the future, of course not, but it is not merely derived from a wiring chart either.

That's a very nicely articulated expression of reasonable concerns. There's certainly no guarantee that Tesla can deliver on the lofty promise of FSD, even given the powerful incentives at work and the corner that they have proactively painted themselves into by pre-selling the capability (something I really admire, BTW).

That said, I see a lot of hope in the prospect of it being delivered for all of the HW2 vehicles. Here's why:

To build an FSD car (or any autonomous system that operates in the real world) you need adequate sensors, adequate software, adequate compute resources, and adequate actuation capability. There are tradeoffs between the capability of the sensors, software, and compute resources where having one of them be weaker raises the bar for the others. Alternately making any one of them better lowers the burden on the other two. But in general you can imagine that there's a minimum bar for all of them that must be achieved. For Tesla to be able to deliver on the promise of providing FSD to all HW2 vehicles at a minimum all of those vehicles must be capable of meeting the minimum bar for each of the 4 major categories of resource.

Any drive-by-wire vehicle meets the actuation criteria already, so that's covered. Tesla went with making the compute and software swappable, so those can be covered at the cost of swapping the compute hardware. (Incidentally, the compute hardware cost is, and likely will remain, below the cost of returning the $3k FSD feature fee, so I don't think there's good reason to believe Tesla will refuse to upgrade customers for economic reasons.) Sensors are swappable only within very restricted limits because anything that can't be accommodated by the existing wiring harness is going to be right out, so in a sense whether it is technically possible for Tesla to deliver boils down to the question: "Is it feasible to perform FSD with the HW2 sensor suite?". I think that most people here understand that issue pretty well, and that's why so much discussion has focused on the capabilities of the sensor suite.

So can you do it with the HW2 sensors? Well yeah, you can. In fact it can be done with just cameras as the existence of human drivers demonstrates. Our road infrastructure, our traffic routing and signaling, and our regulations are all optimized for humans, who drive using essentially nothing but cameras. You can't do it with just lidar, or just sonar, or just radar. But you can do it with just cameras given the right software and compute hardware. In fact, you have to solve the camera problem even if your system uses other sensors too, because no amount of sophistication with non-visual sensors can compensate for the lack of a highly competent visual processing system.

Now the reliability question is an interesting one, because of course vehicles must be fail-safe to a high degree. Braking, throttle, and steering systems in all manufactured cars today achieve a certain level of fail-safety that society finds acceptable even though there are still a small fraction of accidents caused by hardware failures. What is the likelihood that flaw exists in the early HW2 systems which would render them unacceptably non-fail safe and which affords no economical work around? A detailed discussion is probably more than anyone wants to hear in a forum post, but in short the answer is "very low". Now, the situation on the ground could change if we change the bar on 'acceptable' such that it's really different from what we tolerate on our roads today, or if some powerful interest forces unreasonable regulator changes, or if society's tolerance for self driving vehicles turns out to be really low, or... But none of that will be a technical failure on Tesla's part with respect to delivering a competent vehicle. Which is why I think they have the regulator exception boilerplate in the FSD feature description but are otherwise confident of eventual success.

Putting together a failure tree analysis for a particular system architecture is tedious, but it's a straightforward and dependable process. You can be certain that Tesla did that analysis when they architected HW2. Given that they did that competently there aren't any unforeseeable future developments in reliability of HW2 components which have any decent chance of invalidating the HW2 architecture given appropriate margins on their predicted failure rates. Can they make it better? Certainly. Will they? Certainly. Does that mean that the unimproved versions are unacceptable? Certainly not.

I used to consult for one of the first companies that made crash sensors for firing airbags during a collision. Compared to later sensors the early generations were terrible. Early airbag sensors were much, much more apt to fail than later generations. But even the earliest generations were much better than not having an airbag in the vehicle. Despite later advances none of the early generation airbag sensors was upgraded in a mandatory recall and nobody ever suggested disabling them just because the designs were obsolete. They were good enough. And similarly it is extremely likely that while HW2 can be improved on, no matter what improvements come later the original will still be good enough, in a fail-safe sense.

 

[jimmy_d]

Isn't the ECU in all current Model S/X controlled by effectively an outdated phone processor? An old Tegra 3 or something?

Yeah, terrible CPU. It's only about 100x faster than the early Cray supercomputers. Maybe we should blame the software.

 

[stopcrazypp]

nope, those drops are not from above, note how the surface street still has the backup camera clear (even after some rain), but as you increase speed the mist from the road is sucked behind the car and gets on the backup camera.

He did say the mist is still there, but the image is still visible. Too bad he doesn't have a shot of that as an example.

 

[stopcrazypp]

The night footage is here, feel free to process it (there's also a dashcam video for comparison): https://app.box.com/s/jah6ovf6y5eoq9faejeylt7fm2j2od4i

I dumped the snapshot-night-00 dataset into the tool provided by @DamianXVI here (much kudos for that):
AP2.0 Cameras: Capabilities and Limitations?

First of all, the dynamic range of these Aptina sensors is insane! The new 32 bit tiff mode in the tool really is necessary for processing these, the 8-bit bmps (and the 8-bit jpegs I am posting) really don't do it justice.

I picked the set of files beginning with 245 because those appear to have the lowest available light (farthest away from street lights). You can tell how dark it is by the fisheye showing no illumination of the black plastic of the enclosure in the lower part of the frame (other sets those parts were still visible).

I used two processes in Photoshop. The files ending in eg used Auto Contrast and then Exposure and Gamma to convert to 8-bit. The files ending in eh (in another post following) is using your earlier process of Equalize Histogram to convert to 8-bit (this yields a higher contrast image, but not as natural).

No noise reduction or other complicated processing is done (but tiff files have enough in them that I bet a lot more can be done to them). The jpegs are compressed a bit for faster upload.
(Order: fisheye, main, narrow, leftpillar, leftrepeater, rightpillar, rightrepeater)

 

[lunitiks]

Thanks @stopcrazypp

 

[AnxietyRanger]

A dark highway or rural road (no urban lights) would be interesting. Anyone?

 

[verygreen]

I dumped the snapshot-night-00 dataset into the tool

I think there was a more dark location in one of the later snapshots, if you find where this stash is referenced in the AP 2.0 cameras thread there were some descriptions.

Also perhaps more importantly there were a few highway snapshots that probably more realistically represent what cameras would be against during intended usage.

And yes, dynamic range is pretty good. Also do try other HDR processing options in photoshop, I think. They don't look natural, but that's just because our monitors have a limited dynamic range that we need to compress for

 

[stopcrazypp]

These files ending in eh is using Equalize Histogram to convert to 8-bit (this yields a higher contrast image, but not as natural).

(Order: fisheye, main, narrow, leftpillar, leftrepeater, rightpillar, rightrepeater)