Hypothetical: what FSD hardware would you put on a car?

[S4WRXTTCS]

The first thing I'd do is build a good road and traffic simulator to try out the software. Then get the software running and based on what was needed to do that decide what kind of sensors were required. Then there'd be the issue of the necessary processing and memory capabilities, although I suspect this would all be a bit iterative.

I doubt very much that hardware, sensors or processors are even 1% of the work to get this running.

That's a good point especially with where simulators are these days.

For my Hobby robotics projects I'm using Nvidia Isaac SDK, and Simulator. The simulator is based on the unreal engine, and it's pretty nifty.

 

[dgatwood]

I know this forum often talks about FSD hardware and we routinely debate LIDAR or no LIDAR, whether Tesla's FSD hardware is enough, what we like or don't like about another company's FSD hardware. So I thought it might be interesting to play a little game: If you were in charge of designing a FSD vehicle, what sensors/hardware would you pick? And to avoid the obvious answer "I would put everything on the car", you have to take into account costs and power consumption and try to keep your hardware as low as possible.

• A separate camera three inches from each of the two side cameras.
• Additional cameras at both ends of the front and rear bumpers.

That's it. The current sensor suite is basically good enough, and those small tweaks would greatly improve robustness in bad weather and when navigating T-intersections and nose-in/tail-in parking places.

Radio equipment needed to support V2I and V2V to allow the car to be more aware of the surrounding traffic infrastructure and other similarly equipped vehicles.

Definitely not. V2I and V2V are, from a computer security perspective, recipes for disaster. For V2I, all it takes is one person stealing a beacon and modifying it to send false data, and suddenly you're seeing a 90 MPH speed limit around a tight corner. Securing that sort of hardware, if broadly deployed, rapidly becomes infeasible. And for V2V, you don't even have to steal a beacon; every driver owns one.

It is fundamentally impossible to design a V2I or V2V system that can't be weaponized unless the only data it sends is limited to data that the vehicle could do without, and which the vehicle knows how to ignore if that data turns out to be fake. And if both of those are true, then what's the point of sending the data at all?

At best, if the data isn't time-sensitive (e.g. pothole reporting), it might make sense to upload it via cellular, and after being confirmed by enough distinct vehicles (with server-based fraud detection), it could then be distributed to any vehicle traveling that route. But that doesn't require V2V.

V2I would allow traffic junction optimisation and green waving meaning fewer stops at junctions and longer platoons of cars making it through the junction without stopping.

Until the first time the system gets hacked and deliberately sends two vehicles into each other. And then it will be banned worldwide, and you will have spent all that money putting in the infrastructure, only to never use it again. We can already do this with cameras, just a little bit more slowly, and without the added risk of having to trust that every vehicle is reporting its location accurately (which absolutely cannot be guaranteed, because GPS accuracy can be affected by environmental conditions, such as reflections off of walls/buildings, other cars, etc.).

V2V would allow better road utilisation in the vicinity of other V2V cars, including high speed platoons in high density lanes and better merging.

No, it really can't. Either you can see the car in front of you and know when it is braking or you can't. If you can, then you don't need the V2V data. If you can't, then you can't trust the V2V data.

The difference in density you can achieve between a raft of cars controlled by computers with multiple cameras running at 60 fps (let's call it ~20 millisecond response time) and a similar grouping involving V2V radio data (likely single-digit millisecond response time) is only a difference of a few inches of extra space per vehicle.

Except that in practice, you can't even don that. After all, radio transmissions aren't perfect. If a radio burst gets garbled by outside noise and never reaches the car behind you, if that car is counting on receiving that notification, you're screwed. By contrast, you can verify that you have a signal from your cameras, so you can rely on that for determining that the car in front of you has slowed down. You can also verify that you have a signal from your RADAR hardware, so you can also trust that to tell you that the car in front of you has slowed down. Both of these make the V2V redundant unless you're following so closely that those aren't adequate, and if that is the case, you're just asking for trouble.

Really, V2V and V2I are solutions in search of a problem.

 

[electronblue]

@dgatwood

I think you are missing the potential for redundancy though in V2V, V2I — and most importantly the fact that V2V, V2I would still be augmented by vision, other sensors and map databases.

Getting V2V information about a fast approaching car unseen around the corner allows the autonomous vehicle to prepare. They would still use vision etc too to assess the situation. V2I information about a speed limit would still be matched against what is seen in the traffic signs.

Would there be discrepancies? As with all redundancy of course there will and that is where sensor and data fusion comes in. If V2I says 90 mph speed limit while maps and visual says 30 mph, stay slower. Yet if V2V says a car is approaching 90 mph sight unseen from the right and about to run a red light, wait to see if it really is, there’s little harm in that...

The more different types of data you have, less likely you are to be fooled by any one of them.

 

[MrBadger]

All of the above concerns regarding V2I and V2V can be applied to pretty much any walk of life. Off top of my head, for beacon, read blind a car's camera with a mirror, or tamper with a car stop lights/indicators or at traffic lights, or simply bird crap covering the front windscreen cameras. If we took such a view on everything, we would not progress. These risks can be managed and mitigated, just like risks of things that we take for granted now are not the problem that some may have suggested. Its not like people do not think about the consequences of what if things do not function as expected, its often the most significant part of the design.

There is a lot more to platooning than just placing yourself 40cm off the vehicle in front and staying there, something that many vehicles could theoretically do already (it basically requires LKA and ACC) - but they can't simply because they would need V2V in some form - join platoon, break platoon, vehicle x in front has something it needs to communicate to the others etc etc. Hell, CB radio is a form of V2V and may cover some of these solutions, but we have moved on and want to be able to perform other tasks, quickly and with a wider audience.

Disclosure: I use to work for a company who are a major player involved in traffic infrastructure, road safety and transportation systems security. They are deeply involved in V2I and V2V, amongst other things that benefit us all - I was very much on the periphery of some of this, a bit more involved with some technologies but most of the time only involved by virtue of sitting next to colleagues who were - not that we were always allowed to talk about what was going on, even within the same group. But I can tell you now, the technology is in very safe hands. There are huge benefits to be gained from this. It is not a problem waiting for a solution. There are some very real problems that could be hugely mitigated by this technology now.

 

[boonedocks]

For comparison, the Lucid Air will have the following FSD hardware:
- 2 long range radar and 4 short range radar
- 3 front cameras (with different field of vision)
- 5 active surround cameras
- 2 long range LIDAR and 2 short range LIDAR
- 1 driver monitoring camera
- 2 EyeQ4 computers
Lucid Motors' autonomous tech in its all-electric sedan will be powered by Tesla's former partner Mobileye - Electrek

View attachment 422129

- 1 small nuclear generator to power ALL of that.

 

[diplomat33]

- 1 small nuclear generator to power ALL of that.

I admit that while Lucid Air's FSD hardware looks great on paper, cost and power consumption remain two big question marks. I have my doubts that Lucid can make a profit, selling the Air with all that hardware, at low volume and at the same price as the Model S. After all, there is a simple financial reason why Tesla went with cameras+radar and left out LIDAR. And all that hardware would require more power too which would reduce effective range. The Lucid Air looks amazing but is probably one of those concept cars that is too good to be true. I hope I am wrong because it looks like an amazing car.

 

[electronblue]

Actually, thanks to the energy sparing nature of EyeQ4, much of that setup will actually take less energy than the Tesla suite which depends on a running very power hungry sort of NNs.

 

[diplomat33]

Thinking more about this, I think the current AP2 sensor suite probably meets a "minimum requirement" for good weather FSD. But the current AP2 sensors are not good enough in my opinion for bad weather conditions. If your cameras are blinded by the Sun or there is very poor visibility due to heavy rain/snow/fog, I think the current hardware will struggle to do self-driving. I think adding 360 degree LIDAR and 1 rear facing radar to the existing AP2 hardware suite, would probably be enough to meet a "minimum requirement" for true L4/5 autonomy, assuming best quality for all the sensors. The 8 cameras are good. They are well placed and offer good coverage. The forward facing radar is also good. 360 degree LIDAR would provide much needed redundant coverage all around the car and the rear facing radar would provide extra redundancy and help with fast moving cars behind you.

Thoughts?

 

[electronblue]

@diplomat33 Personally I think 360 degree radar (basically 4-5 radars) and a forward Lidar could work too and probably be more cost-efficient.

I mean there are bound to be many different solutions to this that can work.

 

[diplomat33]

@diplomat33 Personally I think 360 degree radar (basically 4-5 radars) and a forward Lidar could work too and probably be more cost-efficient.

I mean there are bound to be many different solutions to this that can work.

I am fine with 360 degree radar instead of 360 degree lidar. But I am curious: what is your rationale for also adding just a forward facing lidar?

 

[electronblue]

I am fine with 360 degree radar instead of 360 degree lidar. But I am curious: what is your rationale for also adding just a forward facing lidar?

Well you have to understand that I’m thinking from the perspective of ways to improve the Tesla suite, not necessarily from the perspective of what I’d put in an FSD suite designed from the ground up.

But in that context, here goes:

360 coverage with radars would probably require only three additional radars to three corners (neatly under bumpers) and replacing the forward radar with a dual forward/corner radar in the fourth corner. This is very reasonable cost and provides 360 degrees of coverage against approaching cars, which is the biggest 360 degree threat. Radar is good at seeing moving cars and can also see past some obstacles so great at detecting approaching cars.

360 degree Lidar probably still costs more and requires more changes for sensor placement. But a single forward-facing Lidar would be reasonably priced and could fairly easily be mounted in the lower forward grille. Most importantly it would do the one thing radar and even vision as is are not good at: provide protection against driving into stationary objects. And this is a skill which is most important to have in the forward direction.

 

[diplomat33]

Well you have to understand that I’m thinking from the perspective of ways to improve the Tesla suite, not necessarily from the perspective of what I’d put in an FSD suite designed from the ground up.

But in that context, here goes:

360 coverage with radars would probably require only three additional radars to three corners (neatly under bumpers) and replacing the forward radar with a dual forward/corner radar in the fourth corner. This is very reasonable cost and provides 360 degrees of coverage against approaching cars, which is the biggest 360 degree threat. Radar is good at seeing moving cars and can also see past some obstacles so great at detecting approaching cars.

360 degree Lidar probably still costs more and requires more changes for sensor placement. But a single forward-facing Lidar would be reasonably priced and could fairly easily be mounted in the lower forward grille. Most importantly it would do the one thing radar and even vision as is are not good at: provide protection against driving into stationary objects. And this is a skill which is most important to have in the forward direction.

Thanks. That makes sense.

 

[banned-66611]

@diplomat33 Personally I think 360 degree radar (basically 4-5 radars) and a forward Lidar could work too and probably be more cost-efficient.

It's not just a case of gluing on enough sensors to cover all directions.

For FSD the car needs to be able to build up a map of the environment around it, as well as identify different objects and road users. Lidar makes that much, much easier than just trying to use radar or cameras alone.

Radar is only really useful for measuring the distance to fairly flat, large objects like the back of the car in front.

Cameras require enormous processing power and AI beyond what even a supercomputer is capable of.

 

[electronblue]

@banned-66611 That is of course true.

What I guess I was trying to get across is that if the likes of Tesla really are intent on mapping that environment visually, what kind of realistic redundancy would serve them in that scenario as a secondary input.

Think of it this way: Tesla — or say MobilEye if you want a more mature example — keeps working on their vision-based system and is generating a 3D environment based on the visual data. Great. You can drive against that data, when it is reliable enough.

But what kind of sensors would help ”make sure” or work as a sanity check? 360 degree radar and forward Lidar would in my view give best bang for the buck in that scenario if you can’t do 360 degree of both (360 degree vision + Lidar + radar would be best of course).

The reason for this is that the biggest threats that can approach an autonomous car from any direction are other moving cars. Radar is great at catching moving cars. So if your vision at that T junction turn or indeed highway lane change says nothing is approaching from the left but your radar says it is, you wait before moving. So again, the main view of the world is coming from visual but the double check would come from radar.

The biggest threats that can approach an autonomous car straight ahead are of different nature though. There are of course moving objects there as well (hence forward radar redundancy is good) but the biggest danger that vision might miss if it doesn’t recognize it and radar is bad at detecting are stationary objects. So the best redundancy to that visual ”3D engine” looking forward is Lidar that can say ”no stationary objects in the car’s path ahead” at high reliability. Again, if vision says all clear ahead but Lidar starts pinging, slow down.

I know I mentioned MobilEye in this message but I must hasten to add this is not how MobilEye has said they are going about their redundancy in their autonomous prototype though. I believe they plan on being able to drive on Lidar alone and then integrating vision and Lidar as fully redundant systems on their autonomous prototype. But some car maker using MobilEye’s vision system could integrate sensor fusion in this type of manner in my view and perhaps to some extent Audi’s Traffic-jam Pilot already does.

 

[dgatwood]

Cameras require enormous processing power and AI beyond what even a supercomputer is capable of.

Depends on whether you have enough cameras. Yes, if you try to do it with one camera per direction, that becomes an AI problem, but as long as you have at least two cameras, you can make a depth map (point cloud) without any AI at all. I'm pretty sure it was possible to do it at ~60 fps on commodity computers built even a decade ago. Modern hardware shouldn't even break a sweat.

 

[Daniel in SD]

Depends on whether you have enough cameras. Yes, if you try to do it with one camera per direction, that becomes an AI problem, but as long as you have at least two cameras, you can make a depth map (point cloud) without any AI at all. I'm pretty sure it was possible to do it at ~60 fps on commodity computers built even a decade ago. Modern hardware shouldn't even break a sweat.

Why doesn't Tesla do this?

 

[banned-66611]

Depends on whether you have enough cameras. Yes, if you try to do it with one camera per direction, that becomes an AI problem, but as long as you have at least two cameras, you can make a depth map (point cloud) without any AI at all. I'm pretty sure it was possible to do it at ~60 fps on commodity computers built even a decade ago. Modern hardware shouldn't even break a sweat.

In theory yes, in practice you have two cameras that are slightly out of sync capturing medium resolution video, partially blinded by rain or sun or reflections. Building a useful point cloud is ambitious to say the least.

 

[dgatwood]

Why doesn't Tesla do this?

They do. They showed a demo of it at Autonomy Day. In fact, your dual-camera cell phone almost certainly does it in real time. This isn't even hard anymore.

In theory yes, in practice you have two cameras that are slightly out of sync capturing medium resolution video, partially blinded by rain or sun or reflections. Building a useful point cloud is ambitious to say the least.

Being blinded is a problem, but rain blinds LIDAR, too. And sun reflections shouldn't be much of an issue for cameras unless the camera lenses aren't properly coated or they don't have adequate dynamic range.

 

[Daniel in SD]

They do. They showed a demo of it at Autonomy Day. In fact, your dual-camera cell phone almost certainly does it in real time. This isn't even hard anymore.

I believe what they showed at the demo day was using a neural net to compute depth from a single camera. Keep in mind that beyond 30 feet humans don’t use stereopsis for depth perception.
The cameras on the front of the Model 3 are only a few inches apart which is nowhere near what would be necessary to compute depth at driving distances.

 

[DrDabbles]

Eh, what the hell, I'll throw my lot in on this.

• 8 or 9 cameras
  • One long-range (narrow FOV) forward facing camera
  • One short range (wide FOV) forward facing camera
    
  • Four corner cameras looking 45 degrees out the A and C pillars (one on each pillar)
    
  • Two side cameras on the B pillars for curb and adjacent lane detection
  • One rear facing camera
  • All cameras are behind heated glass elements
  • Front and rear cameras have washer jets
• 360 degree radar
  • Mid-range side and rear radar
  • Long-range front facing radar
  • Heated radomes to prevent ice buildup
• 360 degree ultrasonics
  • Six front-facing short range
  • Six rear facing short range
  • Six side facing short range (3 on each side)
  • Ultrasonics angled slightly downward to detect curbs more consistently
• 360 degree solid state lidar
  • One forward facing module in the camera housing
  • Two modules in the B pillars (one each)
  • One rear facing module level with the forward facing module
  • Units should share the same heated glass element as the cameras where possible
• All necessary DSPs
• Custom designed ASICs and FPGAs for neural networks
• An army of people validating and labeling data night and day
• Multiple university research teams working on scene recognition and parsing with neural networks
• A few leading sensor fusion experts
• A team of lobbyists to smooth talk regulators
  
• Absolutely none of the ridiculous and dangerous object-to-vehicle communications. That idea is bunk and deadly.
• 10 years time to develop an industry leading, passable, but still seriously flawed system that will likely never be a good idea to let it drive itself in a "Level 5" manner.

Those are my thoughts. I think without any one of those, there's a serious risk of 100% failure.