Augmented Reality

[JeffK]

As for head and eye movement tracking - you can just about imagine how difficult and expensive it is to implement (imagine tall driver, short driver, driver who move the seat all the way forward, driver who recline the seat while driving, driver wearing sunglasses, driver wearing dirty glasses.... you get the idea).

That part is actually pretty cheap ... I mean even a snapchat plugin can do it with a cheap camera sensor.

 

[yesup]

That part is actually pretty cheap ... I mean even a snapchat plugin can do it with a cheap camera sensor.

That's why I said it was cheap to implement it on the center screen, as you are just overlaying graphics on top of camera sensor image - that's very easy to do, just like your snapchat example.
But this is useless in cars, as you are not driving while looking at a sensor image.

For AR with HUD, you are overlaying graphics with your eyes vision - what you see through the windshield! And that's totally different game and very difficult to do with perfect alignment.

 

[JeffK]

That's why I said it was cheap to implement it on the center screen, as you are just overlaying graphics on top of camera sensor image - that's very easy to do, just like your snapchat example.
But this is useless in cars, as you are not driving while looking at a sensor image.

For AR with HUD, you are overlaying graphics with your eyes vision - what you see through the windshield! And that's totally different game and very difficult to do with perfect alignment.

You don't need to be looking at the sensor... it can be embedded in the wheel and be watching you.

Here's a random picture I put through google Vision API. He's not looking at the sensor.

Here's a video on youtube of some guy doing eye tracking with a webcam

Can you tell where he's looking?

 

[yesup]

Can you tell where he's looking?

Your posted video actually demonstrates very well why AR is so difficult to implement well.
Check the eye tracking line in the video, and "cross hair" on the background - it is jittery and jumping around even though the head and eyes were perfectly still!

Imagine your overlayed AR graphics being jittery and jumping around like that?? Not acceptable in a car.
And in reality, you have constant head and eye movement instead of perfectly still head and eye position, the alignment and steady AR graphics is exponentially harder to achieve.

 

[3Victoria]

I think the chance of having AR with HUD in Model 3 is close to zero.

I think the chance is extremely high.
1. HUD is cheaper than a set of dashboard controls. It is simply a projector. In addition, the LCD screen could be replaced with a passive screen with the image projected onto it.
2. While eye-tracking would be useful, even without it the registering of the AR would not be difficult manually, and the overlay does not need pinpoint accuracy. However, eye tracking has been solved, see above.
3. HUD and AR are very programmable, ie upgradable.

Tesla Autopilot 2.0 Is AI, Plus Evidence Of A HUD (Video)
Large HUD Looks Likely In Tesla Model 3

 

[JeffK]

Your posted video actually demonstrates very well why AR is so difficult to implement well.
Check the eye tracking line in the video, and "cross hair" on the background - it is jittery and jumping around even though the head and eyes were perfectly still!

Imagine your overlayed AR graphics being jittery and jumping around like that?? Not acceptable in a car.
And in reality, you have constant head and eye movement instead of perfectly still head and eye position, the alignment and steady AR graphics is exponentially harder to achieve.

Human eyes are almost never still unless you die. One easy way to get rid of jitter is digital stabilization. It's trivial.

Even Porsche is doing it.

and Tesla hired the interior designer of this and a couple of other holographic UI design guys.

 

[Garlan Garner]

That's why I said it was cheap to implement it on the center screen, as you are just overlaying graphics on top of camera sensor image - that's very easy to do, just like your snapchat example.
But this is useless in cars, as you are not driving while looking at a sensor image.

For AR with HUD, you are overlaying graphics with your eyes vision - what you see through the windshield! And that's totally different game and very difficult to do with perfect alignment.

I love how you posted that.

"For AR with HUD"...........

Its extremely nice to hear that you get it.

 

[Garlan Garner]

Human eyes are almost never still unless you die. One easy way to get rid of jitter is digital stabilization. It's trivial.

Even Porsche is doing it.

and Tesla hired the interior designer of this and a couple of other holographic UI design guys.

That's a great example of AR without HUD.

Those graphic displays are extremely clear and not susceptible to the glare of the sun.

 

[dsvick]

I think the chance of having AR with HUD in Model 3 is close to zero. ...
Putting an unproven technology with hard to obtain parts which may disrupt production schedule is not wise for Model 3.

I think the chance is extremely high.

If there is a HUD I don't think it will have AR unless you also get EAP. At that point I would expect it would make use of the additional cameras and sensors to provide a lot more info to the driver. I do agree with @yesup about the unproven technology, that's why I think we'll see an announcement for the MS and MX getting it first. If we don't see that then I think a HUD is still possible for the model 3, but probably with no AR addition - at least not right away - have to love OTA updates!!

 

[dsvick]

I love how you posted that.

"For AR with HUD"...........

Its extremely nice to hear that you get it.

Yes, they are not the same thing, no one has said that they were. What most people have been saying is that they work better together. Did you read the rest of their post?

The only AR implementation which is of use and as seen in concept cars are AR with HUD (windshield).
Glasses are a no go from the start. Requiring people to wear glasses just to drive a Tesla simply won't work.

Yes, you can put AR in the center screen.... but what's the point? There is no use for it even though it is the cheapest to implement.
AR is supposed to seamlessly provide info about the environment while driving without taking the eyes off the road. Since watching the center screen means taking our eyes totally off the road, it really is no use there.

 

[S3XY]

Your posted video actually demonstrates very well why AR is so difficult to implement well.
Check the eye tracking line in the video, and "cross hair" on the background - it is jittery and jumping around even though the head and eyes were perfectly still!

Imagine your overlayed AR graphics being jittery and jumping around like that?? Not acceptable in a car.
And in reality, you have constant head and eye movement instead of perfectly still head and eye position, the alignment and steady AR graphics is exponentially harder to achieve.

I don't think AR in a HUD display would require determining which way your eyes are looking. It just needs to know where your eyes are positioned in relation to the windshield. It's not like it would need to shift the displayed image because your eyes are looking to the left or downward. So as in this video the subject is not moving his head, just his eyes. The display wouldn't have to change to accommodate this. Now if he turns his head or leans to one side the display would have to shift. That only requires tracking the position of the iris not the direction he is looking. Tracking his iris is very accurate in the video so AR would be very doable and could be reliably implemented. Also, the AR overlays will not be granular. If it's indicating which vehicle(s) it's tracking it'll probably just display a bracket slightly below the car(s). They're not going to create an exact outline of each car. So leaning towards one side will not change the placement of the AR that much, especially for an object that's 30 or 40 feet in front of you. For navigation in the Continental AR video they use < ^ > symbols to prompt where you should be heading. Again, it doesn't have to be perfectly placed in the center of the lane, As long as it's placed within the correct lane it's informative enough. I'm not saying the display doesn't have to be shifted with head movement, just that it doesn't have to be accurate down to the millimeter to be effective.

 

[3Victoria]

So as in this video the subject is not moving his head, just his eyes. The display wouldn't have to change to accommodate this. Now if he turns his head or leans to one side the display would have to shift. That only requires tracking the position of the iris not the direction he is looking. Tracking his iris is very accurate in the video so AR would be very doable and could be reliably implemented.

Totally agree with this.

 

[Sparky22]

As per Elon "Spaceship like display", I'm 99.9% sure there will be additional display incorporated into the front windshield, "Tesla glass" ring a bell?
Let's wait and see...

 

[LectrikPower]

I refuse to wait and see!

Tell me NOW!

 

[Garlan Garner]

I refuse to wait and see!

Tell me NOW!

I can wait. I'm saving a dollar a day for my tesla M3.

Lets see.......Only 28 thousand more days to go. LOL

 

[yesup]

Tracking his iris is very accurate in the video so AR would be very doable and could be reliably implemented.

But as I said before, there too many variables like sunglasses, dirty glasses, seat positioning etc. The fact that there is NO production cars with this feature would tell you that it is not a trivial feature to implement. AR with imprecise alignment is very off putting.

Also as I said before, there is NO production car with a HUD which has an angle of view wide enough for AR implementation. All the HUDs cover only a very small area in your line of vision (and they are already bulky and not cheap).

So you need to implement a lot of new and unproven technology/parts/components to make it work. And I don't think they would try it on the Model 3 first.

 

[JeffK]

But as I said before, there too many variables like sunglasses, dirty glasses, seat positioning etc. The fact that there is NO production cars with this feature would tell you that it is not a trivial feature to implement. AR with imprecise alignment is very off putting.

Also as I said before, there is NO production car with a HUD which has an angle of view wide enough for AR implementation. All the HUDs cover only a very small area in your line of vision (and they are already bulky and not cheap).

So you need to implement a lot of new and unproven technology/parts/components to make it work. And I don't think they would try it on the Model 3 first.

Irises are not terribly important for AR projection, what's more important is the head position and orientation.

Google doesn't seem to have trouble figuring out orientation with sunglasses...

Here's a video from over eight years ago. It's come a long way since then.

 

[Garlan Garner]

I just want the car.

For some reason lately - I have been feeling like Tesla isn't as far as they used to be concerning technology. AP1 ( partnering with another company ) seems to be so much better than AP2 ( Tesla going solo concerning the programing ) even in February 2017. With that said...I'm wondering if Tesla would be building its own AR if they choose to go that route.

 

[S3XY]

But as I said before, there too many variables like sunglasses, dirty glasses, seat positioning etc. The fact that there is NO production cars with this feature would tell you that it is not a trivial feature to implement. AR with imprecise alignment is very off putting.

Also as I said before, there is NO production car with a HUD which has an angle of view wide enough for AR implementation. All the HUDs cover only a very small area in your line of vision (and they are already bulky and not cheap).

So you need to implement a lot of new and unproven technology/parts/components to make it work. And I don't think they would try it on the Model 3 first.

And like I said before, since what will be displayed does not have to exactly outline a given object but rather display a bracket or some markings that would work sufficiently as long is it is placed within the appropriate lane, it does not have to be accurate within a few millimeters.

Regarding the width of HUD have a look at this. It also overcomes the issue of polarized sunglasses that comes up often as another negative of a HUD.

 

[yesup]

Irises are not terribly important for AR projection, what's more important is the head position and orientation.
Google doesn't seem to have trouble figuring out orientation with sunglasses...

Jeff, we would just have to disagree on the difficulty in AR implementation, so I would leave it as that.

So let's get back to the reality, there is still no hardware/software implementation in any production car. So if Tesla want components which are economic viable and reliable; and there would be plenty of parts available with no supplier worries for Model 3, it simply does not make sense to implement this type of unproven technology with limited benefit (both in terms of driving and chance of increasing car sales) for their base mass production models.