Waymo’s “commercial” ride-hailing service is... not yet what I hoped

[electronblue]

@strangecosmos Sure, I could sense there was history there. It is fair to point it out and thanks for letting me know.

But forgetting the forum this MobilEye REM mapping seems like the real deal right? Highly scaled autonomous level map ”training” running today inside production products scaling to a wide range of car manufacturers as we speak?

 

[strangecosmos]

this MobilEye REM mapping seems like the real deal right? Highly scaled autonomous level map ”training” running today inside production products?

What do you mean by “training”?

REM seems cool and important but I don’t see how HD maps would be the bottleneck to progress. It seems like Waymo’s hand-annotated lidar maps are better. More expensive, labour-intensive, and time-consuming to create, but essentially ideal. So, why do Waymo’s minivans in Phoenix struggle with left turns?

I don’t get the impression from the online courses I’ve taken etc. that HD mapping is the first, second, or third most difficult or important part of developing a self-driving car. I think it’s cool, but I don’t see how you could improve on what Waymo has and Waymo has not solved self-driving.

I also don’t automatically buy the claim that Mobileye’s visual HD maps are beyond what lvl5 or anyone else is doing. Maybe they are, but what’s the evidence? Companies with competing products often claim their product is the best, and of course they can’t all be right. So, how do we decide? I actually don’t even specifically recall Amnon saying that REM is a step beyond what the rest of the industry is doing.

Mobileye’s clear advantage with HD maps is scale — number of cars with a front-facing Mobileye camera in them. That allows for mapping more geographical area more quickly, and updating it more frequently. But that won’t do any good if the harder parts of self-driving aren’t solved.

 

[electronblue]

@strangecosmos Clarification. In this context by training I meant that they are collecting scalable amounts of data that can the in turn help other EyeQ4 cars drive better. Not to discount Waymo at all as said they have the Google benefit. But it would seem to me multimanufacturer EyeQ4 fleet sending back data could be pretty impressive and happening right now?

 

[caligula666]

It's called a drone and no not the only ones. Company in NY was doing this 6 months ago.

Which company?

 

[strangecosmos]

Is REM impressive? Sure. I’m not sure it’s really a meaningful competitive advantage or an accelerant of technological progress, though. Mobileye, Tesla, lvl5, and others might make HD maps of the entire world before the harder parts of autonomy are solved. I don’t know; I’m just saying it could happen.

I use to think HD maps might be a competitive advantage, but I’ve since learned how quickly and easily they can be made. So I don’t think that anymore. And I don’t know if any one company’s visual HD maps are way better than any other company’s. So ¯\_(ツ)_/¯

In my understanding, you just need to drive a car up and down a road a few times to make an HD map of it. I think Cruise estimated that if you had 1,000 vehicles in a city, you could update the entire city map every day. (Might be getting the numbers wrong, but it was something like that.) So you don’t need a huge amount of scale to make HD maps.

I’m most interested in things that affect neural network performance because it seems like that is where competitive advantage lies and what will accelerate technological progress. I may be wrong, but my sense is that for every unsolved problem in autonomy, the problem is unsolved because a neural network hasn’t reached a high enough level of performance yet. So, what increases neural network performance?

Just two things: 1) the design of the neural network: the architecture and hyperparameters, and 2) the training data.

So, how can companies attain competitive advantage or accelerate progress in neural network design?

How can companies attain competitive advantage or accelerate progress in training data?

I think this is a good, first principles way to think about the problem. Maybe it is a bad framing, but it’s my best effort at the moment.

If I had to bet on a company to develop the best neural network designs, it would be Google. If I had to bet on a company to collect the most training data, it would be Tesla.

A lot of neural network architectures are public domain, like GoogLeNet a.k.a. Inception v1, which according to jimmy_d Tesla used as the basis for its neural network architecture AKnet. It’s possible that Google has secret, proprietary network architectures much better than any public domain ones. I would be interested to know if there’s any hint of this out there.

Tesla seems to lead in training data, since HW2 Teslas have done billions of miles of driving and half a billion miles of Enhanced Autopilot driving, and Tesla seems to collect a lot of data from these miles. Diversity of miles is also more important than, e.g. collecting terabytes of data from driving around the same block thousands of times.

Assuming that Waymo is using a secret, breakthrough neural network architecture, I don’t know what will happen. Assuming it’s using an architecture that is only marginally better than AKnet, it seems like Tesla would win. This is how I think about the problem space.

This may be an incorrect conceptual framework, but I haven’t heard any better ideas yet.

 

[lunitiks]

... the camera is the only real-time sensor for driving path geometry and other static scene semantics (such as traffic signs, on-road markings, etc.). Therefore, for path sensing and foresight purposes, only a highly accurate map can serve as the source of redundancy.

In order for the map to be a reliable source of redundancy, it must be updated with an ultra-high refresh rate to secure its low Time to Reflect Reality (TTRR) qualities.

Road Experience Management™ (REM™) - Mobileye

 

[Bladerskb]

Its wierd to compare REM Map to something like lvl5 maps. One is simply just a demo with currently basic features and a site listing future features they want to support in the future and requires you uploading TBs of video data.

The other is already in production at scale and powering level 3 and level 4 cars.

Not even to bring up the fact that lvl5 lacks many of the requirements of a sdc map like semantic lane meanings, all the path delimiters, drivable paths,etc. Or even the accuracy and tolerance of their network for simple tasks like identifying lane marking and traffic light/signs.

Which is why they still collect video, not only that but they don't have access to the cars gps and obd info making their input inaccurate.

Or the fact that parts of their map process are still manually done and maintained.

I could go on and on

Cant compare a science project to something in production.

 

[electronblue]

@strangecosmos

Okay seems clear enough as theory. Here are some questions though. How useful is training data gathered in remote customer cars in training NNs? How far behind is Tesla with their perception, mapping and other things Waymo and MobilEye have already implemented ie how much of any perceived advantage is used to simply catch up? How large a role NNs even play in solving the last mile of autonomous driving ie does training data offer how big of an advantage there? How far from solving autonomous driving Waymo and MobilEye actually are?

One common argument against Waymo seems to be that they use mapping instead of just throwing a car into an area. How big a hinderance this is in reality is a unknown as we both noted from different angles, but in any case MobilEye has their solution for that. MobilEye is also doing all this with vision only so the question about Lidar being a crutch is also addressed in the competitive sphere already better than we have seen from Tesla. MobileEye also has scale.

Tesla does have potential deployment advantage in that once they have something to deploy they can do it quickly and piece-meal as needed. I agree they can collect vast amounts of data too. These can translate into interesting consumer experiences here today. I fear you may be too pessimistic or critical when assessing the abilities of the competition though. To me it seems obvious Waymo and MobilEye have a remarkably stronger solution in customer use today compared to Tesla.

Offsetting that is a tall order indeed when racing for the car responsible driving finishline.

 

[SOULPEDL]

New Which company?

Here's one. I thought it was in NY, maybe more...
Remote-control driverless car startup partners with vehicle manufacturers

FYI, this is how I think Waymo did their pilot in March '18. Video's of no drivers triggered social media to think they did it and ahead of Tesla... Smoke and mirrors!

 

[strangecosmos]

Thank you for the questions, electronblue. I am always happy to be given the opportunity to opine.

How useful is training data gathered in remote customer cars in training NNs?

I think the data collected from customer cars is as useful as any other data collected anywhere. A photo of a construction zone is a photo of a construction zone, whether it comes from a customer's car or an engineer's car. The advantage of collecting customer car data is the sheer quantity of miles driven (~2.4 billion miles) and the breadth and diversity that brings. The difficulty is that since the cars can't be recording everything all the time, Tesla needs to design triggers that start recording at certain times.

If path planning uses a neural network (which we have been unable to ascertain so far), then for urban, suburban, or rural driving — places where you can't use Autopilot — I don't think customer cars provide any useful data. For that you need engineer test cars.

On highways or other roads where people use Autopilot, I think it's possible to upload some data anytime a driver disengages Autopilot or the system aborts, and I think that data could be used for training if path planning does use a neural network (which, again, we don't know at this time).

How far behind is Tesla with their perception, mapping and other things Waymo and MobilEye have already implemented ie how much of any perceived advantage is used to simply catch up?

I think this is unanswerable question. For all we know, Tesla could be far behind Waymo and Mobileye, or Tesla could be far ahead. I just don't see how we can tell either way. Waymo and Tesla are both highly secretive about their development process and what they have in the works. Even if they were more open like Mobileye, I don't see how you could compare just on the basis of presentations given by the companies.

Getting back to my the theme of my first post in this thread, I think we need unfiltered observation of how a technology works in the wild, under unplanned, unstaged conditions. Or we need some other way to independently verify a technological lead of some kind. Just because a company says "our technology is great!" doesn't mean we should automatically believe them.

I certainly apply this to Tesla as much as anyone. Whenever Elon says, "our X is the best in the world" or our "our X team is the best in the world" I think, well of course he would say that. I'm sure he really believes it too; he loves his company and he's proud of the people he works with. But I'm not going to automatically accept that.

I think Tesla has the most promising approach to autonomy not based on Elon's claims, but based on independently verifiable facts like the number of HW2 cars on the road/number of miles being driven, the rate of data upload from each car, the speed of improvement of Enhanced Autopilot, and hacks by people like verygreen that show how Autopilot works under the hood. What I can verify independently, aside from the mere claims of Elon or the company, is what leads me to devise my optimistic theory.

I could found a startup tomorrow and proclaim to have by far the most advanced autonomous vehicle technology in the world. But it wouldn't be true. I am skeptical — but hopeful — about any secret technology whose capabilities haven't been independently verified. You can never tell until the curtain is drawn whether it's going to live up to the hype or be a real dud. (This is an ongoing topic of debate with General Magic, for instance). If with Waymo and Mobileye and everyone else in autonomy, I hope their eventual publicly available, commercial products live up the hype. But I'm not going to automatically believe hype with no evidence to go on besides what a company has said.

How large a role NNs even play in solving the last mile of autonomous driving ie does training data offer how big of an advantage there?

I'm not an expert in the field of autonomous cars, and I think even among experts there is some disagreement/uncertainty here. My personal sense of things is that autonomous cars are only possible because of neural networks, and that the remaining work to be done is bringing neural network performance up to the point where they exceed human performance on the same tasks.

With perception and prediction in particular, it seems like the sort of thing where we can't feasibly write a list of hand-coded rules that covers all the ways that pixels map onto the way the world is or the way that people's behaviour in the past few seconds foretells their behaviour in the next few seconds. It seems likely that Waymo, Mobileye, and Cruise still struggle with perception and/or predicting based on anecdotes of the failure modes they have, like getting frozen at left turns, running red lights, getting stuck behind food trucks, sideswiping buses, getting bumped into by trucks, etc.

Perhaps some or all these failures are failures in path planning instead of/in addition to perception and/or prediction. I believe Mobileye is applying deep reinforcement learning (i.e. a deep Q-network) to the problem of path planning. As I said, we don't know what Tesla is doing, but it would fit with their general philosophy to use a neural network for path planning as well.

So, my sense of things is that the problems that remain unsolved are unsolved because neural networks haven't gotten good enough at solving them yet.

 

[strangecosmos]

does training data offer how big of an advantage there?

To address the other part of your question, my sense of things is also that radically scaling up training data in tandem with neural network size is the most promising approach to making neural networks good enough at solving the problems that are still unsolved.

By radically scaling up training data, I mean scaling up by 100x or 1000x. The only company today that has the vehicles in place to do this is Tesla. Others may follow suit — and hopefully will — but none have publicly made a move toward that yet.

Research from Facebook and Google finds that you get big performance improvements on image classification by increasing the training data set into the hundreds of millions or billions of images. There are diminishing marginal returns, but that shouldn't necessarily bother us. If our goal is to get better performance, here is one way to do it — even if it is difficult and costly to gather that much training data.

Plus, Google makes an important note:

"It is important to highlight that the training regime, learning schedules and parameters we used are based on our understanding of training ConvNets with 1M images from ImageNet. Since we do not search for the optimal set of hyper-parameters in this work (which would have required considerable computational effort), it is highly likely that these results are not the best ones you can obtain when using this scale of data."
​

For a commercial R&D project with a big budget, you can afford to spend millions on computation to do hyperparameter optimization, if that's what it takes. You can also pay neural network engineers to design new, bigger neural network architectures purpose-built for the task at hand. If you're willing to spend a lot on compute, you can try using Neural Architecture Search (NAS) to see if you can discover new architectures that are better than what humans can design. There is potential to go far beyond what Google accomplished simply by scaling up the dataset.

It’s conceivable that a major fundamental breakthrough in neural network architecture could mean we suddenly need 100x or 1000x less training data to achieve the same performance. Humans drive less than 2 million miles in a lifetime, and learn to drive on less than 100,000 miles. But I think any neural network architecture that could solve autonomous driving with so little data could also solve a lot of other problems in computer vision and robotics. So what I’m saying is that Google or some startup came up with this new architecture, there would probably be a lot of fanfare, and it would be applied to many domains, not just autonomous cars. I doubt that it has already happened in secret.

How far from solving autonomous driving Waymo and MobilEye actually are?

I think it is impossible to say, even for the people inside these companies. Sergey Brin and Chris Urmson were overoptimistic about the timeline for the Google self-driving project. Elon gets a hard time for giving overoptimistic timelines, but one of the things that makes Elon different is that he openly gives lots of timelines (and another is that people really pay attention to what he says). It seems like in science and technology people are constantly making wrong predictions about the stuff they work on. A CEO or CTO can give you their best guess, but it's not like they actually know.

I almost feel like the years that people throw out there are meaningless. You can say "we're going to solve this problem by 2022!" but you can't actually predict scientific/technological/engineering progress. It happens on its own schedule and the universe laughs at your timeline.

One common argument against Waymo seems to be that they use mapping instead of just throwing a car into an area. How big a hinderance this is in reality is a unknown as we both noted from different angles, but in any case MobilEye has their solution for that.

If Google can update Street View once a year, why not HD maps? As I said, I think Cruise estimated once you have 1,000 robotaxis in a city, that's enough to update your HD maps of the entire city every day.

I once thought HD maps mattered competitively, but now I don't. Now I realize how easy it is to make HD maps and how many companies are doing it.

I fear you may be too pessimistic or critical when assessing the abilities of the competition though. To me it seems obvious Waymo and MobilEye have a remarkably stronger solution in customer use today compared to Tesla.

We'll see what Waymo has when we can see actual videos of Waymos driving that aren't made by Waymo's marketing department. Even then, we won't be able to make an apples-to-apples comparison with Tesla. Waymo's use of non-employees in its test vehicles is a money-losing R&D project, not a true commercial service. Tesla also has an R&D project, but it's kept secret. Unless we can compare Waymo's semi-public R&D project to Tesla's secret R&D project, we can't actually compare what Waymo has to what Tesla has.

I don't know why you say Mobileye has a stronger solution in customer use than Tesla. Nothing Mobileye has on the market comes close to offering the features that Navigate on Autopilot offers.

The Insurance Institute for Highway Safety (IIHS) tested Enhanced Autopilot against some driver assistance systems, which I'm guessing some or all of which use Mobileye:

"The 2017 BMW 5-series with "Driving Assistant Plus," 2017 Mercedes-Benz E-Class with "Drive Pilot," 2018 Tesla Model 3 and 2016 Model S with "Autopilot" (software versions 8.1 and 7.1, respectively) and 2018 Volvo S90 with "Pilot Assist" were evaluated."
​

On hills and curves, Enhanced Autopilot trounced every other system:

There has been some hype around Traffic Jam Pilot in the 2019 Audi A8, but it seems gimmicky to me. It doesn't seem to offer any actual additional functionality beyond regular ol' Traffic Aware Cruise Control (TACC). The only thing that differentiates Traffic Jam Pilot from TACC is the fact that it takes the human a little more out of the loop on divided highways at low speeds. The trade-off is some amount of added safety risk for some amount of added convenience. I don't see this as a significant advance in functionality at all. Also, there isn't a single customer in the world using Traffic Jam Pilot right now. The 2019 Audi A8 hasn't begun sales, and it's still unclear to me whether the feature will actually be available on day one or activated some time after.

What Mobileye-powered features do you think are better than what's offered in Enhanced Autopilot?

 

[strangecosmos]

I gave you a "disagree" for these comments:

frankly find your views disingenous

I think this is completely detached from reality.

Using those as examples is really a poor show on your part and disappointing.

This is super disrespectful and condescending. I'm putting you on my ignore list. You could have politely argued your point, but instead you insulted me. So bye.

 

[electronblue]

@strangecosmos I am sorry you feel that way. I found merit in your theory or at least its plausibility overall and tried to convey it in my opening. It seems possible Tesla could catch up through deployment speed and training the ”last mile” of self driving if they catch up in other ways. I think I was respectful towards your theory overall. It is possible Tesla could use these advantages to catch up and even pass the competition. Maybe Tesla could be the fastest to solve global path planning and deploy the system fastest for example.

I think your approach to Waymo and MobilEye in the two quotes I raised was below the belt however so I felt the need to point those two instances out. If my words seemed strong it was only because I felt strongly about those points.

Tesla is not far ahead of Waymo and MobilEye in perception and mapping. That is not in my view a realistic possibility at all and to somehow treat it as such was abhorrent to me. Also using past non-Tesla ADAS examples using MobilEye chips are not representative of EyeQ3 or EyeQ4 autonomous performance or apples to apples comparable to EAP in any fashion so those examples from you were not realistic to me at all. I think here you missed the mark and those harm your theory more than help it. You disagree and that is okay for sure but for me such views make your theory less believable not more believable.

 

[Bladerskb]

@electronblue

Its quite clear to everyone on this forum that Trent @strangecosmos has a distorted reality view and will downplay anything non-Tesla and can't handle when anyone calls him out on his BS. I think even @S4WRXTTCS and @lunitiks(for the first time ever) can agree with me on this.

You see, I can admit i'm pro mobileye, but even i don't drink my own kool-aid, i just sell it *cough* I mean..I know waymo has a currently diminishing lead.

When you have a guy who says Elon releasing level 5 software in 2019 has the same probability of happening as Level 5 taking till 2025 should tell you everything that you need to know about them and their bias.

Trent loves screaming abuse but its just that he can't handle people who challenge his kool-aid stand license.

 

[lunitiks]

Face it Blade I’m never going to agree with you on anything. ANYTHING. Like how many times do I have to be right before ppl start noticing? Hello, rear radar :-)

Anyway, back to the question of crowd-sourced, redundant perception, shall we?

 

[caligula666]

I can totally feel where @electronblue is coming from. Stating that Tesla might be ahead of the pack is not even funny. I understand this is to make a point but given that we bought our cars years ago, were sold a video that was a lie, and that despite Elon is Mr. Spin Doctor who will exaggerate any slight future features he has in sight, v9 is not yet available in most EU countries, I am chuckling at the idea they’d be working on an amazing UFO’esque version of SDC somewhere in a basement but yet would have just “almost-released” this joke of v9 to the “world”. ANY video I’ve seen shared on this forum with any decent traffic shows a car unable to be left alone for more than a few minutes before shadow breaks happen or that the driver has to take over. So, I’m all for opinions but this is probably touching a sensitive nerve for quite a few people who have been lied to (I have to much respect for Mr. Musk’s intelligence to think he was “confused” about the time it would take to make real progress, so I’ll go for either malice or fear of bankruptcy, or both)

Now, this thread (and others around) is pretty cool, would be amazing if some google doc (or other) could be iteratively and collaboratively enriched with the concepts introduced, defining the steps, options to do things, etc. as it is pretty complex and confusing to the newcomer. Just a self-serving idea from somebody who would totally enjoy and benefit from it

 

[electronblue]

Thank you for the sympathy guys. The false equivalence of saying Tesla may be far behind or far ahead of Waymo or MobilEye on perception and mapping was just too much for me. That is simply not supported by anything that Tesla might be far ahead of Waymo or MobilEye in that.

It is one thing to say the road to autonomous driving has many obstacles and we don’t know the eventual winner yet — it might even be Tesla if others falter and they are quick in the ”last mile” due to their deployment advantage or somesuch — but to claim perception and mapping which are some of MobilEye’s and Waymo’s publicly known bread and butter and great advantages could somehow be far behind Tesla who has shown nothing but being far behind in perception and shown nothing in mapping...

No, that claim from @strangecosmos especially was too much for me. It was outrageous.

 

[Bladerskb]

Thank you for the sympathy guys. The false equivalence of saying Tesla may be far behind or far ahead of Waymo or MobilEye on perception and mapping was just too much for me. That is simply not supported by anything that Tesla might be far ahead of Waymo or MobilEye in that.

It is one thing to say the road to autonomous driving has many obstacles and we don’t know the eventual winner yet — it might even be Tesla if others falter and they are quick in the ”last mile” due to their deployment advantage or somesuch — but to claim perception and mapping which are some of MobilEye’s and Waymo’s publicly known bread and butter and great advantages could somehow be far behind Tesla who has shown nothing but being far behind in perception and shown nothing in mapping...

No, that claim from @strangecosmos especially was too much for me. It was outrageous.

Well here's the thing. The reason eyeq3 implementations have sucked is because the development are outsourced to tier-1s who do the bare. For example delphi sources almost all of the adas features for almost every car company. The very few companies that don't outsource it and bring it all inhouse actually deliver great products (tesla with ap1 and Gm with supercruise).

For example GM has around 20 engineers working on supercruise, Tesla famously had 100+ engineers that worked on AP1.

Other car companies simply drop in the trash that Delphi give them.
Not only that, we know that mobileye eyeq4 is used in FSD cars as the main vision system so we know they are capable of much more.
So the logic that @strangecosmos uses to scoff mobileye tech just doesn't add up.

 

[acoste]

Question here, how many useful miles does Tesla collect from users? I don't think the cars have the bandwidth to stream all data constantly do they? Even if it was about using wifi when parked, it's 1Gb+ of data for every battery charge. I guess it is event based, and logs only the critical situations. But a lot of critical situations are not detected by the autopilot.
So how much of autopilot driven miles are actually recorded and useful?

 

[Bladerskb]

Anyway, back to the question of crowd-sourced, redundant perception, shall we?

Google's Street View neural network can now decrypt captchas better than a human - ExtremeTech

Well Sacha Arnoud Director of Engineering for waymo said Google's House number detection network back in 2013 needed around 2.5 billion images for training. But today because of better nn architecture and better data, they don't need anything near that.

Here's what he said "Today we do alot more better. we require less data and can generate those dataset more efficiently. it very common to still need in the millions ranges to train a robust network".

Based on what Sasha is saying, they went from needing 2.5 billion to build robust networks to something like ~10 million. That's a decrease of about 250 x of dataset size.

Its quite obvious that better architecture, quality of data and labeling is better than simply having more data.
The size of dataset required will keep coming down until the day you can show a network or whatever software 1 picture of a cat and they will be able to recognize cats.

Waymo and Google work hand in hand, so not only does Waymo have access to the best architecture, but also the best dataset aswell.

I'm sure everyone here has heard of Google duplex which is entirely based off wavenet which google published.

Google AI Blog: Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone

Everyone and their grandma have tried to implement wavenet to get the same results that matches google and has failed.
The prevailing theory at /r/machinelearning is that it requires a secret sauce of quality labeled data that only google has.
What does that say? it says that better architecture and quality labeled data is what is driving improvement in AI, not the increase in size of dataset.