Search results

Overall, the stuff in the talk on computer vision is fascinating and the 20-minute demo video is very cool. Impressive that the whole system is running on 8 cameras and an EyeQ5 chip. EyeQ5 does 24 TOPS. That's 1/3 the 72 TOPS that Tesla's FSD Computer does (or 144 TOPS total if a redundant...

At Autonomy Day, Elon said: “The car is an inference-optimized computer. We do have a major program at Tesla — which we don’t have enough time to talk about today — called Dojo. That’s a super powerful training computer. The goal of Dojo will be to be able to take in vast amounts of data — at a...

I know there was some confusion in this thread about the different subsystems of the self-driving car software stack, so I wanted to share a helpful diagram from Waymo: As you can see, there are four main subsystems: 1. Perception (including computer vision, radar-based perception, and...

Currently watching the talk. The most interesting part so far is the section around 20 minutes in on perceptual redundancy within the computer vision (i.e. camera-based perception subsystem). Slides:

This is completely untrue. SandiaGrunt a.k.a. tacocat a.k.a. HappyFunBall a.k.a. stylistic_seven a.k.a. MLAVADAS was banned twice from TMC and twice from r/SelfDrivingCars. I'm not a mod in either community, so clearly the mods in both communities found something wrong with their behaviour. I...

How about Andrej Karpathy, Mayank Bansal, Drago Anguelov, or Sebastian Thrun? I gave direct quotes and/or linked to talks from all of these experts in this thread. Don't take my word for anything; just look at the sources I cited. No, I'm not. Look at the title of this thread. It says...

Isn't Tesla doing something along these lines with stop signs? I recall Elon mentioning this as a possibility and I also recall verygreen saying maps were involved in stop sign detection. u/brandonlive on r/SelfDrivingCars suggested Tesla could use maps to detect false negatives. If the map...

I’ve been on the Internet long enough to learn that when people say stuff like this... ...they’re often wrong. I think the reason for this correlation is simple. Epistemic humility begets learning. The more unsure you are about the correctness of your own views or hunches and, conversely, the...

This aged poorly: :p (No offense, croman.)

Forgot to mention: Sebastian Thrun — the guy who opens the video above and helps teach the course — co-founded the Google self-driving car project (now called Waymo). Serious credentials!

There is a small alpha tester fleet (including Elon's car) where the cars will semi-autonomously drive on residential streets, dense city streets, highways, and parking lots. That's way beyond shadow mode. Depending how you define "shadow mode", some form of it is probably running on regular...

@DrDabbles, do you understand the difference between control (i.e. low-level actuator commands) and planning (i.e. making decisions and determining a trajectory) in an autonomous car context? Do you understand that HW2+ Autopilot currently uses a neural network trained via imitation learning...

Andrej Karpathy is a world-class deep learning researcher. Why do you think Karpathy believes in using imitation learning for autonomous driving tasks like path prediction and lane changes? Mid-to-mid imitation learning doesn't need to understand the world from scratch. It uses the computer...

What I have in mind is actually mid-to-mid imitation learning, not end-to-end. Computer vision NN representations in, plan out.

But if to extract 100 years of useful driving data from your fleet for imitation learning, you need 40,000 years of total driving, then you need a large fleet to do this in a practical amount of time. If you have 1,000 cars driving 24/7/365, that will take you 40 years. If you have 10,000 cars...

Sebastian Thrun, a co-founder of the Google self-driving car project (which is now Waymo), explains planning: Thrun explains control: These videos are from the free online Udacity course Self-Driving Fundamentals. What I'm saying is that neural networks and imitation learning are involved...

Udacity has a free, short, and accessible online course called Self-Driving Fundamentals (made in collaboration with Baidu's open source self-driving car project, Apollo). Self-Driving Fundamentals: Featuring Apollo | Udacity

To clarify: my assertion is not that neural networks are involved in Tesla's controls software. My assertion is that — per Andrej Karpathy's remarks on Autonomy Day — neural networks are involved in Tesla's planner. Path prediction is the primary example of this. My point is just that Tesla is...

Yes, I believe this is correct. Path prediction might be an instance of what's known as mid-to-mid imitation learning, wherein the planner or some elements of the planner are neural networks trained via imitation learning, but 1) the computer vision system is trained via traditional fully...

What's your evidence for this? Andrej Karpathy was quite clear at Tesla Autonomy Day (2:13:33) that Tesla is using imitation learning for path prediction, which — according to Karpathy — plays an important role in determining Autopilot's trajectory through highway cloverleafs. He said (my...

Just noticed one instance where Karpathy used the term "shadow mode" on Autonomy Day:

Maybe you could elaborate?

So, I already shared a spreadsheet that made conservative assumptions about HW3 vehicle production, etc. I just created a new version of the spreadsheet that makes more aggressive assumptions: Tesla's Imitation Learning Fleet: Years of Continuous Driving (projected) Constructive feedback...

I suspect the orange ribbon only represents part of the Autopilot planner and the part that stops for/follows a vehicle ahead (e.g. while lane-keeping on the highway) is not visualized in green’s videos. I’ve asked green to weigh in on this thread; hopefully he can shed more light on the matter.

So, is the consensus that the Autopilot path planner runs passively in the background when Autopilot isn’t engaged? If so, that all but confirms shadow mode, as Elon originally described it, does indeed exist in some form. We have already established that the computer vision system runs...

@DrDabbles, what particular point(s) do you disagree on?

Andrej Karpathy made another reference to imitation learning at Tesla Autonomy Day at 2:28:18: "Ultimately, actually designing all the different heuristics for when it's okay to lane change is actually a little bit intractable, I think, in the general case. And so ideally, you actually want to...

If Tesla uploads a data snapshot for some/most/all crashes or "crash-like events" (which it apparently tracks, per its Safety Reports), and if that snapshot includes the "orange ribbon" and other planning-related data, then this would allow Tesla to determine whether Autopilot would have averted...

12:50 in this video. Isn't the "orange ribbon" in green's videos the path planner/path predictor running passively in the background?

Not at all. Tesla uploads data from the fleet to train neural networks on computer vision, prediction, and planning:

Indeed. green's videos demonstrate this. So, we know for a fact that computer vision NNs run passively in the background. Whether you want to call this "shadow mode" is purely semantic and irrelevant to the OP of this thread. The evidence for it is simply that Elon's mentioned it in the past...

green's tweets (which I already linked to earlier in this thread) are consistent with how I envision shadow mode. I don't know where people are getting the idea that shadow mode means a car records data of every second it's in motion and uploads it all to Tesla. When did Elon ever say that? In...

Untrue. For example, when a HW3 Tesla running Software v10 drives through a highway cloverleaf on Autopilot, its trajectory is determined in part by a neural network: At least some of it exists, per the Karpathy video above. It isn't magic, it's imitation learning. In end-to-end imitation...

A clearer way of saying this would be: “5% of cumulative years is useful to collect”. This is a bit silly to include, actually, since I assumed Model S/3/X production would be basically flat from Q3 2019 to Q2 2025. That probably won’t even be true for Q4 2019, but I wanted to be conservative...

Relevant: How Tesla could potentially solve “feature complete” FSD decision-making with imitation learning

Source?

DeepMind’s AlphaStar Supervised is better than 84% of ranked human players at StarCraft II and it trained purely via imitation learning on about 1 million human-played StarCraft II games. At an average game time of 30 minutes (a high estimate), that’s about 60 years of continuous play. Those 1...

I believe green's claim is nothing more than that HW2/3 Teslas aren't recording video or object detections all the time and uploading all of them. I think continuous recording is what green interprets as “shadow mode”, but I don't interpret it that way. I interpret shadow mode as being just...

I don't think this is totally new, actually. I recall seeing this in 2017, I think. But the problem then is the car would drive in the centre of the country road so it wouldn't leave room for an oncoming car to pass.

I suspect this problem is soluble with driving because it was soluble with StarCraft:

I think actually only Teslas built from October 19, 2016 onward have HW2: All Tesla Cars Being Produced Now Have Full Self-Driving Hardware Maybe some built before that date also have it, but I would ask Tesla to give you a definite answer on whether the Model S you're looking at has HW1 or...

Yes, that's my understanding too. I'm really curious to see what happens when the new perception NNs are actually used in Autopilot. Will the dumb behaviours of the sort that @PhaseWhite describes disappear? The thing that makes me feel more confident that decision-making is solvable than...

1) Isn't verygreen using HW2 (or HW2.5) in the videos? 2) What I'm suggesting is that both generations of NNs are running on HW3. If verygreen is using HW3, he may be capturing visualizations from the HW2-gen NNs. I think shadow mode is most likely real. The reasons I think so are: 1) Tesla...

Rewatching this and Karpathy explains what’s described in the patent at 5:30:

@greentheonly, December 23: “Don't have root access on my hw3 unit”

I don’t think green has access to a car with a rooted HW3 computer yet, though.

True, the Q2 update letter may be referring to a smaller, HW2-compatible version of stop light and red light detection.

Thanks. So, what Mobileye has patented appears to be a path planner that includes both a mid-to-mid imitation learned module and an end-to-mid imitation learned module:

u/topper3418 on Reddit pointed out this excerpt from Tesla's Q2 update letter, which was published on July 24, 2019: “We are making progress towards stopping at stop signs and traffic lights. This feature is currently operating in “shadow mode” in the fleet, which compares our software...

Which post? Don't see the link you're referring to. The thing I want to distinguish is a fully end-to-end system like Wayve's and a mid-to-mid system like ChauffeurNet and a system like Tesla's that may (or may not) use end-to-mid for one or two or a handful of tasks — like path prediction* —...