Ant M3
Member
I’ve already completed the internet and am looking for the next challenge
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
FSD / AP Rewrite - turning the corner?
- Thread starter MrBadger
- Start date
That is exactly the kind of situation I would be concerned about on many UK streets. Imagine this video with a steady stream of oncoming traffic and both directions having to read the road well ahead to spot gaps to pull in to.
It is amazing that self drive is doing what it can in the current beta. But it still feels a bit like an advanced science fair project in some ways. Easy to take pot shots at, but very cool too.
Just after 16 minutes. Heavy braking for a cyclist coming out of the shadows on the far right corner of the junction. I wonder if this behaviour partially explains some of the emergency braking events some have reported?
Since starting this thread, I've been down a number of large rabbit holes trying to find confirmed as to exactly what "the rewrite" entails. Alas, each rabbit hole only gave a different view of the tree but I have managed to put together some thoughts on what it might be.
Unfortunately, the answer is that "the rewrite" seems to cover several things, but the good thing to come from it, or bad depending on how you look at it, is that we have probably had parts of it for some time.
It was widely assumed by many, including myself, that "the rewrite" was the 4D view, ie a consolidated 3D view of the world via a fusion of all external sensors, ie cameras, radar, ultrasonic. Then tracking that over time. From what I can gather, this 3D 'birds eye view' (BEV) has been with us for quite some time, quite possibly with the introduction of FSD preview, last December and traffic cones. It may even have come with advanced summon, which was a bit earlier. So we have had parts of it for some time.
What I am not clear about is whether all sensors formed part of this BEV fusion, or just some or all of the cameras. The other unknown is whether tracking objects through time is done or not. Part of me hopes not since many issues thought to be solved with "the rewrite" are still present.
However, the BEV fusion of sensors is not the whole picture. What is also missing are the recognition and identification/labelling of objects in this BEV fusion space (rather than individually from via each discrete sensor feed) and planning (such as taking evasive action or negotiating objects and junctions) what to do when faced with the labelled BEV scene.
I believe now that "the rewrite" that EM spoke about was this ability to label objects in a 3D scene. So rather than the system coming up with many different interpretations on what it is receiving from each sensor and trying to resolve conflicting information, it 'only' needs to interpret the scene once and not have any conflicts to resolve. Again, whether time is involved at this stage is unconfirmed, as is, is all sensor data included?
Empirically, I believe that, at least until recently, this has been work in progress and I suspect some aspects are incomplete, at least with the non FSD beta. I base this on several observations. First, there has been some recent change in behaviour for some functionality, eg recent traffic light functionality changed how close traffic light were detected, indicating either a change in camera, or a more complete view of the lights. Some regressions reported with phantom braking and lane keeping seem quite fundamental. Behaviour of static objects not persisting through time, such as 'dancing' lights at railway level crossings when warning lights flash - I have seen some similar references of this in FSD beta.
Another observation is the very distinct operational difference between the FSD beta functionality and visualisation for city streets, and the highways driving. Even on FSD beta vehicles, the look and feel of the car on highway driving seems to be identical to what we have for the same software version. Add in observed software modules from hacked vehicles showing city streets functionality as a discrete unit indicate that there is separate highways and city streets driving/FSD functionality. It is also clear that city streets driving/FSD is far more capable than highways.
So the questions are, will city streets and highways ultimately converge and, what if any, overlap will there be?
I personally believe that the perception from city streets and highways will significantly converge and will benefit all. I believe that the planning part of city streets will also extend to highways driving, but only for enhanced autopilot and FSD NoA, summon and autopark functionality. Driving on city streets, turns intersections, roundabouts, lane changing etc will remain a FSD thing only. I hope that safety features such as avoiding pedestrians, cyclists and parked cars etc will carry over for all. But I suspect that these aspects will not happen until FSD planning moves from being a software module to being handled in the FSD neural networks (NN), which I think will be where Dojo will start playing a role with its ability to learn to drive from simply watching videos.
imho.
To be honest my guess is “not much” .. Tesla want to drive ppl to buy FSD, so are incentivized to keep existing AP as-is. And they have repeatedly said AP and EAP is feature complete.
However, in the long term when Tesla have more competition they may make FSD standard, but dont hold your breath.
A couple of relatively short FSD beta videos to show low light capability.
If you only have a few moments to spare, the last minute or so of the first one is the most impressive imho.
As suspected, and one for @pgkevet , clearly the FSD computer has better perception than what is shown on the TeslaCam/screen etc. Wonder if it can detect a sheep bum in the bushes?
Also includes some visualisation and driving with headlights out.
If you only have a few moments to spare, the last minute or so of the first one is the most impressive imho.
As suspected, and one for @pgkevet , clearly the FSD computer has better perception than what is shown on the TeslaCam/screen etc. Wonder if it can detect a sheep bum in the bushes?
Also includes some visualisation and driving with headlights out.
oddly enough i was running down a single carriageway A road the other day on FSD (obviously not the new beta) and spotted a woman walking along on my side of the road towards me. The car braked (harder than i would have bothered) and with no oncoming traffic it straddled the white line to give clearance. Pedestrian showed up blue on my binnacle.
In truth if I'd been driving it I would just have lifted off rather than braked and I'd have only given enough clearnance to allow for a stumble or faint but better car over-reacts than otherwise.
It does ignore sheep bums in hedges at the moment on UK FSD
goRt
Active Member
IMO what we're seeing is the new software NN being trained on city driving with a very limited release to keep data volumes manageable with human oversight.
EM has stated that the 3D integrated video of this is next (rather than picture processing)
Then rapidly to 4D (planning)
Then roll out to a wider audience as only edge cases will need to be examined.
Adding (as we've seen in videos) speed humps, puddles...
Then this will move from city streets to highways, etc.
Functions will be software limited to the package you have purchased *for the car* ie FSD, EAP, etc. - there'll only be the single code base.
HW3 will be the requirement with anything less feature complete.
My understanding is the rewrite along with the whole 3D mapping etc will come to those with just AP, and therefore should improve AP and hopefully less issues and phantom braking, I would expect much new features from it, just an improvement to how AP is. Though this will be limited to HW3 of course, it's thought that will be the end of development for any older AP HW.
The current beta already is using 4D, which is essentially 3D integrated camera view with path mapping (time prediction). That's the basis of the entire beta FSD feature.
I agree once City Streets is in wider beta and is in "tweak mode" they will turn to EAP rewrite using 4D, then Smart Summon and Reverse Summon (smart parking). AT that point they will have an end-to-end 4D AP experience.
goRt
Active Member
Someone needs to tell musky that's how his beta's running as your position isn't supported by his tweets iro 4D
I think its confusion about terminology. The V11 software in beta features a reworking of the AI core so that it generates s single integrated view from all cameras (not literally a stitched together image, its unified at the object level). This includes a time dimension in that it also delivers movement data that allows path prediction.
This is what has colloquially been called the 4D view and forms the underlying tech on which the FSD beta is running (and would probably not be possible without it). It is also only possible using the much more capable NN engine in HW3.
Your timeline implied that FSD beta was using the old NN stack, and the new stack would come later, which is not true.
There was a distinct discussion about some form of top- down 360 camera view stitched from the cameras but that was a misinterpretation about an EM tweet about the object view (though that doesnt mean such a view is not possible, though it is tricky given the cameras orientations and quality).
Last edited:
goRt
Active Member
Correct, there is confusion, but not on my part
Musky clearly stated that in the current iterations the input into the NN is single frames and coming shortly is all camera integrated video input. The 360 birdview is an unrelated byproduct of this.
There is not currently 3D processing (the 3rd missing dimension being height).
My timeline did not imply the old stack for city streets, I reference the input into this namely single frames and not the coming video feed
Review musky's tweets and see if they fit your explanation or mine better
Watch Karpathy discuss the BEV nets (what Muck has called "4D" though its an inaccurate term) here (minute 17 onwards). This is the V11 stack that is now handling the FSD beta .. you can even see the similarity between his examples in the video and the on-screen display in the cars running the FSD beta. It's this re-write (which needs HW3) that is the foundational rework that has allowed Tesla to realize the FSD beta, as I noted in my earlier post.
Similar threads
