Tesla Autopilot maps

[scottf200]

Great find and interesting!
What about multiple lanes? There should be a flag that enable auto lanechange (or number of lanes data) as well I would think.

Seems like that would be a ton of wasted bits (storage). I would think you would only want exceptions. I would think cameras would 'easily' pick up multiple lanes (and center line) as well as the HD map data where they have seen previous cars traveling in those lanes.

 

[DamianXVI]

Can you link some of these too?

I didn't saved those so I started searching for them again. Because of this I realized that there is one more common reason for those places to show up - power cables. Low hanging cables above the road seem to be another source of radar echo. That explained some situations.

This time I assumed, that this value ranging up to 100 is a probability of existence (or something like that). That means we should not trust places with this value in low ranges, so I search only in entries with "probability" > 80 (although I still don't know if that is its meaning). Turns out it is quite hard to find those "unexplained" places that way. When looking only at the places with large "probability" almost always there is a sign/board/lights above the road, bridge, power cables, or trees most likely responsible. And there are a lot of sections caused by trees.

I had some strange places previously, now I can't find them. Here are couple examples of more unexplained ones that I managed to find this time:

Link
Link
Link
Link

What about multiple lanes? There should be a flag that enable auto lanechange (or number of lanes data) as well I would think.

There are a bunch of boolean flags (or other single byte values) for each spline that I mostly don't understand. I don't have any hint for their meaning, and all I can do is to look on the map for context and try to draw some conclusions.

There is a flag probably indicating that the section is a road with traffic in opposite directions (as not a road with some/any divider between the lines, even on local roads). There is a flag for parking lots (and only local parking lot routes, not just roads with parking spaces on sides).

There is a value indicating whether a path is an off-ramp, or an on-ramp, like this:


There is a flag active mostly for non-highway roads, and two flags active mostly for highways and highway ramps, that I'm not sure about their specific meaning. Here is an example (green when one flag active, blue when second flag active):


There are three more flags activated in patterns so random, that I don't even know where to start.

Structures in this file also have quite dynamic nature. That is, any field can be present or not, and it can grow in new versions with full backward compatibility. There are three fields that are empty in all tiles that I checked. Maybe they are not used anymore, or maybe I have bad luck. Anyway, there is still more information there that I still don't understand.

 

[verygreen]

There are a bunch of boolean flags (or other single byte values) for each spline that I mostly don't understand. I don't have any hint for their meaning, and all I can do is to look on the map for context and try to draw some conclusions.

There is a flag probably indicating that the section is a road with traffic in opposite directions (as not a road with some/any divider between the lines, even on local roads). There is a flag for parking lots (and only local parking lot routes, not just roads with parking spaces on sides).

There is a value indicating whether a path is an off-ramp, or an on-ramp, like this:
View attachment 261461

There is a flag active mostly for non-highway roads, and two flags active mostly for highways and highway ramps, that I'm not sure about their specific meaning. Here is an example (green when one flag active, blue when second flag active):
View attachment 261462

There are three more flags activated in patterns so random, that I don't even know where to start.

This mostly aligns well with the list I see in the code:

Branches have various properties like what road class is it, whenever the next branch exit is an offramp (to the left and right too), if the branch is itself a ramp, should autosteer be enabled, should autopark be restricted, is it a two way traffic branch, should "pedal misapplication mitigation" (?) be enabled, should radar braking be enabled, side collision assist enabled, length of this branch in meters, speed limit, speed limit units and some more cryptic stuff

 

[DamianXVI]

This mostly aligns well with the list I see in the code

Yes it does. It is still not clear exactly which field does what, but things simply make sense.

One of the missing things from your firs post that bothered me was the speed limits. I think I finally found them in the maps. It looks to be specified as one per spline in meters per second.

Each spline can have attached a four element structure, where every element is a single precision floating point number. When converted from m/s to km/h or mph (appropriately to the region) the first value (when not empty) ends up being a nice round number typical for a speed limit (50km/h, 70km/h, 90km/h, 25mph, 30mph, 50mph, etc.).

Other three values (when populated) are close to the first one, but are not round, and are mostly in ranges that make sense for a given road.

 

[_jal_]

Right now it looks like this:

• I'm finding Geohash of the place using an online decoder.
• I'm downloading the tile files using a web browser.
• Then processing this tile using a piece of code in C# that was created as a result of my experiments with the file format, and that right now is a horrible hackish mess with lots of hardcoded paths, and stuff...
• This code is creating an image, draws bounding boxes of geohashes found in it and on top of it draws spline paths (using Bezier curves) from the points read from the file.
• Next I'm grabbing a screenshot of Google maps and composing it with the image of paths in Paint.NET. Sometimes stitching a couple screenshots together. First images were combined using geohash boundaries as reference (to be sure that the roads actually match by their coordinates, not by me stretching them), last images I just combined so that the roads match with the map.

I believe this is not the full answer for your question... Right now I don't have much time, but I do intend to work on this more in a couple of weeks, and maybe (but just maybe) create some tool or viewer.

In the meantime, have a piece of San Francisco Bay Area.

Are you from the future?

 

[scottf200]

Here are couple examples of more unexplained ones that I managed to find this time:

Link
Link
Link
Link

In this one the google street view camera took a picture when it was next to a Tesla. That current Tesla car smelled a previous Tesla car like a bloodhound. AI is freaky!

Perhaps the low signage or Y.

Odd raise in the road by construction or the near road sign (mistaken as it was on the adjacent road)

 

[schonelucht]

This mostly aligns well with the list I see in the code:

Apologies if this has been asked before. Are you reviewing the assembly, or possibly decompiling binaries? Or is the code written in a higher level interpreted language that is available in human readable format?

 

[BigD0g]

Apologies if this has been asked before. Are you reviewing the assembly, or possibly decompiling binaries? Or is the code written in a higher level interpreted language that is available in human readable format?

He’s decompiling the code.

 

[DamianXVI]

I promised to provide some kind of viewer for map tiles, so here is a very basic tool.

First, and most importantly: Please DO NOT take the visualizations and information provided by this tool too seriously. I figured out the meaning of most of the data using "common sense", just by looking at the data in some context. I might be mistaken with some stuff.

This time it’s a .NET Windows Forms application, requires .NET Framework 4 to work. It is a simple viewer, nothing fancy – move with the mouse, zoom with the scroll wheel, click on the roads to show more info. It can load multiple tiles at once (and stitch them together), but it does not have any optimizations typical for map viewers, so don’t expect it to work fluently when displaying half of the district, and don’t expect it not to crash when loading half of the city.

Now about how the data looks. Main part of the database map is a list of splines (I previously called them paths). Spline is a curvature reassembling a section of a road. Single road can be composed out of multiple connected splines. Multiple splines ending/beginning in a single point can form an intersection, etc. Each spline have a set of parameters providing information about the section of the road. Some parameters can divide spline into smaller sections. You can click a road in the viewer to highlight a single spline and see its parameters.

In tiles that I examined I observer from 25 to 27 fields in a single spline record. At the beginning I named them F1…F27 and later changed the names when found out their meaning, so fields of unknown purpose are still called like that. Here is what I know about each field:

1. 32bit global ID of a spline. Used also to reference the connections between splines, and presence of a spline in smaller tiles.
2. Length of the road section in meters.
3. List of spline points/nodes. Each node have its location described as Latitude and Longitude. Coordinates are stored with full 32bit precision (that’s like 1cm accuracy), but its displayed with 8 decimal places in the viewer. Additionally there is a sort of control vector for each node defining road direction in node point. This vector is used to perform spline smoothing. I still don’t know how exactly the spline smoothing should be performed, so visualization might not be perfect. Usually the nodes are provided in order of driving direction, and control vectors are pointing towards the driving directions. Sometimes however things look different. I can’t say those are errors in the database, because I don’t know how should correct state look like, but I think that sometimes the system gets confused by the cars driving in opposite direction, close to the opposite direction line, at complex intersections or maneuvering at parking lots. It is not very common, but it results in glitches in viewer visualizations, so I’m trying to preprocess the splines and remove such occurrences by replacing the control vectors (but of course this is not perfect).
4. ID of next spline on the road in driving direction.
5. (F5) List referencing sections of the spline and providing two sets of four bytes for each such section. I only seen those bytes to be empty or equal to 16 (with bit 4 set). At some point I thought this may have something to do with the road lines. Now I think those are some sort of hints for line keeping. Maybe something like suggestion to keep on left or right, or use right or left line markings, but I don’t know for sure and it is not easy to figure out.
6. Unknown, empty in all tiles I checked.
7. (F7) Single byte Boolean flag, unknown meaning.
8. List of IDs of splines, with connections from this spline (following splines).
9. List of IDs of splines, with connections to this spline (preceding splines).
10. (F10) Floating point value. I think it may have something to do with the width of the road, like the distance from the center to the left edge.
11. (F11) Similar to 10, maybe a distance to the right edge, or other way around.
12. Probably a list, but empty in all tiles, so unknown.
13. Single byte value in range 0…100. I think this is a "Trust" value in percent. If so, this is important parameter to check, because it will indicate how reliable are other information describing the spline.
14. Structure of four floating point values, all most likely describing speed in meters per second. First looks to be a speed limit, others have unknown purpose/origin.
15. List with local speeds along the spline. This looks to be describing speed observed on the road with greater granularity. List does not refer to the specific points along the spline, so I assumed it should be spread along the whole spline in equal distances (it looks reasonable when visualized on the map). Each entry contains two fields, first looks to be speed, second might be an observed acceleration.
16. Ramp type, 1=On Ramp, 2=Off ramp, empty for splines that are not highway ramps.
17. List of radar “white list” zones. Zone is defined as a section of the spline. Like I said before, it also contains what look to be a probability of existence/confidence level/trust, and a height at which the obstacle is detected by the radar.
18. Boolean flag byte. Because it is close to the previous field I suspect it might be a “Radar Breaking Enabled” flag that verygreen was talking about, which looks reasonable on the map, but I’m not sure.
19. Unknown, empty in all tiles I checked.
20. List referencing off ramps down the road. For highway splines it references (by IDs) splines, that forms off ramps down the road, providing distance to them and some flags/types. Most of the times off ramp is formed from multiple splines, and all those splines are in this list.
21. (F21) Unknown Boolean flag, active mostly for non-highway/local roads.
22. (F22) Boolean flag active when the road does not have a center divider, also in case of local roads, although some exceptions can be found, so I’m not fully sure.
23. (F23) Boolean flag active for parking lot roads. This could be "Pedal Misapplication Mitigation Enabled" flag mentioned by verygreen, but that is only a guess.
24. (F24) Boolean flag, active mostly for highways and highway ramps, unknown purpose.
25. (F25) Unknown Boolean flag. I’ve seen it active only in tiles from "live" branch (not in "dev"), so maybe this is something that is about to be removed.
26. (F26) Boolean flag, active mostly for highways and highway ramps (like F24), unknown purpose.
27. (F27) Unknown single byte field, always equal 2.

Fields 26 and 27 are not present in some tiles from "live" branch, so maybe they were added recently.

Once again – I could be wrong about anything above.

Viewer have a list of “Highlight” modes that can visualize many of those fields (especially flags) with colors on the map. This is a way to make some sense out of them in the context. Those are not the best usage instructions for the viewer, but I hope you will figure it out.

One last tip: right clik on the map point to quickly copy its coordinates and be able to paste them into google maps search box.

 

[SucreTease]

I promised to provide some kind of viewer for map tiles, so here is a very basic tool....

Now about how the data looks....

@DamianXVI, this is absolutely brilliant work. I am a software engineer, so I know this isn't anywhere near as trivial or obvious as you make it out to be. You are to be commended, respected, and appreciated, dear sir.

 

[BigD0g]

Ditto! Fantastic work @DamianXVI and @verygreen !! Brilliant stuff really.

 

[Tjhappel]

Damian I am not a software engineer or anything close to it but!!!! This is some amazing reading and like the others said above your Level of detail is so awesome and it’s so fun to read and learn about what you’ve been able to uncover and learn.

Really from a semi laymen’s perspective but active reader of these forums/tesla driver and investor I can’t help but wonder if ANY of the other major car companies understand tesla has 200k +++ cars getting detailed maps from 8 different cameras with 1 cm accuracy?!?! It’s really crazy to think what things could look like driving an ap2 car in the next 6 months, thanks to several of you “super” members we all get a glimpse into the future.

So again, thank you guys for the awesome posts and all the hard work. It’s very very impressive.

 

[verygreen]

tesla has 200k +++ cars getting detailed maps from 8 different cameras with 1 cm accuracy

Reality check:
cameras do not produce maps.
There's no constant video feed from every car that feeds every camera footage into the mothership.

 

[chillaban]

Reality check:
cameras do not produce maps.
There's no constant video feed from every car that feeds every camera footage into the mothership.

I agree.

But in your opinion, how are these maps being created? A long time ago, a Sterling Anderson presentation made it seem like high resolution GPS and wheel odometry was contributing to fleet-learned maps.... But Tesla is one of those places where a year is like ancient history.

I presume if the car uploads high resolution odometry, you would’ve seen it already.

 

[verygreen]

I agree.

But in your opinion, how are these maps being created? A long time ago, a Sterling Anderson presentation made it seem like high resolution GPS and wheel odometry was contributing to fleet-learned maps.... But Tesla is one of those places where a year is like ancient history.

I presume if the car uploads high resolution odometry, you would’ve seen it already.

The reports were of Tesla drivers driving around on important routes and collecting data, sometimes redoing it when they captured wrong.

There is code (disabled by default, but you can enable it in dev mode) to collect and upload this data, I have not seen it activated on my car, but I did not pay close attention. It's possible they only enable it for specially trained drivers or some other conditions and otherwise guess the data from some sort of a preexisting map?
Things like number of lanes are not really detectable by odometry, neither are classes of roads driven, on/off ramp information and such - so there most definitely is some extra source of info for this.
I think fake radar targets are sent automatically, but I did not look too closely into this, what I know is there's a particular spot that I regularly drive by and I used to have FCW there 100% of the time and then it suddenly stopped. ap2 snapshot was not generated in any those cases (I do monitor these).

 

[lunitiks]

Things like number of lanes are not really detectable by odometry, neither are classes of roads driven, on/off ramp information and such - so there most definitely is some extra source of info for this.

With IMU, uBlox M8L, steering wheel torque sensor, inverter and brake intelligence, they should be able to produce high precision, offline maps without cameras or lidars.

They seem to be developing an inertial navigation system. From one of their job listings:

Other Potential Responsibilities
Contribute to design, implementation, and maintenance of in-car software with the following aims:
[snip]

• Vehicle localization (positioning) via development of an Inertial Navigation System (INS)

Why is it important to use an inertial navigation system (INS) on a mobile mapping vehicle? – OxTS
Bosch creates a map that uses radar signals for automated driving

 

[verygreen]

With IMU, uBlox M8L, steering wheel torque sensor, inverter and brake intelligence, they should be able to produce high precision, offline maps without cameras or lidars.

They seem to be developing an inertial navigation system. From one of their job listings:

Why is it important to use an inertial navigation system (INS) on a mobile mapping vehicle? – OxTS
Bosch creates a map that uses radar signals for automated driving

The reaity is inertial measurements right now are very off as is seen in various tunnels and the like.

Once they perfect it - they still can only somewhat approximate the lanes since they need multiple runs in every lane to confirm its existence, exclude shoulders, turning lanes in the middle and the like.
The on-off-ramp detection without cameras and preexisting maps seems improbable at the very least. same for detection of separated vs nonseparated highways.

 

[AnxietyRanger]

Superb @verygreen and @DamianXVI - you guys are THE BEST!

 

[AnxietyRanger]

By the way @buttershrimp - this thread could totally explain why we have different AP2 experiences based on location...

 

[DamianXVI]

(...) tesla has 200k +++ cars getting detailed maps from 8 different cameras with 1 cm accuracy?!?!

I think I need to clarify a bit. First, we still don't know how exactly those maps are created, so we can't say for sure that all Tesla cars are generating maps at the moment. We just don't know.

Second, this 1cm accuracy I mentioned is the accuracy of the mechanism of storing geographical coordinates in the file. This does NOT mean that the overall accuracy of the map is at the same order of magnitude. We don't know this. It does not even give us any hint about the accuracy of the map. All I can tell about it is that the person who choose the format for storing the coordinates did the smart thing. I only wanted to point out the actual precision of the values in the file when compared to how I decided to display them in the viewer.

Sorry for the confusion.