Neural networks technical discussions

[DanCar]

For very large neural networks (in the billions of parameters), the weights do end up approximating something like a database. That's how LLMs end up remembering specific facts about topics.

In terms of file size for those weights, we can look at Llama v2. Their 70 billion parameter model has 16 bit floating point precision, so it works out to about 130 GB.

I believe HW3 has 64 GB of flash storage, and even if it could hold 64 GB of weights, it wouldn't be able to process them due to 8 GB of memory per chip.

But this NN size constraint is a part of what induces generalization. The network isn't capable of solving the task it's being trained to do by memorizing specific answers, so it minimizes the loss by learning generic answers. And some of these generic learnings may help v12 avoid collisions in the first place.

 

[enemji]

Not only that, but localization data/algorithms will be loaded based upon the location of the car. No point of having San Francisco exceptions being loaded onto a car being driven in Austin TX.

 

[diplomat33]

Nice chart that summarizes the state of the art in AV:

https://twitter.com/x/status/1703656105916670149

The paper is 15 pages but a great read if you want to learn some details on E2E and the current state of the art in AVs.

 

[DanCar]

Expert gives his thoughts on how the bot is likely trained.

https://twitter.com/x/status/1705982525825503282