Is Tesla using machine learning for AP?

[BeesKnees]

Greetings,

I believe that with Mobileye there was a machine learning approach in play for AP. Post parting of the ways, there was lots of talk of "shadow" mode data collection and such. Do we believe that the plan forward for Tesla is machine learning based? Not, do we think is "should be". I'm wondering if it "is". What do we know?

Thanks.

 

[chillaban]

What exactly do you mean?

With regards to the computer vision approaches used by both MobileEye and Tesla Vision for image recognition, both are inherently machine learning style approaches, whether you'd like to call them neural nets, deep learning, etc etc etc.

However, in terms of the actual driving control algorithms, neither the one Tesla implemented in AP1, AP2, nor the stock one from MobileEye is machine learning per se. Those are human coded algorithms / control loops leveraging the data produced by the AI vision model.

 

[mongo]

Greetings,

I believe that with Mobileye there was a machine learning approach in play for AP. Post parting of the ways, there was lots of talk of "shadow" mode data collection and such. Do we believe that the plan forward for Tesla is machine learning based? Not, do we think is "should be". I'm wondering if it "is". What do we know?

Thanks.

I know that AP2 has multiple cameras, Nvidia processing power, a network connection to Tesla HQ, and that it requires drivers to give near continuous feedback regarding it's operation when using AP.

Since I don't work there, I don't know if they are taking the force feedback and braking inputs from the driver and feeding them into a NN to improve the control algorithm. I also don't know if they are using the drivers' control strategy and comparing it to what AP would do if it were active. I also don't know if they have flagged certain drivers as good examples of driving and other as bad examples. I do know I would be surprised if they weren't. ;-)

 

[BeesKnees]

What exactly do you mean?

With regards to the computer vision approaches used by both MobileEye and Tesla Vision for image recognition, both are inherently machine learning style approaches, whether you'd like to call them neural nets, deep learning, etc etc etc.

However, in terms of the actual driving control algorithms, neither the one Tesla implemented in AP1, AP2, nor the stock one from MobileEye is machine learning per se. Those are human coded algorithms / control loops leveraging the data produced by the AI vision model.

What exactly do I mean is always a good question for me. But this answer is what I'm looking for. Thanks. No AP algorithm. Vision systems are yes.

 

[BeesKnees]

I know that AP2 has multiple cameras, Nvidia processing power, a network connection to Tesla HQ, and that it requires drivers to give near continuous feedback regarding it's operation when using AP.

Since I don't work there, I don't know if they are taking the force feedback and braking inputs from the driver and feeding them into a NN to improve the control algorithm. I also don't know if they are using the drivers' control strategy and comparing it to what AP would do if it were active. I also don't know if they have flagged certain drivers as good examples of driving and other as bad examples. I do know I would be surprised if they weren't. ;-)

This possible NN for AP is what I'm curious about. I know Nvidia has done some full end to end machine learning on AP at least as an experiment. Wondering about Tesla control algorithm. Other reply says nope.

Thanks.

 

[chillaban]

This possible NN for AP is what I'm curious about. I know Nvidia has done some full end to end machine learning on AP at least as an experiment. Wondering about Tesla control algorithm. Other reply says nope.

Thanks.

I mean, I think an AI based control algorithm is interesting but for now I think that's still a science experience / research field. I think for now, the industry expects aircraft Autopilot style design, where software is explicitly written by humans to obey the rules of traffic, and can be formally verified and proven.

There's been a lot of interesting research in driving simulators and in video games for having a pure neural net trained the way you'd train a human. But I think that is years away from being a practical alternative.

To date I don't know of anyone in this industry actually productizing such an approach, and that includes MobileEye and Waymo.

 

[Tam]

...it "is"..

AI is a generic term for the other two since 1950's. It got more sophisticated and you got Machine Learning in 1980's. Now in 2010's, it got even better so you got "Deep Learning"

I believe Tesla is using all of them.

The Difference Between AI, Machine Learning, and Deep Learning? | NVIDIA Blog

 

[RubberToe]

Tesla has opened the floodgates of Autopilot data gathering

 

[chillaban]

Tesla has opened the floodgates of Autopilot data gathering

That's data gathering for training the camera vision neural nets, though.

What the OP is referring to is, for example, Follow the Rainbow Road - Training AI to Play Mario Kart | NVIDIA Blog

In Autopilot terms, you'd basically tell the car "ok here's two pedals that take inputs 0% to 100%, and a steering wheel that takes a force value. Here's millions of hours of footage of what outputs humans give for a given goal and visual input. Go learn!"

It's an interesting field of research, but not really something that I see any major player trying.

 

[JeffK]

One thought might be to check out the various other threads on autopilot and machine learning... we've had pages and pages of conversation on this topic.

 

[BeesKnees]

Thanks for the conversation. I did look and found some threads. Did not see anything that talked about current Tesla reality. Thus this post. Thanks again.

 

[stopcrazypp]

Thanks for the conversation. I did look and found some threads. Did not see anything that talked about current Tesla reality. Thus this post. Thanks again.

You are unlikely to find anyone that can say what Tesla is really doing unless an employee leaks it. Everyone so far has been guessing based on known information (aka they would have to qualify with "likely" rather than "is" as you are requesting).

 

[verygreen]

Here's the list of convolution kernels (I think this is called) from 17.17.4 that are run on nvidia cuda side:

b2f.cubin
bbias.cubin
bgemm_128x128x8_TN_vec.cubin
bgemm_32x32x32_TN_vec.cubin
bpool.cubin
copy_transpose_f4_27_4_4_26_40_0_3_1_4_2.cubin
copy_transpose_f4_5_16_16_26_40_0_3_1_4_2.cubin
f2b.cubin
i8conv_3x3_20x4_16_N1_q8.cubin
i8conv_3x3d2_20x4_16_N1_q8.cubin
i8conv_5x5_20x4_8_N1_q8.cubin
i8conv_5x5d2_20x4_8_N1_q8.cubin
sbias.cubin
sconv_direct_fprop_64x32_N1_bias_relu.cubin
sconv_winograd_2x2_3x3_32x32_K256_W40_Q40_N1_bias_relu.cubin
sgemm_32x32x32_TN.cubin
sgemm_32x32x32_TN_vec.cubin
sgemm_tn_128x128_vec.cubin

Here's mapping to more readable names:

  name: "conv1"
  class_name: "neon_convolution_kernel"
  kernel_name: "sconv_direct_fprop_64x32_N1_bias_relu"
  module_name: "sconv_direct_fprop_64x32_N1_bias_relu"
  name: "convert_to_int8"
  class_name: "tesla_conversion_kernel"
  kernel_name: "f2b"
  module_name: "f2b"
  name: "pool1"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3"
  module_name: "bpool"
  name: "conv2_1x1_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "conv2_1x1"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "conv2_1x1_bias"
  name: "conv2_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "pool2"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3"
  module_name: "bpool"
  name: "inception_3a_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_3a_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "inception_3a_merge_bias"
  name: "inception_3a_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_3a_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_3a_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_3a_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_3a_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_3a_pool_proj_bias"
  name: "inception_3b_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_3b_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "inception_3b_merge_bias"
  name: "inception_3b_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_3b_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_3b_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_3b_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_3b_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_3b_pool_proj_bias"
  name: "pool3"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3"
  module_name: "bpool"
  name: "inception_4a_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4a_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "inception_4a_merge_bias"
  name: "inception_4a_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_4a_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_4a_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_4a_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4a_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4a_pool_proj_bias"
  name: "inception_4b_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4b_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "inception_4b_merge_bias"
  name: "inception_4b_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_4b_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_4b_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_4b_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4b_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4b_pool_proj_bias"
  name: "inception_4c_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4c_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4c_merge_bias"
  name: "inception_4c_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_4c_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_4c_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_4c_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4c_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4c_pool_proj_bias"
  name: "inception_4d_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4d_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4d_merge_bias"
  name: "inception_4d_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_4d_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_4d_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_4d_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4d_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4d_pool_proj_bias"
  name: "inception_4e_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4e_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4e_merge_bias"
  name: "inception_4e_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3_20x4_16_N1_q8"
  module_name: "i8conv_3x3_20x4_16_N1_q8"
  name: "inception_4e_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5_20x4_8_N1_q8"
  module_name: "i8conv_5x5_20x4_8_N1_q8"
  name: "inception_4e_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_4e_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_4e_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_4e_pool_proj_bias"
  name: "pool4"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_5a_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_5a_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_5a_merge_bias"
  name: "inception_5a_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3d2_20x4_16_N1_q8"
  module_name: "i8conv_3x3d2_20x4_16_N1_q8"
  name: "inception_5a_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5d2_20x4_8_N1_q8"
  module_name: "i8conv_5x5d2_20x4_8_N1_q8"
  name: "inception_5a_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_5a_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_5a_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_5a_pool_proj_bias"
  name: "inception_5b_merge_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_5b_merge"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_128x128x8_TN_vec"
  module_name: "bgemm_128x128x8_TN_vec"
  next_kernel_spec_name: "inception_5b_merge_bias"
  name: "inception_5b_3x3"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_3x3d2_20x4_16_N1_q8"
  module_name: "i8conv_3x3d2_20x4_16_N1_q8"
  name: "inception_5b_5x5"
  class_name: "device_q8_convolution_kernel"
  kernel_name: "i8conv_5x5d2_20x4_8_N1_q8"
  module_name: "i8conv_5x5d2_20x4_8_N1_q8"
  name: "inception_5b_pool"
  class_name: "tesla_pooling_kernel"
  kernel_name: "bmaxpool_3x3_s1x1"
  module_name: "bpool"
  name: "inception_5b_pool_proj_bias"
  class_name: "tesla_bias_scale_kernel"
  kernel_name: "bbias"
  module_name: "bbias"
  name: "inception_5b_pool_proj"
  class_name: "openai_gemm_kernel"
  kernel_name: "bgemm_32x32x32_TN_vec"
  module_name: "bgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "inception_5b_pool_proj_bias"
  name: "convert_to_fp32"
  class_name: "tesla_conversion_kernel"
  kernel_name: "b2f"
  module_name: "b2f"
  name: "conv_pool_26x40_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_relu"
  module_name: "sbias"
  name: "conv_pool_26x40"
  class_name: "openai_gemm_kernel"
  kernel_name: "sgemm_32x32x32_TN"
  module_name: "sgemm_32x32x32_TN"
  next_kernel_spec_name: "conv_pool_26x40_bias"
  name: "conv_pool_13x20_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_vec_relu"
  module_name: "sbias"
  name: "conv_pool_13x20"
  class_name: "openai_gemm_kernel"
  kernel_name: "sgemm_32x32x32_TN_vec"
  module_name: "sgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "conv_pool_13x20_bias"
  name: "conv_pool_9x14_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_relu"
  module_name: "sbias"
  name: "conv_pool_9x14"
  class_name: "openai_gemm_kernel"
  kernel_name: "sgemm_32x32x32_TN"
  module_name: "sgemm_32x32x32_TN"
  next_kernel_spec_name: "conv_pool_9x14_bias"
  name: "conv_pool_5x7_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_vec_relu"
  module_name: "sbias"
  name: "conv_pool_5x7"
  class_name: "openai_gemm_kernel"
  kernel_name: "sgemm_32x32x32_TN_vec"
  module_name: "sgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "conv_pool_5x7_bias"
  name: "conv_PSP"
  class_name: "neon_convolution_kernel"
  kernel_name: "sconv_winograd_2x2_3x3_32x32_K256_W40_Q40_N1_bias_relu"
  module_name: "sconv_winograd_2x2_3x3_32x32_K256_W40_Q40_N1_bias_relu"
  name: "deconv_16x16"
  class_name: "neon_compound_deconvolution_kernel"
  kernel_name: "sgemm_tn_128x128_vec"
  module_name: "sgemm_tn_128x128_vec"
  next_kernel_spec_name: "deconv_16x16_shuffle"
  name: "deconv_16x16_shuffle"
  class_name: "neon_transpose_kernel"
  kernel_name: "copy_transpose_f4_5_16_16_26_40_0_3_1_4_2"
  module_name: "copy_transpose_f4_5_16_16_26_40_0_3_1_4_2"
  next_kernel_spec_name: "deconv_16x16_bias"
  name: "deconv_16x16_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_vec"
  module_name: "sbias"
  name: "reshape_to_input_27x104x160"
  class_name: "openai_compound_deconvolution_kernel"
  kernel_name: "sgemm_32x32x32_TN_vec"
  module_name: "sgemm_32x32x32_TN_vec"
  next_kernel_spec_name: "reshape_to_input_27x104x160_shuffle"
  name: "reshape_to_input_27x104x160_shuffle"
  class_name: "neon_transpose_kernel"
  kernel_name: "copy_transpose_f4_27_4_4_26_40_0_3_1_4_2"
  module_name: "copy_transpose_f4_27_4_4_26_40_0_3_1_4_2"
  next_kernel_spec_name: "reshape_to_input_27x104x160_bias"
  name: "reshape_to_input_27x104x160_bias"
  class_name: "tesla_bias_kernel"
  kernel_name: "sbias_128x1_vec"
  module_name: "sbias"

They are used with some sort of a neural net if this message is to be believed in the code: "constructured Tesla conv kernels for network".
I don't think they train the net in the car, though I don't know much about the subject.
There's a ~30M binary blob of the trained network data they use instead that is supplied.

 

[BeesKnees]

Fun. Thanks

And to the msg on expectations, I do not expect complete clarity. Not my first user group : ) but any info on what we know, think and believe is useful.

I am puzzling on ML strategies in general and adding what my car MAY be doing to the pile couldn't be passed up.

 

[Wander]

They are used with some sort of a neural net if this message is to be believed in the code: "constructured Tesla conv kernels for network".
I don't think they train the net in the car, though I don't know much about the subject.
There's a ~30M binary blob of the trained network data they use instead that is supplied.

That are indeed parts that are used with neural nets. Quite an interesting list, they seem to be using stuff from OpenAI, one of Elon Musks other compagnies and (parts of) Inception, which is a neural network architecture from Google.

It is very unlikely that they are using the cars to train the network, they don't have the bandwidth (shuffling around video files and neural networks is a bit much for a lte modem) nor do they have the computing power to train a big neural network.

Edit: here is a link to the Google inception neural network paper:
https://arxiv.org/pdf/1409.4842.pdf

You can recognize a lot of the kernels from Tesla in the inception paper

 

[3Victoria]

Here's my guesses:
AP1: Mobileye processes raw video to recognize objects, Tesla supplies the driving logic.
AP2 now: Tesla uses AI to recognize objects, and supplies the driving logic to achieve AP1 parity.
AP2 future: Tesla uses deep end-to-end AI, with tweaks, to achieve level 3/4.

The demo videos used the second method, but this code/AI code is not fully developed, so is only good for demos. It requires very large datasets, which they will collect from S/X/3s and simulations, to train the AI. The AI training will be done on large fast computers. The resulting AI/models will downloaded to the car to be executed by the NVidia AI/video processors.

 

[JeffK]

end-to-end AI,

Be careful when you use the term end-to-end AI because many people think end-to-end reinforcement learning which would absolutely not work for proper self driving even though it works for little demos like Nvidia's BB8.
More than likely it'd be a combination of a few different types of neural networks and some logic.

 

[verygreen]

Another interesting bit, the trained net data in 17.11.45 was ~14m, so it more than doubled to 17.17.4, I wonder what's the change in 17.22.46...

 

[chillaban]

Here's my guesses:
AP1: Mobileye processes raw video to recognize objects, Tesla supplies the driving logic.
AP2 now: Tesla uses AI to recognize objects, and supplies the driving logic to achieve AP1 parity.

It's also worth noting that the ME architecture uses a custom ASIC to run the computationally-intensive parts of the neural net, and the firmware portion runs on a few MIPS cores that pale in comparison to a Linksys router. I'm not sure how much they can achieve in terms of fundamental updates to recognition algorithms in the EyeQ3 hardware out there.

AP2 runs their neural nets on a purely software-defined GPU + ARM CPU setup. In theory this makes Tesla's architecture more flexible to change and improve with software updates, including dramatic redesigns of their vision pipeline should they choose to do so in the future.

 

[BigD0g]

Another interesting bit, the trained net data in 17.11.45 was ~14m, so it more than doubled to 17.17.4, I wonder what's the change in 17.22.46...

Sign recognition in shadow mode perhaps?