Less Wright: Reducing your labeled data requirements (2–5x) for Deep Learning

[strangecosmos2]

There's some cool new work out from DeepMind on self-supervised (a.k.a. unsupervised) learning. The results are described in a blog post by Less Wright (who was not involved in the research and is not affiliated with DeepMind):

“Some comparisons drive home the significance — image classifiers trained with CPC 2 and only 1% of ImageNet data achieved 78% top-5 accuracy, outperforming supervised (regular labeled training) trained on 5x more data.​

Continuing with training on all the available images (100%), CPC2 ResNet outperformed fully supervised systems, also trained on the full dataset, by 3.2% (Top-1 accuracy). Note that with only half the dataset (50%), the CPC ResNet matched the accuracy of fully supervised NN’s trained on 100% of the data.​

Finally, to show the generality of CPC representations— by taking the CPC2 ResNet and using transfer learning for object detection (PASCAL-VOC 2007 dataset), it achieves new State of the Art performance with 76.6% mAP, surpassing the previous record by 2%.”
​

Reducing your labeled data requirements (2–5x) for Deep Learning: DeepMind’s new “Contrastive Predictive Coding 2.0”

 

[strangecosmos2]

DeepMind plans to open source CPC v2, so anyone will be able to use it. Also, the idea of publishing a (pre-print of a) paper is that the results should be reproducible by other, unaffiliated researchers, so folks outside DeepMind should, in theory, be able to recreate CPC v2 now, even before the open source code is available.

Tesla is of course pursuing self-supervised learning for computer vision tasks such as depth mapping. Tesla is developing the Dojo training computer to accelerate self-supervised learning while reducing cost, per 2:26:42 in the Autonomy Day video:

“The car is an inference-optimized computer. We do have a major program at Tesla — which we don’t have enough time to talk about today — called Dojo. That’s a super powerful training computer. The goal of Dojo will be to be able to take in vast amounts of data — at a video level — and do unsupervised massive training of vast amounts of video with the Dojo computer. But that’s for another day.”​

Yann LeCun (a Turing Award winner and Chief AI Scientist at Facebook) has a nice talk on self-supervised learning. I found it to be an accessible, general introduction to the topic:

My personal mental framework for thinking about what Tesla is doing with deep learning breaks everything into these 5 categories:

• Fully supervised active learning for computer vision (Tesla example; Nvidia example)​

• Weakly supervised learning for computer vision (academic example)​

• Self-supervised learning for computer vision

• Self-supervised learning for prediction (Tesla example)​

• Imitation learning — and, possibly, deep reinforcement learning — for planning/decision-making (Tesla example; Waymo example)​

I find it helpful to mentally organize things into these 5 buckets. I hope it's helpful for others too.