AI News

[Terminator857]

Deep mind says their language model has the reading comprehension of a high schooler.

https://storage.googleapis.com/deepmind-media/research/language-research/Training%20Gopher.pdf

Quote:
We highlight two reading comprehension tasks RACE-m and RACE-h, multiple-choice exams
pitched at a middle-school and high-school level respectively. Inspecting the accuracy in Table 4
we see Gopher extend upon the current LM SOTA for high-school reading comprehension (47.9%
Megatron-Turing NLG → 71.6% Gopher) and the middle-school comprehension accuracy (58.1%
GPT-3 → 75.1% Gopher). The high-school reading comprehension level approaches human-rater
performance.
/quote
If that were true and if we extrapolate that to human level intelligence of a high schooler, then we have the AI breakthrough needed for FSD. (Yeah, I know a lot of ifs). So we only need FSD hardware 6 with 260 billion parameters in the neural network to get real A.I.

 

[Terminator857]

Youtube version at: 3:05:

 

[Terminator857]

Retro from deepmind uses a database to outperform neural networks 25 times its neural network parameter size.

[2112.04426] Improving language models by retrieving from trillions of tokens

arxiv.org

https://arxiv.org/pdf/2112.04426

Quote:
With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters.
/quote
The above youtube video discusses it at 7:22 into the video:

 

[Terminator857]

List of companies developing A.I. accelerators:

AI-Chip

A list of ICs and IPs for AI, Machine Learning and Deep Learning.

basicmi.github.io

1. Lightelligence.
2. Cerebras.
3. Habana.
4. Google Tensor Chip.
5. Intel Loihi 2.
6. Tesla Dojo.
7. Untether AI.
8. startup Innatera Nanosystems.
9. startup EdgeQ.
10. startup Quadric.
11. startup Analog Inference.
12. Tenstorrent.
13. Google.
14. SiMa.ai.
15. startup Neureality.
16. Cerebras.
17. Groq.
18. Nvidia.
19. SambaNova.
20. Baidu.
21. Deep Vision.
22. Flex Logix.
23. Tenstorrent.
24. Synaptics Katana Platform.
25. Graphcore MK2 PERFORMANCE BENCHMARKS.
26. SambaNova.
27. Esperanto's ML Chip.
28. AWS Trainium.
29. startup SimpleMachines.
30. SK Telecom SAPEON X220.
31. Imagination AI accelerator.
32. Mythic.
33. MLPerf Inference Results 0.7.
34. Qualcomm Cloud AI 100.
35. MLPerf Training Results 0.7.
36. Neural Network Accelerator Comparison in Reference.
37. AIchip Paper List in Reference.
38. Nvidia A100.
39. Sony's Intelligent Vision Sensors.
40. Wave Computing.

Funding received by different A.I. hardware startups:

AI Funding Spree: +$300m for Groq, +$676m for SambaNova

www.anandtech.com

AI AcceleratorsâââPart IV: The Very Rich Landscape

One of the hottest fields in the global tech industry is AI hardware. Billions of Dollars have been poured at companies that developâ¦

medium.com

Quote:
Billions of Dollars have been poured at companies that develop accelerated AI solutions via startup acquisitions and fundings, and via the stock market. In this article, we will go over the current state of the AI hardware industry and present an overview of the different bets companies are making in finding the best way to tackle the AI hardware acceleration problem.

 

[Terminator857]

https://twitter.com/x/status/1482117604399480833

Neural net architecture that requrires little or no training will win over those that require lots of labeling , even lots of auto labeling.

https://twitter.com/x/status/1482117604399480833

 

[AlexVincki]

Out of interest how FSD works, and a lot of free time during covid, i set myself to develop a Full Self Gliding autopilot for a semi professional flight simulator.
The result is really satisfying (broke all the records) and learned a lot of the challenges Tesla is facing

 

[Terminator857]

https://twitter.com/x/status/1483741343335821313

Elon says they want to help stop A.I. overlord takeover with decentralization. Tesla bot should help.
Quote:
Tesla AI might play a role in AGI, given that it trains against the outside world, especially with the advent of Optimus
> Non Elon comment: Glad the foundational seeds of AGI are in the hands of a benevolent company, led by someone who is aware of—and extremely cautious about—AGI gone awry.

Will do our best. Decentralized control of the robots will be critical.

 

[mspohr]

Looks like your frying pan can run AI:

Science | AAAS

www.science.org

Everyday objects can run artificial intelligence programs ​

Nontraditional hardware could help neural networks operate faster and more efficiently than computer chips​

 

[Terminator857]

Yannic Kilcher tells us that research is shifting towards more learning from data. Learning about the world by analyzing data versus programming A.I. Similar or same as unsupervised learning. 4:30 minutes: Yannic smartly declines to make an estimate when FSD will be real.

Tiimestamps:
00:00 - Introduction
00:58 - Impressions of Tesla FSD
03:00 - Fundamentals of vision and autonomous driving
06:10 - Computer mistakes vs human driving mistakes. Yannic suggests people won't tolerate the computer making mistakes. People won't tolerate robot cars killing people.
09:05 - Lidar integrating with vision - More data is probably better. Known in the field as multi modality.
11:50 - Pros and cons of Lidar vs vision only - More data is probably better
13:20 - Who is in the lead for physical world AI - Yannic: Don't know
15:16 - Can Tesla expand into other fields of AI? - Yannic: Yes
17:30 - Importance of data to physical world AI? - Yannic: Very important
18:35 - Data advantage in autonomous driving? - Quality data is more important
20:16 - AI expertise vs data advantage - No secrets in A.I. research.
21:35 - Tesla Dojo - Yannic: Will save Tesla in training costs. Research improvements have come at the cost of exponential costs in compute.
25:44 - Limits to neural nets performance - Improvements in models have just come by just creating bigger models.
29:00 - Tracking AI improvements
31:25 - Github Copilot, DeepMind AlphaCode, OpenAI Codex - github copilot is a game changer
32:15 - Main challenges in creating AI human robot
34:27 - Why is artificial intelligence important? - Great a perception problems... Everything that recommends content on the internet is powered by A.I.
37:00 - Augment jobs or replace? - OP opinion: People have worried about this for decades, nothing new.
38:36 - Who’s leading - OpenAI, DeepMind/Google, Facebook, etc. - Hard to say. Amazing things coming almost weekly from all sources.
30:20 - Nvidia thoughts - No idea if Nvidia hardware will continue to dominate.
41:45 - GPT-4 expectations - GPT3 training was painful money wise. Training GPT4 will cost 10x more if they just use the bigger model theory.
43:00 - A.I. Bot. Hard because walking on two legs is hard. Walking in unfamiliar environments hard.
48:30 - Conclusion - Yannic is working on an A.I. startup processing legal documents.

 

[Terminator857]

If you want to watch a video on AI accelerators:

Lots of fluff in the first 6 minutes. More technical around 12 minutes. 18 minutes discussion about flexibility versus hardware for specific purposes. 31:30 talks about Google TPU. 50 minutes accelerators are good for a subset of ML and deep learning.
0:00 - Intro
5:10 - What does it mean to make hardware for AI?
8:20 - Why were GPUs so successful?
16:25 - What is "dark silicon"?
20:00 - Beyond GPUs: How can we get even faster AI compute?
28:00 - A look at today's accelerator landscape
30:00 - Systolic Arrays and VLIW
35:30 - Reconfigurable dataflow hardware
40:50 - The failure of Wave Computing
42:30 - What is near-memory compute?
46:50 - Optical and Neuromorphic Computing
49:50 - Hardware as enabler and limiter
55:20 - Everything old is new again
1:00:00 - Where to go to dive deeper?

 

[Terminator857]

Analog computers are well suited for inferencing.

21:40 into the video.
Matrix multiplication can be done with analog circuitry.

 

[mspohr]

Anastasi has a lot of good AI information on her channel

 

[Terminator857]

Andrej impressed with Facebook research in object detection. (FAIR = Facebook A.I. Research)

https://twitter.com/x/status/1509524954911498240

Quote: Andrej Karpathy 7h
“Exploring Plain Vision Transformer Backbones for Object Detection” https://arxiv.org/abs/2203.16527 Excellent read as usual from the FAIR team. Strong object detection results with only minor tweaks on the vanilla (ViT) Transformer backbone.
Exploring Plain Vision Transformer Backbones for Object Detection
We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection...

Loving the philosophy of preserving simple Transformer as a Universal (Neural) Computer, where the core architecture is not meddled with much. Domain knowledge is “factored out”, only enters only through position encodings, sparsity masks, loss functions, data augmentations, etc.

In this paper the Mask RCNN is still bolted on for detection. Philosophically (and I’m guessing authors might agree and are curious) would be exciting to see a fully E2E approach win eventually, simply adding another decoder Transformer, directly outputting the boxes.

At that point we can just throw away a few decades of object detection research and it will be great Bitter sweet. Will happen to everyone.

6:36 AM

 

[Terminator857]

Defense department declares that A.I. is the mother of all technologies, when it comes to military operations.
The side that masters A.I. will have a very significant advantage over opponent. $15-20 billion dollars of investment in A.I. in defense budget. We have to leverage commercial efforts in A.I. for military application. Will be an enormous advantage in future combat. Service academies are teaching A.I.
55 minutes into the video.

 

[Terminator857]

Just pointing out the power of large language models in the use of coaching people via chatbot

1. Free coaching app that helps you to journal and track your daily leadership challenges - app - IDENTIFY GOALS WITH YOUR A.I. COACH
2. Home Page - Mobile Coach - Enterprise Chatbot Platform - app - Get customers onboard, teach, assist, follow-up
3. Design a chatbot coach using Engati chatbot template | Engati - Ever felt the need of having someone to talk to in the middle of the night? Someone who can listen to you, show concern and give you the support you need while you’re contemplating certain life situations? We all need somebody, but what if that somebody could be an advanced bot, well-versed with providing digital coaching and online assistance?
   An everyday routine of a coach, be it of a life coach, a wellness coach or an instructor, can be quite demanding. Whether you’re a life coach who helps people set and achieve personal goals or a leadership coach who works with high-level executives, your end goal remains the same. You want to attract more clients. A chatbot for coaching is great at getting that done; let’s show you how.
4. Coach M - Lever - Transfer of Learning - enterprises must embrace AI chatbots to streamline their HR & Learning processes. Creating behavioural change as part of learning is an age-old problem.
   It’s proven that following up learning with a coaching component will deliver far superior results than training alone. With time poor managers and limited resources to invest in outsourcing coaching solutions, what can you do?
5. 9 Coach Chatbot Templates - Create Coach bots for Facebook Messenger in minutes. No coding or technical skills required.
6. Alexa Skills, Chatbots & Conversational AI for Learning & Development - ALEXA SKILLS, CHATBOTS & CONVERSATIONAL AI FOR LEARNING & DEVELOPMENT
   EXPERTS FOR INTERACTIVE LEARNING
   Conversational interfaces lift E-Learning to the next level. Create better results through digital training partners that help:
7. Chatbot Coach | LEADx - Now you can access a live leadership expert in the palm of your hand!
   The LEADx Certified Leadership Coaching Team (who goes by the name “Coach Amanda”) can answer questions related to leadership, management, and productivity.
8. Can a bot be a coach? - At PocketConfidant AI, we have coaching expertise and experiences, we are entrepreneurs and we are building an innovative coaching technology using a chatbot.
9. Chatbots for Online Coaches: 10 Experts Share Their Best Tips - Create your own automated conversion funnel
10. Advantages of a Digital Coach Chatbot for Knee Surgery Recovery - MyComfortMD - Advantages of a Digital Coach Chatbot for Knee Surgery Recovery

 

[Terminator857]

The forum software changed the above http links into titles, so above sounds more like an advertisement than intended.

 

[Terminator857]

U.S. Department of Defense appoints Craig Martell, formally head of ML at Lyft, as Chief A.I. officer.

Lyft exec Craig Martell tapped as Pentagon's AI chief: Exclusive Interview - Breaking Defense

"I'm convinced we're going to be able to do some great things," Craig Martel told Breaking Defense in an exclusive interview. "But they're going to be hard things. It's going to be a challenge."

breakingdefense.com

DoD Announces Dr. Craig Martell as Chief Digital and Artificial Intelligence Officer

The Department of Defense announces Dr. Craig Martell as Chief Digital and Artificial Intelligence Officer. Dr. Martell brings extensive industry experience and expertise in artificial intelligence

www.defense.gov

https://www.linkedin.com/in/craighmartell/

If I were in Craig's shoes, I'd be tempted to declare the need for a Manhattan style project for A.I. Start at the research level. Create military schools for A.I. research. Start funding A.I. hardware research and development.

Craig will be less than totally successful if he underestimates the importance of A.I. Craig will be successful in my eyes, if constantly touts the need for huge focus on A.I. in the military and sets big moonshot goals. For example: ability for people in the field to improve ML models based on recent in the battle field experience(s).

Drone warfare is becoming increasingly the new "thing". Like airplanes during WW2. Easy to show the value of A.I. with avalanche of importance of current and upcoming drones.

 

[Terminator857]

Everything but the kitchen sink, do it all neural network:

[2205.06175] A Generalist Agent

arxiv.org

Google DeepMind

Artificial intelligence could be one of humanity’s most useful inventions. We research and build safe artificial intelligence systems. We're committed to solving intelligence, to advance science...

www.deepmind.com

https://arxiv.org/pdf/2205.06175.pdf
1.2 billion parameters. FSD 3 is 30 million? If we guess at every two years doubling of parameters for FSD hardware then it will take till about 2030 for FSD hardware to get to this point.

The numbers game here is just to give you an idea of what is going to happen. If / when general A.I. does happen, it will be on a trillion+ neural network. If you think FSD requires general A.I. then that gives you an idea of how far away full / not partial self driving is in the future.

 

[AKAlex]

I am stunned at the progress of AI that I've seen in the past year. It seems like we've reached some kind of tipping point. Dall-E 2 is a great illustration of this. Extrapolate out the capabilities seen in this video just a bit and the implications are stunning...

 

[Terminator857]

GATO generalist model explained.

The summary thoughts at 19:30 were especially good.