As I said: they need a breakthrough. One that prevents hallucination and possibly looks very very different in structure and behavior than what is used for LLMs and the generative AI of today. This will requires billions of continued investment with unknown prospects of success. It seems extremely unlikely to “just” require “a bit more training.”
Yes this is happening now! I did not suggest it was not.
I don't disagree, in a very general sense, that breakthrough(s) are needed. But this seems hardly insightful, as it's almost by definition how any nascent technology progresses. Not a linear or predictable smooth function of measurable progress, until the core technology is well understood and modeled.
I might point to Moore's Law as one of the most famous examples of progress prediction in a (now) relatively mature discipline. The end of the line for Moore's Law continues to be pushed out by mini-breakthroughs which are individually not easy to predict, but in aggregate the curve continues to be followed with perhaps a minor diminishing of the growth exponent. But, there were some decades of bumpy and less certain electronic and computing development, before the overall paradigm of shrinking feature size and formalized design processes could become predictable (and credit Gordon Moore at al for the insight to capture it in a macro-level prediction).
I don't think we're at that point yet with ML, more specifically self-driving and more generally AI. You and others are constantly reminding everyone of how seemingly slow and disappointing the progress is, but from a historical perspective that might seem to be unfounded heckling from the peanut gallery. Perhaps kind of a spoiled reaction to what is actually an amazing (and somewhat unsettling) dawn of an entirely new and disruptive technology - probably
at least as significant as the last century's world-changing developments in transportation, telecommunications, computing and the internet.
Regarding "billions of dollars needed", well yes but that can go in so many different directions. Raising (or printing) billions of dollars to throw into a pot stirred by councrls of "experts" it's probably not the right starting point (and I know you didn't say that). Billions of investment dollars is already happening; fortunately not yet highly captured, directed and grifted by governments and high councils.
I don't think either of us will be surprised if, in 20 years, we can look back and see some dead ends, some correct but initially failed approaches, and some conclusions that will seem easy and obvious later, but not readily apparent today. That's just how it goes with new technologies and industries. Billions of dollars maybe spent today in searching for and identifying the right techniques; it is after that point that further billions will be directed towards industrial scaling and execution of the increasingly predictable implementations. Maybe some of that will be like today's training on supercomputers, and maybe some of it will be something quite different.
