Jurgen Schmidhuber on Humans co-existing with AIs

Linear Transformers

The origins of linear transformers reveal their foundational role in AI's evolution. Jürgen Schmidhuber introduced the concept in 1991, emphasizing its efficiency compared to modern quadratic transformers. He explains that the linear transformer separates storage and control, akin to traditional computers, allowing for more efficient computation 1. This architecture laid the groundwork for future innovations in AI, including the development of keys and values in neural networks 2.

The linear transformer of 1991 does this to minimize its error. It learns to generate patterns that in modern transformer terminology are called keys and values.

--- Jürgen Schmidhuber

Schmidhuber's early work continues to influence AI research, highlighting the importance of efficient computation in neural networks.

   

AI Credit Issues

Controversies in AI credit assignment highlight significant historical contributions often overlooked. Jürgen Schmidhuber criticizes prominent figures like Yann LeCun and Geoffrey Hinton for not acknowledging earlier work by pioneers such as Ivaknenko and Lapa from Ukraine 3. He argues that many foundational AI techniques were developed decades before being popularized by these figures. Schmidhuber emphasizes the role of Japanese researchers in CNN development, noting their early contributions to convolutional neural networks 4.

Their most famous work is completely based on work by others whom they did not cite.

--- Jürgen Schmidhuber

These disputes underscore the need for accurate historical recognition in AI's rapidly evolving field.

   

Generative Curiosity

Artificial curiosity and generative adversarial networks (GANs) play pivotal roles in AI's development. Schmidhuber's concept of artificial curiosity involves robots exploring environments, with prediction machines minimizing errors and controllers generating actions 5. This approach has influenced modern GANs, where networks learn through adversarial processes. Additionally, predictive coding aids in efficiently encoding actions and observations, creating compact representations and self-symbols 6.

The self awareness is just a natural byproduct of the data compression process of the world model.

--- Jürgen Schmidhuber

These innovations demonstrate AI's potential to autonomously learn and adapt, paving the way for more advanced systems.