This exciting initiative is directed at the wider Tübingen AI community and begins with a thought-provoking session by renowned researcher, Antonio Orvieto. Antonio has just joined the Core Faculty of the Tübingen AI Center as head of an Independent Research Group with close ties to the Max Planck Institute for Intelligent Systems and the new ELLIS Institute Tübingen.
Details of the Talk:
- Date: Thursday, 19 October 2023
- Time: 11:00 - 12:00
- Location: Ground-floor lecture hall, Tübingen AI Center, Maria-von-Linden-Str. 6, 72076 Tübingen
Title: Accurate and Efficient Processing of Long Sequences and Large Graphs without Attention
Abstract: When applied to sequential data, transformers have an inherent challenge: their attention mechanism leads to quadratic complexity with respect to sequence length. This issue extends to graph transformers, where complexity scales quadratically with the number of nodes in the network. Today, we’ll explore theoretically grounded alternatives to the attention mechanism that hinge on carefully parametrised linear recurrent neural networks. Unlike the more commonly known LSTMs and GRUs, linear RNNs are particularly GPU-efficient. This efficiency enables us to scale up the architecture, successfully study signal propagation, and achieve competitive performance. We’ll present how, with a Linear Recurrent Unit (LRU) replacing attention, we can achieve state-of-the-art results on sequence modeling and graph data. This approach offers a promising direction for future research, especially in genetics, protein structure prediction, and audio/video processing and generation.
We invite all enthusiasts, researchers, and students to join us and dive deep into the realms of AI. Don't miss this great opportunity to kickstart your learning journey with the Tübingen AI talk series. See you there!