Beyond Transformers

Assessing Next-Generation Neural Architectures for Multimodal Sequence Modeling

Abstract

The Transformer has been the dominant architecture for sequence modelling for almost a decade, but its self-attention mechanism compares every position with every other, so its cost grows rapidly with sequence length and its inference memory grows with the length of the history. These costs are especially burdensome in multimodal settings, where text, audio, video, and sensor streams are combined into very long sequences. A new generation of architectures, including structured state space models, linear and retentive attention, long convolutions, modernised recurrent networks, and hybrids that mix several of these with a few attention layers, claims to remove the bottleneck. This paper provides a systematic assessment of these families under a controlled protocol. Six architectures and two hybrid designs were trained with matched parameter budgets and identical data on five multimodal tasks: video-text retrieval, audio-visual speech recognition, long-document question answering with figures, wearable sensor and text fusion, and long-horizon multimodal.......