AI & ML
Clustering in pure-attention hardmax transformers and its role in sentiment analysis
arXiv stat.ML0 views
arXiv:2407.01602v3 Announce Type: replace-cross Abstract: Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called \textit{leaders}. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.
Source:https://arxiv.org/abs/2407.01602Related posts
TacBridge: Learning Unified Tactile Representations through Cross-Sensor Correspondence
0arXiv cs.RO
Transferable Neural CBFs via Generalized Closure
0arXiv cs.RO
QTOP: Scaling Humanoid Motion Tracking with Quantile Critics
0arXiv cs.RO
Non-Imaging Optical Tactile Sensors for Robotics: Transduction Principles, System Integration, and Robotic Applications
0arXiv cs.RO
Robotic Reward Models in the Foundation-Model Era
0arXiv cs.RO
Selective Sensor Acquisition for Multimodal Place Recognition
0arXiv cs.RO