Computational Linguistics Reading Group

A key feature of natural language is its hierarchical compositional semantics: the meaning of larger wholes depends not only on its words and sub phrases, but also on the way they are combined. To investigate how neural networks can learn and process such compositionality, we define the artificial task of processing spelled out nested arithmetic expressions, and study whether LSTM and GRU networks can learn to compute their meanings. We then use a technique called diagnostic classification, in which a simple neural meta-model is trained to quantatively evaluate hypotheses about the information that is represented/encoded in the hidden state of the trained networks. Using this method, we analyse the hidden layer activations, but also the gates of trained networks. We argue that diagnostic classification, unlike most visualisation techniques, does scale up from small networks in a toy domain, to larger and deeper recurrent networks dealing with real-life data, and may therefore contribute to a better understanding of the internal dynamics of current state-of-the-art models in natural language processing.