Inspecting and Directing Neural Language Models

Thanapon Noraset

doi:https://doi.org/10.21985/N29200

Work

Inspecting and Directing Neural Language Models

Public Deposited

Download PDF

Download All Files (.zip)

The ability of a machine to synthesize textual output in a form of human language is a long-standing goal in a field of artificial intelligence and has wide-range of applications such as spell correction, speech recognition, machine translation, abstractive summarization, etc. The statistical approach to enable such ability mainly involves defining representations of textual inputs and computation of likelihood of the outputs text sequence. With the recent advancement in a neural network or deep learning research, machine learning models can construct general vector representations of words, also known as word embeddings, that are useful for many natural language processing tasks. Furthermore, neural language models can accurately assign a probability to a sequence of text, and become a default choice of researchers and developers. One of the key advantages of these deep learning models is that they require little to none human heuristics in feature engineering, and instead, are optimized using a large amount of data. Despite the performance improvement and convenience of the neural language models, they lack a useful property found in the previous statistical model such as an n-gram language model -- the parameters of neural language models are not directly interpretable. This leads to a set of challenges when using such models. For instance, the word embeddings do not directly convey what information is captured, hence which semantic gaps exist in the embeddings is uncertain. Furthermore, the likelihoods of a sequence of text are defined by a series of non-linear functions, making the behavior of the models, even for a short phrase, difficult to ascertain or change to the desired direction. This dissertation addresses these two shortcomings by exploiting the models' ability to generate interpretable text. First, we propose a more transparent view of the information captured by a word embedding, and introduce the Definition Modeling, the task of generating a definition for a given word and its embedding. Second, we study an explicit approach to adjust the model's behavior, and present Dynamic KL Regularization, a method for training neural language models to follow a given set of statistical constraints. Finally, we explore efficient solutions to a fundamental, yet lacking the capability of the start-of-the-art neural language model: accurately computing a likelihood of a short phrase that it will produce.

Last modified