Design and Analysis of Neural Networks Using Information

Kim, Hyun Jin

doi:https://doi.org/10.21985/n2-sze5-0m61

Work

Design and Analysis of Neural Networks Using Information

Public

Download PDF

Download All Files (.zip)

We present two ways in which measures of information can be used for the design and analysis of neural networks in both the brain and the computer. In the brain, stimulus is often represented as a distributed pattern of activity in a network of neurons. The quality of such population codes can be measured by the amount of information about the original stimulus that can be decoded from them. In this context, Fisher information has been widely used to quantify coding error and specify optimal codes. However, when data is sparse, there is a large discrepancy between Fisher information based predictions and the actual error. We show that this discrepancy is caused by non-local errors not accounted for by Fisher information. We use this insight to design optimal codes and study the resulting scaling properties. We then show that analogous considerations also apply to networks storing the memory of a stimulus through continuous attractor dynamics. Counterpart to designing networks is analyzing how a given network behaves. A technique in neuroscience called maximally informative dimensions can be used to identify aspects of the input stimulus that most affects the neural response. However, it suffers from scalability issues of the optimization, which limits its applications. Here, we present an efficient algorithm for extracting maximally informative dimensions based on recently developed methods that use deep learning to estimate mutual information. We then demonstrate various ways in which maximally informative dimensions can be used to analyze artificial neural networks, as well as other predictive machine learning models more generally.

Creator