Efficient Learning with Spiking Neural Networks

Artificial neural networks constitute a vast field which has reached a level of maturity in some areas. However, a big problem with state-of-the-art methods is the impractical amount of compute power they require when scaled. Spiking neural networks are third generation neural networks with a higher level of biological realism, which typically comes at the cost of even greater computation. Along with their complexity, this makes them impractical for general machine learning. However it may be possible for spiking networks to be more efficient than earlier networks for some problems. This is because they have the capacity to exploit some of the brain's own efficiency techniques.This thesis explores those techniques in the literature and contrasts them with digital computation. Experiments reproducing some of them in artificial neural networks are then presented. Efficiency of computation is measured using cputime, which is related to how fast an algorithm runs as well as how much energy it consumes. The computational costs of spiking learning are quantified for the first time using the supervised learning algorithm SpikeProp. This was found to be between one and two orders of magnitude slower than the multilayer perceptron, and even worse with a receptive field encoding.Efficiency enhancements to SpikeProp are then presented, including reduced precision, fewer synapses, lookup tables and event-driven computation. The computation required by the enhanced algorithm SpikeProp+ to learn classifiers is again compared to Backprop on the multilayer perceptron, using several standard datasets. The compute time required to train the improved networks was reduced by up to seventeen times, surpassing Backprop on the XOR problem. In that case, the SpikeProp+ network runs in the same time as the perceptron, although it remains over two times slower on other datasets.Finally, in applying SpikeProp+ to a stance-detection application, it was discovered that the spiking algorithm is intrinsically able to learn from data containing many missing values to a high level of accuracy, without any pre-processing. This property was verified with a standard dataset, and is relatively unique amongst machine learning techniques.Thus a case for using spiking neurons as a substitute in more conventional neural network architectures is presented, to both accelerate them and permit the use of lossy data. These properties are desirable in view of the vast sizes and variable quality of modern data.