SAILnet

I was at OCNS this weekend. There was one poster that I though was pretty interesting about sparse coding. It hasn't been published, yet, but it was really just an extension of this paper:

Zylberberg, J. Murphy, JT. DeWeese, MR. (2011) A Sparse Coding Model with Synaptically Local Plasticity and Spiking Neurons Can Account for the Diverse Shapes of V1 Simple Cell Receptive Fields. PLoS Computational Biology 7(10): e1002250.

The main update that they had made was that they seperated the excitatory and inhibitory populations, in this paper their neurons are allowed to inhibit and excite each other. There were some interesting ideas for learning rules between E->I, I->E, and I->I synapses, which could be of use later.

Notes:
The literature has shown that V1 is representing visual space through these gabor-like receptive fields of the principal neurons. I'll have to do some further reviews. Oshausen and Fields showed that by minimizing the neural activity and maintaining a full representation of the input, you could learn similar receptive fields from natural images. This is sparse coding - minimal representation of the input. There's been a ton of other work trying to reproduce V1 receptive fields, another paper to review later is one where they show that ICA (indepentent components analysis) of images also generates similar looking receptive fields.

This paper's primary contribution is to translate the sparse coding algorithms into rules for spiking neural networks that have local plasticity. Local plasticity means that there are no global learning rules - like you can't normalize all of your synapses based on the synaptic weights elsewhere in the population. A lot of computational models fail to be realistic because they use these non-local learning rules, which are physically impossible for a brain to be implementing.

Leaky integrate and fire. Foldiak learning rule - units are active for a small but non-zeros fraction of time and maintain uncorrelated activity with respect to other units:

Where W is the inhibitory weight between neurons, Q is the excitatory weights coming from the inputs, theta is the threshold of each neuron, p is the target firing rate for each neuron, and alpha, beta, and gamma are learning rates.

Here are the receptive fields that it learns, it looks pretty good compared to the standard receptive fields:

So, sparsity is interesting, and the poster I saw was basically a slight modification of those learning rules to seperate out excitatory and  inhibitory populations, but the ideas were about the same.

Figure 2 of this paper shows a reconstructed stimulus from the activity of the neurons in the network. It does a fairly good job, but the idea is that the network has come up with a complete representation. This means that it should be able to reconstruct any stimulus with the features that the network has learned. The problem/extension of models like this is that there is nothing that pushes the network to have a fully complete representation. All the neurons could learn roughly independent features, but there is nothing that is forcing the network to build a complete feature set. A new way of designing these networks such that they do build a full feature-set would be a nice step forward.