How to create a mind 1-3

So, Ray Kurzweil has just published a book called "How to create a mind". I've read the first three chapters so far. It's pretty interesting, but feels like its missing the details in much of the way Grossberg's work was.

He makes some interesting points about Einstein and Darwin and what their thought-processes were when they were making their big discoveries. Einstein is famous for his thought experiments and Kurzweil walked through his version of Einstein's thinking process. He makes the case that to understand the brain we have to make these types of thought experiments - thought experiments about thought.

It makes sense. There's definitely a lot of insight one could get by "introspecting". I mean it's a really interesting question: What information does my consciousness have access to? It's not everything (we have little access to the "Where" pathway, we see illusory contours, so we don't have access to the raw data.)

Kurzweil's big thing is talking about hierarchical pattern recognizers. His whole "unifying theory" is called PRTN - pattern recognition theory of neocortex. He goes into explaining hierarchies - ilustrating how certain patterns make up letters, letters make up words, words make up sentences and so on. He says in a way that language is crafted by the brain's hierarchical organization - our hierachical pattern recognizer lead us to creating a hiearchical pattern communication system. Oh yeah, he calls spoken language our "first invention", and he calls written language our "second invention"...

So far its pretty simple in AI circles - some "pattern recognizer", lets say a neuron, gets bottom-up inputs that have size (firing-rate), variance (spike-timing), and a weight (synapse strength). He even talks about dendrites and axons, but he does it in a strange way - either he's just too much into the AI field and not actual neuro, or he is maybe hinting at the idea that dendrites can each be a pattern recognizer (i'll probably find out later).

So the neuron recognizes its pattern and outputs its own pattern having size and variance information, which goes on to the next level. In the next level the same thing happens and so a hiearchy is arranged, and ultimately it could also be recursive. He talks about how top-down is coming back down constantly, and that its all feedback etc. Top-down is predicting bottom-up style.

So we'll see where he goes. It seems simple now, but its still just the beginning of the book - maybe 1/5th. What he has said so far seems just obviously true in such a vague sense, the real question is in the details.

I'm still really digging the resonance idea. One thing I've been thinking about recently is that a simple system that could do like an instant PCA on its data set, and then as more data get collected the best PCA dimensions are pulled out. Or it could be equivalently done for clustering.

I'm struggling with the question of how do the timescales change as we go up the hierarchy? What is happening at the top? So I can see how visual cortex is basically operating as an online system, processing and basically memorizing all of the inputs coming in constantly. But then as you go up the visual hierarchy, the neural representations seem to be for things that move more slowly and can be in a larger area. For the object recognition game, it definitely seems like spatial invariance is being done by simple and complex cells that stack up in a hierarchy. The complex cells are important to spatially invariant object recognition. I can see how you could build a hierarchical pattern recognizer with layers of complex and spatial cells to build to a spatially invariant object classifier. But at the same time that you go up in space, it also feels that you should go up in time. There's something to objects being both spatially and temporally invariant, and I think that this temporal invariance needs to be encoded.

Cortex is tasked with the job of simultaneously making a representation of its inputs, but also figuring out how to make the lowest-dimensional representation. Low-dimensionality is something to strive for, as it forces you to come up with simple explanations. But when there is no good answer you have to be high-dimensional in the description. So each layer in the hierarchy is a pattern classify (or maybe each column).

So a cortical column gets as inputs a bottom-up signal from the level-below, bottom-up like signals from the neighboring columns, and then top-down signals from higher-levels. A cortical column is then like a few thousand simple cells that are like layer 4, and complex cells that are like layer 2/3. The simple cells make the representation, and they are both forced to be more excitable (hence increasing dimensionality) when they are bad, and making the lowest they can make when they are starting to work. The complex cells use the simple cells to pattern complete, making a spatial invariant pattern of the simple cell representation.

So it just seems that one thing that would help shaping the feedback signals is to look for temporal invariance. I think back to a paper I remember about how the brain learns slowness - things that are temporally invariant. It would just be a useful signal to learn from - if your higher-level is changing less then your representation is good...