Machines Like Us

Geoff Hinton has a new TiCS paper describing recent advances in algorithms used to train multilayered neural networks.

First, a little background: neural networks of a sufficient size can calculate any mathematical function (an infamous proof among neural network modelers). Unfortunately, the tricky part is figuring out how to set the connections in that network to calculate those functions.

This is where learning algorithms become necessary -- unless you want to tweak each connection by hand until you get a working network (not a problem if you don't care how the brain works), then you need to focus on how the network can learn.

Hebbian learning is a standard algorithm that does seem to operate in biological neural networks, but it has a problem: it's not very good for training deep networks (those networks which have multiple "hidden layers," i.e., networks where only a small portion of the units receive input from outside the network). In the 1980s, a new learning algorithm was developed which could overcome these limitations -- known as backpropagation of error, or just "backprop."

Read full story in Developing Intelligence