h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Not a big expert in Neural Nets, but is the gradient decent training algorithm the same as the one used for Genetic Algorithms? If so, can it get stuck in locally optimal solutions and then fail to improve?
For example, if you picture your solution space as a wiggly line, like a ridge of mountains, with bad solutions being at the tops of the mountains and good ones being in the valleys, then the optimal solution is at the bottom of the deepest valley.
However, standard genetic algorithm techniques work by deciding whether the latest answer was better or worse than the previous one, like a walker that can only walk downhill. If you then end up in a valley which isn't the deepest one, there's no way to get out, as all the solutions you can reach on your next evaluation from where you are, are worse than the one you currently have

On another point, Neural Nets have been used to construct physical circuits too. Using a clever gizmo[1] which can create electrical circuits in a dynamic way, boffins[2] have managed to evolve circuits for detecting differences between a high and a low audio tone, starting from a few basic components.
Interestingly, when they looked at the circuits the system provided they shouldn't have worked. It seems the neural net was able to use latent characteristics of the components in the system in undocumented ways to reach a solution.

First case of intuition in a computer perhaps?


[1] Technical term meaning, "I've no idea what it was."
[1] Technical speak for "and I can't remember who they were either"

Hello,

Yes, it's always a risk for any gradient-climbing algorithm, but I think if you use one that isn't the most greedy, you're usually okay. A lot of modern algorithms provide for a little bit of wiggle I think and don't leap towards a solution as quickly just to avoid getting trapped local maxima.

What's interesting to me is how the new technology of neural nets, genetic algorithms, and support vector machines all do rather intelligent things but without the standard notions of mental rules. They basically just form intelligence out of statistical analysis, meaning it's pretty hard to analyze later what "rules" the system is following.

Yours,
Jake

Trouble is, people like to know how things work, so they can check to see if they're correct or not.

In reality, I admit, this is a purely psychological notion. Very few computer programs are formally analysed anyway (which is why we have bugs!) so there's no real difference, but at least with a computer program you can, normally, identify why it went wrong.

Neural nets provide no sense of why. This also limits their usefulness for decision-making tasks. I quite often want to know why a system reached an answer, not just what the answer is.
Be interesting to see if a neural net could be produced which could analyse its own decision making process. Would that then be a conscious machine?

Hello,

It's true, you'll never get rules per se. The most you can really do is look at the weights and perhaps intuit what aspects of the input were more important to how it determines the solution. This is also true of genetic algorithms and SVMs both of which involve some weight adjustment really. Neural nets and SVMs both use statistical methods, while genetic algorithms breed solutions. And it's true, we like to analyze the solutions, but it's hard.

The other problem with neural nets is that they can become mathematically intractable. Usually, you have to select the most relevant features of a problem because otherwise you'd have too many nodes to find a solution in a reasonable time. This could potentially be bad since you're then discarding some aspects of the input that could be useful to the solution (like deciding that black and white is a perfectly valid substitute for color vision). So philosophically it's a bit scary, but it usually works out rather well regardless.

Yours,
Jake

The researchers are Adrian Thompson and Phil Husbands, and the gizmo is an FPGA: basically a "big" (in terms of number of components) circuit board where you can reprogram what connections are made in what order bettween components.
For full story see... oh, yeah, no URLs! Go to NewScientist (a dot.com), click "hot topics", and then "AI&A-Life". Then add "primordial.jsp" to the end of the address to jump to the article in question. (Look moderator: no URL!)

[Then join the Zaphodistas (http://www.h2g2.com/A520769) so we can start being more helpful in forum posts!]