How machines learned to learn

Why it mattered

McCulloch and Pitts proved that networks of these switches could carry out logic. For the first time, thinking and arithmetic looked like the same kind of thing — which meant a machine might, in principle, do it.

The dials were physical

The Mark I Perceptron's weights were potentiometers turned by electric motors. You could stand in the room and watch a model learn. Everything since has been the same idea with the dials hidden inside silicon.

The cruel part

Stacking layers would have fixed it, and everyone knew that. Nobody knew how to train the stack — there was no way to work out which dial in a hidden layer deserved the blame for a wrong answer.

Backpropagation

It is not a learning algorithm. It is a bookkeeping trick for computing gradients efficiently. The learning is still just gradient descent — the same downhill walk you are about to do by hand.

Why this still matters

Modern systems increasingly do both — a learned model that also searches over its own possible answers. Knowing which half is doing the work is how you predict where a system will fail.

The uncomfortable lesson

The idea had been correct and ignored for twenty-five years, waiting for hardware. A lot of AI history is not new ideas. It is old ideas finally becoming affordable.

Why it had to change

There is a second problem hiding here. Reading strictly in order means you cannot start word ten until word nine is done — so it cannot be spread across thousands of chips at once. The design that replaced it could be, and that is as much a reason it won as accuracy was.

Still just next-token prediction

A large language model has one training objective: guess what comes next. Everything you have watched it do is a side effect of getting extremely good at that one task.