Chapter 8: With a Little Help from Physics

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement, not replace, the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

You know that feeling, right?

A familiar tune, maybe a certain smell, and suddenly your brain just floods with old memories.

Oh, definitely.

That whole experience.

That's associative memory.

Our brain's pulling up these entire experiences from like just a tiny fragment.

And it really sparks this fundamental question.

Could we actually engineer a system that does something similar, you know, build an artificial memory that works like ours?

Well, that exact question really grabbed John Hopfield.

He was a brilliant physicist, you know, trained in solid state physics.

But then in the late 70s, he shifted his focus.

He got interested in this new frontier,

how living systems, well, compute.

His first step was into biology, looking at these seemingly messy biochemical reactions in cells.

Like building proteins.

Yeah.

Fundamental.

Exactly.

Things like protein synthesis.

And his physicist's perspective was fascinating.

He looked at these multiple pathways in biology.

Which most people might see as inefficient or error prone.

Right.

But he saw them as crucial for accuracy, a kind of built -in error correction, biological proofreading, he called it.

In a really key paper back in 74, he basically showed that networks of reactions can achieve things, functions, that individual molecules just can't.

It was the network, the system as a whole, that was solving a problem.

So this idea of emergent problem solving from interconnected bits in biology,

how did that steer him towards AI?

Well, that biological insight was really the seed.

You know, Hopfield started looking for a big problem in neuroscience, something he could tackle with the tools of theoretical physics.

He was actively searching for a specific challenge.

Yeah.

He immersed himself in it, went to meetings, looked at different research at MIT, but he wanted something more unifying, a fundamental principle for how the brain actually computes.

And associated memory was the problem he landed on, that cue triggering a whole memory thing.

Exactly that.

He saw that retrieving a complete memory from just a fragment, that act itself was a computational challenge.

He started thinking, could a network of artificial neurons with memories somehow stored in their connections?

Could it pull back the full memory from just a piece?

Precisely.

He framed it as a dynamical system, something that evolves over time and hopefully settles on a solution, the complete memory.

Which, kind of surprisingly, connects to the physics of magnets, right?

Ferromagnetism.

Yeah, that seems like a jump.

But to understand the principles, Hopfield looked at ferromagnetism and this simplified math model called the Ising model.

OK.

What was amazing was the parallel he saw, how magnetic materials organize themselves and how a network of neurons might settle into a stable memory state.

The abstract math was the bridge.

Magnets and brains.

Still feels like a stretch.

I know, but think about it like this.

Window glass, right?

Amorphous solid.

No real ordered structure.

Then you've got a ferromagnet, like on your fridge, little magnetic bits.

The moments are all lined up.

It's ordered, creates a field.

Got it.

But then there are materials where those little magnets are just randomly pointing all over the place.

No overall magnetism.

They call those spin glasses.

Spin glasses.

OK.

And they're disordered, like the glass.

Kind of, yeah.

They share that disordered complexity.

And this is where the math comes in.

The Ising model.

Right.

Developed back in the early 1920s.

Wilhelm Lenz and his student Ernst Ising, a simplified model for these magnetic materials.

And Ising looked at it in just one dimension first.

Like a chain.

Yeah.

For his PhD, a 1D chain of these tiny magnetic moments spins.

They could point up, call it plus one or down, minus one one.

And the key idea was each spin only interacts directly with its immediate neighbors.

So like dominoes, one flips, it might cause its neighbors to flip.

Exactly like that.

It creates this dynamic in the system.

But interestingly, Ising's math showed that in 1D, the system would never actually become ferromagnetic.

All the spins wouldn't just line up.

And he thought that applied to 3D too.

He did, mistakenly, yeah.

It took Rudolf Peierls in 1936.

He studied the 2D model rigorously spins on a grid, not just a line.

Okay, a grid makes more sense for a material.

Right.

And Peierls proved that at low enough temperatures, a 2D system would become ferromagnetic.

And then he said, elegantly, it holds a fortiori for 3D.

Meaning the case is even stronger in three dimensions.

A fortiori.

Yeah.

So picture this grid spins up or down.

Each feeling its neighbors.

Exactly.

A 2D grid of arrows, up, down.

Each one, unless it's on the edge, has four newest neighbors.

And what makes them point one way or another?

An external magnet, maybe?

That's one factor, yeah, an external magnetic field H.

The other is the internal field from its neighbors.

And that depends on the interaction strength J between them.

And whether it's ferromagnetic.

Where they like to align.

Or anti -ferromagnetic, where they like to point opposite.

Let's stick with ferromagnetic for now.

Okay, so how does a random jumble of spins end up aligned?

Ah, that's where we talk about energy, the Hamiltonian.

It's just a mathematical equation to calculate the total energy of the whole grid of spins.

The Hamiltonian.

Right.

For the 2D Ising model, one part sums over adjacent pairs of spins.

You multiply their values, plus one or negative one, multiply by that interaction strength J.

Do that for every pair.

There's also a term for the external field H, interacting with each spin.

But the really crucial part is the negative sign in front of these terms.

Why is that so important?

Because if two neighbors are aligned, say plus one and plus one, their product is positive.

The minus sign makes it a negative contribution to the energy.

It lowers the total energy.

And if they're opposite, plus one and negative one.

Their product is negative.

The minus sign in the Hamiltonian makes that term positive.

It raises the energy.

So aligned spins equals lower energy.

Opposing spins equals higher energy.

Exactly.

Which strongly suggests that the state where all spins are aligned must be a minimum energy state.

Like a ball rolling down into the bottom of a bowl.

Gradient descent you mentioned.

Precisely that idea.

Physical systems tend towards their lowest energy configurations.

And whether it's ferromagnetic J positive, anti -ferromagnetic J negative, or spin glass J's random between pairs,

it's all about energy.

And Hupfield knew about spin glasses from his physics background.

Oh yes.

He was very familiar with them.

And he saw the connection.

He realized the Ising model's math could describe a network of artificial neurons.

Storing and retrieving a memory could be like the network settling into a stable, low energy state.

Determined by the connections.

The weights between neurons.

Exactly.

The memory becomes a stable configuration.

Like the aligned spins.

So it's not like a file stored somewhere.

It's this stable pattern of activity the network naturally falls into.

That's the core idea.

You give it a partial or corrupted memory that pushes the system to a higher energy state.

But the network's dynamics, how the neurons influence each other, should guide it back down to the low energy state, the original memory.

Retrieval is literally finding that energy minima.

That's the process.

And the building block is the artificial neuron itself.

Which, you know, had its own history.

That Minsky and Paper book Perceptrons kind of put a damper on things for a while.

It really did.

Their 1969 book was hugely influential.

They proved mathematically that single layer perceptrons were limited.

They could only solve problems that were linearly separable.

Meaning you could draw a straight line to separate the categories.

Basically, yes.

And worse, they speculated quite strongly that training multilayer networks would be computationally infeasible.

Maybe even impossible.

Wow, that must have been discouraging.

Hugely.

Minsky's view basically suggested multilayer networks weren't fundamentally better.

It really stalled mainstream research.

Though some folks in the 70s kept plugging away quietly, working on early ideas for training them, the seeds of backpropagation.

But computing power was a bottleneck then too.

Definitely.

So it's into this environment that Hopfield arrives, looking for his next big research question.

And what did his artificial neuron look like?

Was it complex?

Actually, quite simplified.

Inspired by earlier models like McCulloch -Pitts.

Imagine a neuron, maybe just two inputs, by one and by two.

Crucially, in Hopfield's setup, these inputs are bipolar.

Only plus one or mega one.

Okay, bipolar inputs.

Each input gets multiplied by a weight, W one and W two.

Then you just sum them up.

W one by one plus W two by two.

And the output depends on that sum.

Yep.

If the sum is positive, the neuron outputs plus one.

If it's zero or negative, it outputs mega to one.

Simple threshold logic.

We can ignore the bias term.

For now, it doesn't change the main idea here.

Okay, simple neuron.

But the network is the key.

How did he connect them?

He imagined networks where neurons are connected bidirectionally.

If A sends its output to B, then B also sends its output back to A.

Ah, mutual influence.

Exactly.

Think of a tiny two -neuron network.

Neuron one's output, Y one, goes to neuron two, multiplied by weight W 21.

Neuron two's output, Y two, goes back to neuron one, times weight W 12.

And importantly, no self -connections.

A neuron doesn't feed back into itself directly.

So they're constantly affecting each other's state.

Constantly.

The output of each neuron depends on this weighted sum of outputs from all the other neurons it's connected to.

Scale it up to three neurons, it gets more complex.

Neuron one gets W 12, Y two, plus W 21, Y three.

Neuron two gets W 21, Y one, plus W 23, Y three, and so on.

You can write a general formula for that, I imagine?

Yeah, there's a neat summation formula for any number of neurons.

And this interconnectedness really strengthens the Ising model analogy, doesn't it?

It does.

The random initial states, neurons firing plus one or making one, that's like the disordered spins.

Right, and each neuron listens to the others, does its weighted sum, and flips its output if the sign changes.

It's really similar to how a spin reacts to its neighbor's magnetic fields.

And the weights are like the interaction strengths in the magnet.

Perfect analogy.

Now Hopfield first looked at networks where the weights weren't symmetric.

So W 12 could be different from W 21.

He even defined an energy function for these, trying to analyze their behavior.

But asymmetric weights didn't quite work, not stable.

Exactly, they wouldn't reliably settle down.

And then came the big insight.

What if the weights were symmetric?

What if we always equaled widgey?

Symmetry was the magic ingredient.

Absolutely.

Hopfield saw immediately that symmetric connections would mathematically guarantee stable points.

That was the critical link to associative memory.

Okay, so symmetric weights guarantee stability.

But how do you set those weights to actually store a specific memory?

Right, so imagine you want the network to remember a specific pattern of plus ones and minus ones across all neurons.

That's your target memory.

Okay.

The goal is to set the symmetric weights so that this exact pattern becomes a stable, low energy state for the network.

That pattern is the stored memory.

So if you give it a messed up version.

You give it a messed up version, maybe an incomplete pattern.

That puts the network in a higher energy state.

But because the weights are set up to favor the original pattern, the network dynamics just pull it back down towards that stable, low energy state.

It retrieves the memory.

It's like the network is predisposed to fall back into the patterns of nodes.

Exactly, and that symmetry is what makes it work reliably.

Okay, okay, this is starting to click.

So how do we set those weights?

What's the actual method?

Good question.

This brings us to the core of understanding Hopfield networks.

Like what does storing a memory even mean here?

It means making a specific pattern of neuron output stable.

Right, and stable means?

Stable means if the network is in that state, it stays in that state.

No neuron wants to flip its output.

Okay, and how do we pick the weights to achieve that?

And how does energy fit in?

Let's use a simple three -neuron example.

We need a weight matrix, W.

It's three by three.

The entry way is the weight from neuron J to neuron I.

And it's symmetric, way goes Ouija, and the diagonal is zero.

No self loops.

Exactly.

Now, say we want to store the pattern necklace one, one necklace one.

So, neuron one should be negates one, neuron two plus one, neuron three to one in the stable state.

How do we set W12, W13, W23, et cetera?

This is where Hebbian learning comes in.

Neurons that fire together, wire together.

Precisely.

The Hebbian rule gives us a simple way.

For Hopfield networks, the rule is the weight wedge, which equals Ouija, is just the product of the desired outputs, ye and uj in the memory pattern.

So if ye and uj are the same, both plus one or both make us one, their product is plus one, the weight is plus one.

Right, they reinforce each other.

And if ye and uj are different, plus one or minus one, the product is minus one, the weight is minus one, they inhibit each other.

You got it.

So for our necklace one, one necklace one example.

Let's see, W12 is a necklace one, one, equals one, one.

Right, and W21 is also minus one by symmetry.

W13 is minus one, minus one equals one.

Good, and W31 is also plus one.

W23 is one, minus one, one.

Perfect, and W32 is minus one.

The diagonals, W11, W22, W3 are zero.

That defines our weight matrix for this one memory.

Okay, and you mentioned doing this with matrix math too.

Yeah, it's often cleaner.

If you write the memory pattern as a column vector, y, one, os, one, one, nine, os, one, transpose.

Transpose, right.

Then the weight matrix W is the outer product of y with itself, so y times y transpose, and then you subtract the identity matrix I.

Yyti, y subtract the identity matrix.

That just zeros out the diagonal elements automatically.

The outer product yyt gives you a matrix where element ij is ye, i, yj, which is exactly the Hebbian rule.

Clever.

Okay, so we've set the weights using Hebb's rule.

How do we know that pattern is now stable?

Why won't the neurons flip?

Let's check.

Take any neuron, say neuron i.

Its new output depends on the weighted sum from all other neurons j.

So the sum over j, not equal to i, of we, e, i, j.

Now substitute the Hebbian rule, we, e, sup, e, ne.

The sum becomes sum over j of e, i, y, j.

Which is sum over j of yi, yj squared.

Exactly, and since yj is always plus one or nummy one, what's yj squared?

Always one.

Right.

So the sum simplifies to sum over j of yi, which is just yi times the number of other neurons.

The crucial point is the sign of the sum.

It's always the same as the sign of u itself.

Ah, so the neuron calculates its input and the sign matches its current output, so it doesn't flip.

Precisely.

The pattern stored using Hebbian weights is inherently stable.

And this stable state is an energy minimum.

Yes, that stored pattern corresponds to a local minimum in the network's energy landscape.

The energy function is e off of a sum over i, sum over j of yi, yi, j.

If you nudge the network away from that stored pattern, say by flipping one neuron, the energy goes up.

The network dynamics then naturally push it back down towards lower energy states.

And any flip decreases the energy.

In a symmetric Hopfield network, yes.

Any neuron flip that actually happens must decrease the overall energy or keep it the same if it's already at a minimum.

So the network just keeps flipping neurons until it can't lower the energy anymore.

It gets stuck in a local minimum.

It rolls downhill into the nearest valley and the valleys are the stored memories.

That's a great way to visualize it.

Each stable state is a valley.

You mentioned storing images like handwritten digits, 28 by 28 pixels.

Yeah, imagine that.

784 pixels.

We treat each pixel as a neuron output, plus one for white, next one for black maybe.

So you need a network of 784 neurons to store one image represented by that 784 element vector Y.

You calculate the huge 784 by 784 weight matrix, W equals YYTI.

That one calculation encodes the image as a stable state.

You can store multiple images in the same network.

You can.

A common way is just to add up the individual weight matrices for each image you wanna store.

If you have images Y1, Y2, RN, the total weight matrix W is the sum over K of YKYKT minus N times the identity matrix.

Just sum them up.

Does it always work perfectly?

Well, there's a limit.

Hopfield found the capacity is roughly 0 .14 times N memories for N neurons.

So for 784 neurons, maybe around 100 memories, give or take.

And ideally, the memories should be fairly different from each other mathematically, close to orthogonal, otherwise they interfere.

Modern Hopfield networks have pushed this capacity way higher though.

So retrieval.

You feed it a noisy picture, say a blurry five four.

How does it clean it up?

You set the initial neuron states to match the noisy image pixels, plus ones and X ones.

Then you run an update loop.

What does that involve?

A common way is pick a random neuron.

Calculate its weighted input sum from all other neurons based on their current states.

Determine the neuron's new output plus one or recus one based on the sign of that sum.

If it's different from its current output, flip it.

And you just keep doing that.

Randomly picking neurons and updating.

Keep doing it.

Capulate the network energy sometimes to track progress.

Eventually the network should settle down.

Neurons stop flipping.

It's reached a stable state, a local energy minimum.

Hopefully that state matches the clean five.

And you showed examples where that worked incredibly well, even from really messy inputs.

Yeah, it can be quite robust.

And even starting from total random noise, the network will often just fall into one of the stored memory valleys.

It dynamically finds the patterns.

But sometimes weird things happen.

Like getting the negative image back.

Ah yes, that happens because if a pattern Y is an energy minimum,

it's exact opposite.

All plus ones become negative ones.

All negative ones become plus ones, often as two.

With the same energy level.

So depending on where your noisy input starts, you might roll into the Yer Valley.

Or the Vier Valley.

Interesting.

And it could even retrieve the wrong memory entirely.

Like you put in a noisy eight and get back a five.

That can happen too, yeah.

If the noisy eight is actually closer in that energy landscape sense to the valley representing the stored five, the dynamics might just pull it towards the five minimum instead.

It's really amazing that this physics inspired model shows these complex memory behaviors.

It really is.

Hopfield's 1982 PNAS paper was a landmark.

It really cemented the idea of modeling neurobiological systems as dynamical systems.

It showed concretely how emergent properties, like associative memory, could arise from simple connected units.

Even though neuroscientists and computer scientists were initially a bit skeptical.

Apparently so.

Which might be why he ended up publishing in PNAS via his National Academy membership.

That five page limit probably forced incredible conciseness.

You wonder if that brevity, like you said, almost helped.

Made it denser, invited others to fill in the gaps.

It's a cool thought, right?

Like Hemingway's iceberg theory.

Maybe leaving things unsaid spurred more research.

The paper's impact has certainly lasted.

And this was happening around the same time people were figuring out back propagation for more complex networks.

How do Hopfield nets fit in?

They're different beasts.

Hopfield networks are often called one -shot learners.

You calculate the weights once with Hebb's rule.

But real learning is often incremental, right?

Learning from lots of examples over time.

Like training a deep learning model today.

Exactly.

Back propagation enabled that kind of gradual learning from multi -layer networks.

But Hopfield's work was foundational.

It showed the power of network dynamics in this energy landscape metaphor for computation.

So let's recap.

We've seen how Hopfield connected physics.

The Ising model specifically, to neural networks.

He designed these networks with bi -directional symmetric connections.

Crucial symmetry.

And used Hebbian learning to store memories as stable states, which are basically low energy points in the network's energy landscape.

Yep, memories as valleys.

And retrieving a memory is just a network dynamically settling back into one of those valleys when you give it a partial cue.

That's the essence of it.

It's a really elegant connection between physics and computation.

Absolutely, that aha moment of seeing how a system based on physical principles could mimic associative memory.

It's pretty profound.

It really is.

And it makes you think, doesn't it?

About energy landscapes, stability.

How might these ideas apply elsewhere?

In other complex systems, natural or artificial?

What other emergent behaviors could pop out from simple interconnected rules?

It's a great question to ponder.

A real testament to looking across disciplines for fundamental insights.

It definitely gives you a glimpse into principles that might underlie intelligence itself.

Definitely something for you, the listener, to chew on.

And with that thought, we'll wrap up this deep dive into Hopfield Networks.

We've covered the journey from physics and biology through the Ising model and neuron design, the critical roles of symmetry and energy, Hebbian learning for storage, and the dynamics of retrieval.

We've really hit all the key points from this foundational work.