Chapter 1: Desperately Seeking Patterns

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement not replaced the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

Welcome to the Deep Dive.

We're kicking off a fascinating journey today, right into the bedrock of machine learning.

Yeah, really getting into the fundamentals.

Our goal is pretty simple,

really, to get a solid handle on the core ideas that power modern AI,

you know, without getting tangled up in overly technical language.

Right.

It's about tracing that line from the very first sparks of insight.

All the way to the complex AI systems we see now.

And our guide for this first part is chapter one of Anil Ananthaswamy's book, Why Machines Learn, The Elegant Math Behind Modern AI.

The great starting point.

It really is.

People like Yoshua Bengio have praised it for offering a, quote, clear and engaging explanation of the fundamental mathematical ideas.

It helps connect the dots, you know.

Exactly.

So we'll be looking at how machines learn to recognize patterns.

We'll start with some, well, surprisingly simple examples.

And then follow that early inspiration that came from studying arguably the best learning machine out there, the human brain.

OK, let's get started then.

Our deep dive into chapter one, desperately seeking patterns.

So the chapter kicks off with Conrad Lorenz,

his work on imprinting in ducklings.

Why start there?

What's the connection to machine learning?

Well, it's actually a brilliant way to frame the problem.

Lorenz discovered this thing called imprinting.

You know, ducklings hatch.

And they basically latch on to the first moving thing they see.

Uh -huh.

Usually the mother duck?

Usually.

But what's really wild is that it's not just about a specific individual.

The book mentions mallard ducklings imprinting on the idea of sameness.

Sameness.

How does that work?

So, like, if they first see two red balls moving together, later on they'll follow any two balls of the same color, maybe two blue ones, but they won't follow two balls of different colors.

They have the abstract concept.

Wow.

And difference too, presumably.

Exactly.

They can grasp difference as well.

That's incredible that these tiny creatures have this, like, built -in knack for recognizing abstract concepts same, different from just basic sensory stuff.

Right.

And then they use that understanding.

It makes you ask, how on earth do they do that?

And that's the link the book makes.

AI researchers are essentially trying to figure out that same kind of efficient pattern recognition.

How to learn quickly, maybe with less data, just like the ducklings.

Yeah, absolutely.

And okay, maybe our current AI isn't quite that efficient yet.

Not quite duckling -level efficient, no.

But the basic idea is there.

Finding and learning from patterns and data, which naturally leads to the question, what is a pattern and data?

The book is a really simple example here.

Just the table.

Three columns.

By one, by two, and why.

And it asks you, the reader, to spot the connection.

What's the pattern there?

It turns out to be pretty straightforward in this case.

Y equals by one plus two times by two.

So Y equals by one plus two by two.

Simple enough?

Simple, but it makes a key point.

Which is?

That in the real world, these relationships are usually not obvious at all.

So the book generalizes this.

It says, okay, let's write it as Y equals W one by one plus W two by two.

These W one and W two are called coefficients, or more commonly in machine learning, weights.

They sort of dial in the importance of each input.

Exactly.

They define that specific relationship, and then it expands even further.

Imagine you have N inputs.

Y equals W one by one plus W two by two plus W three by three.

All the way up to U and XN.

Okay, that could get really long and complicated fast.

It could.

And just imagine trying to figure out all those W values, maybe 50 of them, just by staring at a giant spreadsheet.

Yeah, practically impossible.

And that's where learning comes in.

Machine learning uses algorithms to automatically figure out those weight values from the data.

Okay, so the algorithm finds the weights.

What's the big payoff?

Why do we want it to do that?

Predictions.

That's the real power.

Once the algorithm learns the weights from some initial data, the training data, you can use those learned weights to predict the Y value for new X values it's never encountered before.

Like the house price example in the book.

Precisely.

If my one is bedrooms, my two is square footage, and Y is the price.

You train the algorithm on data from houses already sold.

It learns the weights W one and W two.

And then you can give it the bedrooms and square footage of a new house.

And it predicts the price Y.

That's the goal.

And this whole process, learning from labeled examples,

input features like bedrooms and square footage, and the known output, the price, that's what we call supervised learning, isn't it?

Exactly.

Supervised because you're supervising the learning with the known correct answers, the labels.

And specifically because we're predicting a number, like price, it's called regression.

Spot on.

Regression is a type of supervised learning.

And it's worth pointing out, like the book does, that modern ML often needs huge amounts of this labeled data.

Right, which is quite a contrast to those efficient ducklings learning from maybe just one or two examples.

A big contrast.

But still, this relatively simple idea, learning weights to predict an output, it's the absolute foundation for even the incredibly complex deep neural networks we use today.

Speaking of those early steps,

the chapter brings in a key figure,

Frank Rosenblatt.

And his invention, the perceptron.

Ah, yes, the perceptron.

Really significant.

Why was it such a big deal back then?

Well, it was one of the very first algorithms inspired by the brain that could actually learn patterns just by looking at data.

That was revolutionary.

Worthy of its own.

Yes.

And crucially, there was a mathematical proof.

It showed that if a pattern the perceptron could learn existed in the data, we'll get to what that means, linearly separable, the algorithm was guaranteed to eventually find it.

Wow, a guarantee that must have caused some excitement.

Oh, huge excitement.

Maybe even too much excitement initially.

But that guarantee, that convergence proof was a massive step.

Now, the book mentions that the groundwork was laid even earlier back in 1943.

McCulloch and Pitts,

a neuroscientist and a logician.

Warren McCulloch and Walter Pitts.

They were thinking about how the brain itself might compute things, like could it perform logic?

So they built a model.

A very simplified model of a biological neuron.

They called it an artificial neuron, or sometimes a neurode.

How did it work?

Super simple, really.

It took binary inputs, just bureau or one.

Let's say by one and by two, it would just sum them up.

Add them together.

Add them together.

And if that sum hit a certain threshold value, they used the Greek letter theta.

For the threshold, the neuron would output a one.

If the sum was below the threshold, it output a zero.

OK, so it's a little decision maker based on a threshold.

Exactly.

And the cool part was just by changing that threshold value, you could make this simple neuron behave like basic logic gates.

Like A, D, and O, R.

Precisely.

Such a two, and it acts like an A and D gate, needs both inputs to be one to output one, set it lower, maybe 1 .5 or even one, and it acts like an O, R gate.

Huh, so these simple units could do logic.

That was the big idea.

Connect enough of these McCulloch -Pitts neurons, these MCP neurons together, and theoretically, you could perform any logical calculation.

It's basically the foundation of digital computing.

But there was a catch, right?

A big catch.

The threshold had to be set by hand.

You had to figure out the right threshold for the logic you wanted.

The MCP neuron itself couldn't learn the threshold from data.

Ah, OK.

So it could compute, but it couldn't learn how to compute based on examples.

Exactly.

And that's where Rosenblatt comes back in.

He built on McCulloch and Pitt's work.

He made it learn.

He did.

His perceptron could learn from data.

That was the crucial difference.

How did he achieve that?

What was his inspiration?

He drew from McCulloch and Pitt's, but also from psychology, particularly Donald Hebb.

Hebb had this idea about how learning might happen in the brain.

The neurons that fire together wire together idea?

That's the one.

The connections between neurons get stronger if they're active around the same time.

Rosenblatt took this idea and applied it to his artificial neuron.

The strength of the connections became adjustable weights.

OK, so weights that could change based on experience.

Exactly.

The book tells a funny little story about Rosenblatt and a student, George Nagy, just showing how sharp Rosenblatt was.

And it mentions the markup perceptron built in 58.

Yes, the markup.

It could actually recognize letters, which was impressive for the time.

But the really big deal wasn't just that it could recognize them.

It was how it learned.

Right.

Nagy said it learned to recognize letters by being zapped when it made a mistake.

Huh.

OK.

Maybe a bit dramatic, but it gets the point across.

Learning from errors.

That's the core of the perceptron learning algorithm, trial and error.

So picture a simple perceptron.

It gets inputs, say, by one and by two again.

Each input has a weight, w1 and w2.

It calculates a weighted sum, w1 times by one plus w2 times by two.

Then, critically, it adds another value called a bias, often written as b.

A bias.

What's that for?

Think of it as giving the neuron a bit more flexibility, shifting its decision point.

So you have the weighted sum plus the bias, then that total value goes through a threshold function.

Like the MCP neuron.

Similar, but often simpler.

If the sum w1 by one plus w2 by two plus b is greater than zero, the output y is plus one.

If it's less than or equal to zero, the output is zero one, or sometimes zero, depending on the setup.

OK, so key differences from MCP inputs can be continuous numbers, not just zero one.

Right.

It uses weights and this bias term.

And the most important part, it can learn those weights and the bias automatically from data.

That's the breaker.

The book uses an example for this learning part, right?

Obesity classification.

Yeah, a straightforward example.

Classify people as obese plus one, or not obese, not a qun, based on just two features,

weight by one and height by two.

So you have some starting data people you already know are obese or not.

Exactly.

Label data again.

The perceptron's job is to learn the weights w1, w2, and the bias b, so it can correctly classify everyone in that initial data set.

And there's a key assumption here.

Yes.

The assumption is that the data is linearly separable.

Meaning?

Imagine plotting all your data points on a graphed weight on one axis, height on the other.

If you can draw a single straight line that perfectly separates all the plus one points obese from all the plus one points not obese, then the data is linearly separable.

Okay, so the perceptron is trying to find that separating line.

Precisely.

The book gives a nice visual for this.

The perceptron starts with a random guess for the line.

Maybe its initial weights and bias are just random numbers?

This line is probably wrong.

It misclassifies some people.

Right.

So it looks at a misclassified point.

If it predicted minus one, but the truth was plus one, or vice versa, it adjusts its weights w1, w2, and its bias b.

How does it adjust them?

It nudges them in a direction that pushes the separating line closer to correctly classifying that point.

It repeats this process for all the data points, potentially many times.

So the line keeps moving, rotating, and shifting.

Exactly.

The weights w1, w2 basically control the slope or orientation of the line.

The bias b controls its offset where it crosses the axis.

Eventually, if the data is linearly separable, the perceptron is guaranteed to find a set of weights and bias that define a line correctly separating the groups.

And once it's learned that line?

Then you can give it the weight and height of a new person, someone not in the training data.

The perceptron calculates w1 by 1 plus w2 by 2 plus b for this new person.

If the result is positive, it predicts plus one, obese.

If negative, it predicts minus one, not obese.

It sees which side of its learned line the new point falled on.

But the book hints that these predictions might still be wrong, even if the line perfectly separated the training data.

Why?

Well, think about it.

There might be multiple different straight lines that could perfectly separate the original training points.

The perceptron learning rule finds one of them.

But is it the best one for classifying new unseen data?

Maybe not.

Maybe another line would generalize better.

Right.

That gets into more complex ideas about finding the optimal model, not just a model that fits the training data,

minimizing future errors.

Exactly.

It hints at challenges beyond just finding a separating line.

So this works with two inputs, weight and height.

Finding a line.

What if you have more inputs?

Good question.

If you have three input features, maybe weight, height, and age by one, by two, by three, then you can't use a simple 1D line to separate the data in 3D space.

You need a plane.

Exactly.

A 2D plane to slice through the 3D space and separate the plus ones from the Eneka ones.

And if you have even more inputs, say n inputs, you need what's called a hyperplane.

It's the higher dimensional equivalent of a line or a plane.

Which brings us back to the Mark Iperceptron again.

It had 400 inputs, right, from the pixels.

Yeah, 20 by 20 pixels.

So it was operating in a 400 dimensional space, trying to find a hyperplane to separate different letters.

Wow.

Okay, that sounds complicated.

The math gets more involved, but the principle is the same.

It had this network of artificial neurons, connections with varying strengths,

those learned rates, and the knowledge of how to classify letters was stored in the values of those weights.

But the chapter ends on a slightly cautionary note, highlighting limitations.

It does.

While these perceptrons were amazing for learning correlations directly from data, it begs that fundamental question.

Is finding statistical patterns the same as real understanding?

Like, did the more I understand what a B or a G meant.

Or was it just very good at finding the combination of pixel weights that distinguish the shapes?

It's a deep question, and it's a debate that's still very much alive today with modern deep learning systems, which are full mentally vastly more complex descendants of Rosenblatt's perceptron.

This incredibly sophisticated pattern matching equal intelligence or reasoning.

A tough one.

It really is.

And the book acknowledges the path from these early models to today's AI wasn't smooth.

There were ups and downs, periods of hype and disappointment.

But a key success was that proof we mentioned earlier, the perceptron convergence theorem, proving it could find that separating hyperplane if one existed.

Right, and the book signals that understanding that proof requires diving into the mathematics of vectors.

Okay, so let's quickly recap what we've covered from this first chapter.

Sounds good.

We started with the basic idea of pattern recognition, seeing it in nature with Lorenz's ducklings.

And then formalizing it in data with simple linear relationships, like why it goes W1 by 1 plus W2 by 2.

Right, introducing weights and the idea that learning is about algorithms finding these weights automatically.

Then we looked at the precursors, the McCull -Pitts neuron,

a simple logic unit inspired by the brain, but it couldn't learn on its own because its threshold was fixed.

Which led us to Rosenblatt's perceptron.

Also brain inspired, but crucially it could learn by adjusting its weights and bias based on errors, learning from mistakes.

We saw how it could find a separating line or a hyperplane in higher dimensions for linearly separable data, using the obesity classification example.

We touched on its limitations and the ongoing questions about pattern recognition versus true understanding.

These early ideas, the MCP neuron, the perceptron, the concept of learning weights, they really are the absolute building blocks for everything that followed in AI and deep learning.

Couldn't agree more, it's the foundation.

And as the book points out, the next logical step in understanding this better involves getting comfortable with vectors, which is key to representing and manipulating data in machine learning.

Definitely.

Vectors are fundamental for the math that comes next.

So maybe a final thought for you, the listener.

Think about how you recognize patterns constantly in your daily life.

Faces, voices, traffic patterns, maybe even social situations.

How does your own pattern recognition compare to these simple yet powerful ideas we discussed?

Yeah, and what does it mean for a machine to learn these patterns?

It's fascinating stuff to ponder with lots of implications.

And with that, we have covered the entirety of the first chapter of why machines learn.

We've laid down that essential groundwork.

Thanks for joining us on the Deep Dive.