Chapter 6: Depth & Space Perception

Welcome to Last Minute Lecture.

This free chapter overview is designed to help students review and understand key concepts.

These summaries supplement not replaced the original textbook and may not be redistributed or resold.

For complete coverage, always consult the official text.

Welcome back to the Deep Dive.

Today we are tackling a subject that is, um,

simultaneously the most obvious thing in your life and, quite possibly, the most complicated engineering problem your brain ever has to solve.

It really is.

It's one of those topics where once you start to peel back the layers, you realize that seeing isn't a passive act.

It's not just opening your eyes and letting the world pour in.

It's a construction project.

It's a full -on construction project.

It's hard work happening every millisecond you're awake.

We are talking about space perception and binocular vision.

Basically, how we navigate a three -dimensional world when our, well, our primary equipment, our eyes, are essentially two -dimensional sensors.

It's a fundamental mismatch.

2D input, 3D output.

But before we get into the, you know, the geometry and the neurons and all of that, I want to start with a scenario from the reading that, honestly, it kind of stressed me out just reading it.

Ah, the bear in the meadow.

The bear in the meadow.

It's the hook the text uses.

And man, it really works.

So, and listener, I want you to picture this.

You're out hiking in a meadow.

It's a beautiful day.

You spot a little bear cub playing in the grass.

And being a modern human, your first thought isn't danger.

It's content.

Exactly.

I need a photo of that.

Yeah.

So you creep forward.

You have your camera out.

And then the classic, the classic mistake happens.

You hear a huff.

You look up.

Mother bear is there.

She sees you.

She is not happy.

And she charges.

She charges.

And the text describes this as the ultimate final exam for your visual system.

It really is.

Because in that moment, your survival depends entirely on your brain's ability to perceive space accurately.

Right.

You don't freeze.

You run.

The text describes this frantic escape.

You're sprinting across the field.

You're dodging through this thicket of trees.

You leap over a stream, scramble up a slope, and finally dive into your Jeep and peel out.

You survive.

You survive.

But the point the book makes is, while your conscious brain was just screaming bear, bear, bear, your visual brain was doing something absolutely miraculous.

A miracle of computation.

I mean, think about the physics of what just happened.

You didn't just move.

You navigated an incredibly complex cluttered 3D environment at high velocity.

You had to pick a path through those trees.

That means you knew instantly where each tree was in relation to your body and your direction of travel.

Not just where they were, but how thick they were, the gaps between them.

Then you get to the stream.

You had to judge its width.

If you judge it too wide, you fall in and well, you know.

The bear gets you.

The bear gets you.

If you judge it too narrow, maybe you overshoot and waste energy.

You calculated the slope of that hill so you could place your feet without face planting.

And you did all of this without pulling out a ruler or a physics textbook.

You just did it.

Exactly.

You successfully perceived space.

And that is the core theme of this entire space perception.

How do we and for that matter, other animals like the rabbit, the bear might have been hunting.

How do we interact with the structural layout of the world?

But this immediately brings up a philosophical problem, which I have to admit I wasn't expecting in a chapter about biology.

The text makes this distinction between realism and positivism.

It's a crucial distinction because it really defines the rules of the game for vision science.

The stance the textbook takes and the one we have to take to get anywhere is realism.

Which is what?

Exactly.

It's the basic assumption that there is a real physical world out there.

The bear is made of atoms.

The trees are solid wood.

And if you run into one, it will hurt.

The world exists independent of our minds.

That feels like common sense.

I mean, it's a pretty useful assumption.

If I don't believe the bear is real, I'm probably going to get eaten.

It's the most useful assumption you can make.

But there is a very powerful counter argument in philosophy called positivism.

And this view argues that, well, we never actually touch the world directly.

The only data we ever get is the evidence of our senses.

That's electrical signals shooting around in our brain.

Patterns of neural firing.

So this is the classic brain in a vat scenario.

Or, you know, for a more modern audience, the matrix.

Right.

There is no spoon.

Exactly.

If I could stimulate your optic nerve in the exact same pattern that the light bouncing off the bear produced, you would see the bear.

You would fear the bear.

You would run from the bear, even if you were just sitting in a chair in a dark lab.

So positivism says the world might just be a very convincing, very consistent hallucination.

Or a simulation.

The writer Philip K.

Dick built his entire career on exploring that idea.

But for the purpose of studying vision science, we have to set that aside.

We have to be realists.

We assume the world is real.

And the goal of the brain is to create an accurate model of it.

But here's the catch.

And it's a big one.

The equipment we used to build that model is, well, it's kind of flawed.

That's putting it mildly.

This brings us to the geometry problem, which is laid out really well in figure 6 .1 of the text.

Okay, let's walk through that.

So on the one hand, you have the world.

We assume the world operates on Euclidean geometry.

Like high school math class.

Parallel lines stay parallel forever.

The angles of a triangle always up to 180 degrees.

An object has a fixed size.

Right.

The rules are stable and predictable.

But then you have the retina, the film, at the back of your eye.

And your eye isn't a flat piece of graph paper.

It's a sphere.

It's curved.

It's curved.

So when that perfect Euclidean world gets projected onto the curved surface of your retina, the geometry gets warped.

It becomes non -Euclidean.

Those parallel train tracks now look like they converge.

A straight line becomes a curve.

And on top of that, the lens flips the image.

Completely.

It's upside down and backward.

So the fundamental question we are trying to answer today is this.

How does the brain take two curved, warped, upside down, two -dimensional images and from that reconstruct a stable, upright, three -dimensional world that we can trust our lives with when a bear is charging?

It seems almost impossible when you lay it out like that.

And the text suggests that a huge part of the solution comes down to a very simple fact.

We have two eyes.

The two -eye setup.

Binocular vision.

Now, I've always just assumed the main reason for having two eyes was, you know, for redundancy.

Like having a spare tire.

That's the spare part theory.

Yeah.

If I poke one eye out on a branch while running from that bear, I'm not totally blind.

I can still sort of function.

And that is a legitimate evolutionary benefit.

It's absolutely true.

Redundancy is great for survival.

But if that were the only reason, evolution might have given us an eye on the back of our head or maybe on our knees.

The specific placement of our eyes tells us there's more going on.

Right.

We have frontal eyes.

Rabbits have lateral eyes.

And this is the predator versus prey distinction.

It's a classic trade -off.

And figure 6 .3 in the textbook illustrates this beautifully.

If you look at the diagram for the rabbit, you see its eyes are positioned on the sides of its head.

This gives it that massive field of view.

A massive field of view.

The text calls it a planetarium dome of vision, which is just perfect description.

A rabbit can see nearly 360 degrees without moving its head.

It can see its own tail.

It can see a hawk diving from almost directly above its head.

So for a prey animal, the number one priority is detection.

Is there danger anywhere in my environment?

It doesn't need to know exactly how far away the fox is.

It just needs to know there is a fox.

And then it runs.

But there's a trade -off.

Because their eyes face in opposite directions, there's very little overlap.

The left eye sees the left world.

The right eye sees the right world.

They don't really see the same thing.

Very little overlap.

Whereas humans and other predators, like cats or owls, we have frontal eyes,

our total field of view is actually pretty terrible compared to a rabbit.

We only see about 190 degrees horizontally.

We're completely blind to what's happening behind us.

But because our eyes are right next to each other, facing forward, we have this huge area, about 110 degrees of binocular overlap.

This is the region of space that both eyes are looking at simultaneously.

And that overlap is where all the magic happens.

It allows for two crucial advantages.

The first and simpler one is called binocular summation.

This is basically the idea that two detectors are better than one.

Exactly.

Let's clip the scenario.

Imagine you are the bear now.

You're trying to spot a rabbit hiding in the tall, camouflaged grass.

It's a faint signal against a noisy background.

It's hard to see.

The text had some probability math here that I found really interesting.

It wasn't as simple as just two eyes are twice as good.

No, it's more clever than that.

It's what we call probability summation.

Let's say, just for the sake of argument, your left eye, on its own, has a 50 % chance of missing the rabbit.

And your right eye also has a 50 % chance of missing it.

Okay.

If you only had one eye, you failed to see your lunch half the time.

But with two eyes, for you to miss the rabbit, both eyes have to fail at the exact same time.

So you multiply the probabilities, 0 .5 times 0 .5, which is 0 .25.

Exactly.

Your failure rate drops from 50 % down to 25%.

Your success rate in spotting the rabbit jumps from 50 % to 75%.

Just by having two independent and frankly somewhat noisy detectors working on the same target, you become a much more efficient hunter.

But the real superpower of that binocular overlap isn't just about detection.

It's this concept of disparity.

This is the core concept of the entire chapter.

Because your eyes are separated by, on average, about six centimeters, they are viewing the world from slightly different vantage points.

They never see the exact same image.

And there's a demo for this in the book, figure 6 .2, the finger jump.

Everyone should do this right now.

It's the best way to feel what we're talking about.

Hold one finger up, maybe about six inches from your nose.

Now look at a distant object, like a bookshelf across the room.

Okay.

Now while looking at the bookshelf, close your left eye, then open it and close your right eye.

Just swap back and forth rapidly between your two eyes.

The finger jumps a lot.

It leaps from side to side.

That jump is the disparity.

Now do the same thing, but this time look at your finger.

And notice what the bookshelf in the background does.

It jumps.

Right.

So whatever I'm not looking at is what seems to move.

Exactly.

The brain uses that difference, the amount of that jump, to calculate distance.

Things that jump a lot are close.

Things that jump only a little are far away.

This is the raw data for stereopsis.

Which literally means solid sight.

It's the ability to turn that two -dimensional difference, that disparity,

into a three -dimensional sensation of depth.

I tried the pen cap demo that was mentioned in the intro section.

You hold a pen in one hand and a cap in the other, and you try to put the cap on.

With both eyes open, it's easy.

Trivial.

Then I close one eye, and it was embarrassing.

I missed.

I jabbed my own thumb.

It felt hesitant and clumsy.

I felt like I'd lost my superpower.

That's a perfect description of it.

You lost stereopsis.

And with it, you lost that source of high -resolution, precise depth information.

Without it, you're just guessing based on other cues.

And this is a really critical point the book makes.

You can still function.

Yes.

This is so important.

People who are one -eyed or monocular can still drive cars, land airplanes, play sports, and cap pens, even if it's a bit harder.

It's not like the world goes flat.

Exactly.

If stereopsis were the only way we saw depth, closing one eye would turn the world into a flat cartoon.

But it doesn't.

The world still looks 3D.

And that's because the brain is incredibly resourceful.

It has a whole committee of other cues it can rely on.

And we call these monocular cues.

Also known as pictorial depth cues,

because painters figured them all out centuries ago.

Painters were the original hackers of the visual system.

I mean, think about their problem.

I have a flat, two -dimensional canvas.

How do I trick the viewer's brain into seeing a vast landscape or a cathedral that seems to go back for miles?

They had to discover these rules by trial and error.

So let's run through the checklist of these cheats, these tricks the brain uses.

The first one is the most obvious one of all.

Occlusion.

Occlusion is simply obstruction.

If object A is blocking your view of object B, then object A must be closer.

It's simple, it's robust, it almost never fails in the real world.

But the text brings up a really subtle and interesting point here about accidental versus generic viewpoints.

Yes.

This gets into the idea that the brain is making assumptions, which we'll see more of with the Bayesian approach later.

So imagine you see a perfect square, and there's a circle in front of it blocking one of the corners of the square.

Okay, I'm picturing that.

A circle overlapping a square.

Your brain immediately concludes there is a full square behind that circle.

Now, technically that might not be true.

It could be that you're looking at a circle sitting right next to a weird L -shaped object that looks like a square with a bite taken out of it.

And I just happen to be standing in the one in a million spot where they line up perfectly to look like overlap.

Right.

That would be the accidental viewpoint.

But the brain just rejects that possibility.

It's too unlikely.

The brain assumes a generic viewpoint.

It assumes we are not in some special magical position.

It bets on the most likely explanation.

It's a complete square, and it's just behind the circle.

So occlusion gives us the order of things.

A is before B.

But it doesn't tell us how much distance is between them.

Correct.

That's why we call it a nonmetrical depth queue.

It tells you the order, but not the magnitude.

It doesn't tell you if the square is an inch behind the circle or a mile behind it.

For that, we need other queues.

Queues like relative size.

This relies on something called projective geometry, which is just a fancy way of saying we all have this implicit knowledge that objects create smaller images on our retinas as they get farther away.

The brain runs that logic in reverse.

Small image equals far object.

Right.

But it only works if you make a key assumption that the objects are physically the same size to begin with.

The example in figure 6 .6 is the red balls.

Exactly.

You see a picture with a bunch of red balls on it.

Some are drawn large.

Some are drawn tiny.

Your brain doesn't conclude, oh, look, a collection of giant beach balls and tiny marbles all lined up at the same distance.

It instantly builds a 3D model.

The big ones are close.

The tiny ones are far away.

And this connects directly to the next queue, texture gradient, which is basically just a lot of similar objects packed together.

Think of a field of flowers or a pebbly beach.

Or to stick with our theme, the field of rabbits from figure 6 .7.

A texture of rabbits.

I like that.

If you look at the rabbits near your feet, they look big and fluffy.

You can see their individual whiskers, the details.

As you look out toward the horizon, the images of the rabbits get smaller.

They get packed closer together on your retina.

The density of rabbit texture increases.

That smooth change in size and density, the gradient, tells your brain this is a flat ground plane receding away from me.

It's a very powerful queue for perceiving surfaces.

And it often works together with relative height.

This one is simple, but incredibly powerful.

For objects on the ground, the higher they are in your visual field, which means the closer they are to the horizon line, the farther away they are perceived to be.

So in that rabbit picture, the rabbit at the very bottom of the photo looks the closest.

A rabbit near the middle of the photo, closer to the horizon, looks far away.

But look what happens when you mess with these queues.

Figure 6 .10 shows this amazing illusion.

They take that far rabbit, which is drawn really small to look far away, and they just copy and paste it to the bottom of the picture, next to the close rabbit.

And it looks ridiculous.

It looks like a tiny miniature toy rabbit.

Because the queues are now in conflict.

Its position, its relative height, says this object is close.

But its tiny size says this object is far.

So the brain compromises and concludes, it must be a tiny object that is close to me.

This all relies on assumptions, though.

What if we actually know the size of an object?

Now you're talking about familiar size.

This is a huge one.

And unlike the others, it can be a metrical queue.

It can give you an estimate of the absolute distance in, say, feet or meters.

The classic example is a human hand.

You know, roughly how big a hand is.

Right.

So if you see a hand in your visual field that looks tiny, you don't think, wow, what a tiny person.

Your brain does a quick unconscious calculation based on the known size of a hand and the size of its image on your retina, and concludes that person must be about 20 feet away.

The text mentions that experiment from Figure 6 .11, where they showed people familiar objects like playing cards in a completely dark room.

Yes, this is a brilliant study.

When they showed people a normal -sized king of spades, people were pretty good at judging how far away it was.

But then they would show them a giant poster -sized playing card, but position it really far away so that it created the same size retinal image as the normal card up close.

And people got it wrong.

They got it wrong.

They reported that it was a normal -sized card that was close to them.

Their knowledge of the card's familiar size overrode the other queues.

But here's the kicker.

When they did the same thing with blank white rectangles of different sizes, people were terrible at judging the distance.

They had no idea.

They had no idea.

They needed that king of spades identity, that prior knowledge, to anchor their perception of distance.

It shows that knowing what something is directly changes where you see it.

That is wild.

It's like object recognition and space perception are totally intertwined.

They have to be.

Okay, let's move from objects to the atmosphere itself.

Aerial perspective.

Why do distant mountains look blue and hazy?

It's not just an artistic choice.

It's pure physics.

The atmosphere between you and that mountain isn't empty.

It's full of particles, water vapor, dust, pollution.

Gunk.

Lots of gunk.

And when light from that mountain travels through miles of this gunk to reach your eye, it gets scattered.

And short wavelengths of light blue and violet light scatter the most.

That's Rayleigh scattering, right?

The same reason the sky is blue.

The exact same principle.

So if you're looking at a mountain 10 miles away, you're effectively looking through a 10 mile long blue fog.

The light from the mountain gets washed out.

It looks lower in contrast, hazier, and tinted blue.

So our brain learns this rule.

Hazy and blue equals far away.

And sharp and high contrast equals close.

And this can create dangerous illusions too.

The book mentions driving in a heavy fog.

The car in front of you is very hazy because of all the water droplets.

Your brain's aerial perspective module says hazy object must be far away.

So you might not break in time because you misjudged the distance?

It's a terrifyingly real world example of a depth cue leading you astray.

All right, let's get to the big one.

The one that literally changed art history.

Linear perspective.

This is the heavyweight champion of the pictorial cues.

The rule is simple.

Parallel lines in the 3D world appear to converge at a single point.

The vanishing point in a 2D image.

The classic example is train tracks.

Train tracks, a long hallway, a straight road, disappearing into the distance.

We know intellectually that the two rails of the track are parallel.

They never touch.

But look down a long stretch of track and they form a perfect triangle pointing to the horizon.

The brain learns this rule and runs it in reverse.

If these lines in my vision are converging, they must represent parallel lines receding in depth.

And artists didn't always know this?

Not at all.

If you look at a lot of medieval art, it looks flat.

Objects seem stacked on top of each other rather than being behind one another.

It wasn't until the Renaissance artists and architects like Brunellesi and Alberti that they started to systematically codify the rules.

They treated painting like a geometry problem.

They realized that if you want to paint a realistic room, you have to use a ruler.

You have to pick a vanishing point on your canvas and make sure all the receding parallel lines, the edges of the floor tiles, the ceiling beams, all go to that one point.

It was a technological revolution in art, all based on hacking this one powerful depth cue that our brains, and even dogs' and cats' brains, already knew implicitly.

And this leads to some really weird and cool stuff like anamorphosis.

This is when you take linear perspective to an extreme.

The most famous example shown in Figure 6 .19 is Hans Holbein's painting The Ambassadors.

It's this incredible photorealistic painting of two wealthy, serious -looking diplomats surrounded by scientific instruments, very stately.

But then at their feet, smeared across the tiled floor, is this weird stretched out gray shape.

It looks like a mistake, like someone dropped the painting and the paint smeared.

It's totally jarring, but it's not a mistake.

If you walk to the far right of the painting and crouch down, looking at the canvas from a very, very sharp angle,

that smear suddenly compresses and resolves into a perfectly rendered human skull.

A memento mori.

A reminder of death, hidden in plain sight.

Exactly.

It's a secret message that you can only see from one specific accidental viewpoint.

And we see this today with those amazing sidewalk chalk artists like Julian Beaver.

They draw what looks like a giant canyon or a swimming pool in the middle of a city street.

And if you stand in the one sweet spot, it looks perfectly three -dimensional.

You feel like you could fall in.

But if you take two steps to the side, the illusion shatters.

It just looks like a weird, stretched out, distorted drawing on the pavement.

That is anamorphosis.

Okay, so all of those are static cues.

They work on a single, still picture.

But we are not static creatures.

We move.

And our movement gives us a hugely powerful cue called motion parallax.

Yes.

This is one of the triangulation cues.

It's incredibly powerful.

The easiest way to understand it is the train window example from figure 6 .21.

I do this every single time I'm on a train or in a car.

Of course.

You're looking at the window.

The flowers and fence posts right next to the tracks are flying by.

They're a blur.

Zip, zip, zip.

But if you look a bit farther out, at a farmhouse in the middle distance, it seems to glide by much more slowly.

And then you look at the mountains on the horizon.

And they barely seem to move at all.

They look like they're following you.

Right.

So your brain learns a very simple and reliable rule.

Fast visual motion equals close.

Slow visual motion equals far.

And you don't even need a train to generate this.

You can do it just by moving your head.

Exactly.

This brings us back to animal behavior.

Have you ever watched a cat or a bird getting ready to jump onto a branch or a table?

Oh, yeah.

The cat does that little head bobble, the side to side wiggle.

They're not just psyching themselves up.

They are actively creating motion parallax.

By moving their head side to side, they are generating relative motion between the edge of the table, the chair behind it, and the floor below.

That motion gives their brain a precise, metrical estimate of the distances involved.

They are triangulating the jump.

That completely changes how I see that behavior.

It's not a quirky habit.

It's an engineering calibration.

It is.

They are arranging their target.

What about the cues from our own bodies?

Can we feel our eyes working to judge distance?

We can, to a limited extent.

The text groups these as the other triangulation cues.

Accommodation and convergence.

Accommodation is the lens of your eye changing shape, right?

Yes.

Your ciliary muscles squeeze the lens to make it fatter and more curved to focus on near objects.

When you look far away, the muscles relax and the lens flattens.

Your brain gets a copy of that muscle signal.

We are squeezing this much and can use it as a rough cue for distance.

And convergence.

That's the angle of your eyes.

To look at something close, like the tip of your nose, your eyes have to rotate inward.

They have to cross.

The angle between them is large.

To look at something far away, your eyes rotate outward until they're nearly parallel.

Your brain senses the tension in those eye muscles and uses that angle to calculate distance.

But the textbook says these are pretty useless for anything that's not right in front of your face.

That's right.

Beyond about two or three meters, maybe six feet, the angle of convergence is almost zero and your lens is fully relaxed.

So the signal just flatlines.

It can't tell you the difference between 10 feet and 100 feet.

It's only really good for within arm's reach distances.

Unless you are a chameleon.

Chameleons are the amazing exception.

They have these incredible eyes that move independently.

So they can't use convergence the way we do.

They seem to rely almost entirely on accommodation.

They will lock one eye onto a bug.

And the amount of muscular effort it takes to bring that bug into sharp focus tells their brain the exact range.

And researchers prove this by putting glasses on them.

I still can't get over this.

It's one of the all -time great experiments.

Harkness did this.

They fitted chameleons with tiny glasses containing minus lenses, concave lenses, the kind that make things look smaller.

Optically, this forces the eye to accommodate more than usual to get a sharp image.

Quote, the glasses trick the chameleon into thinking the bug is closer than it really is.

Precisely.

The chameleon's brain gets the signal.

We are squeezing the lens really hard.

This bug must be very close.

And so the chameleon shoots its amazing tongue out and falls short.

It misses the bug by the exact distance predicted by the optics of the glasses.

It's a cruel but absolutely brilliant proof that they use focus as a metrical depth cue.

It's airtight.

OK, so we've covered the whole suite of monocular keys.

We've covered motion.

Now we have to get back to the heavyweight champion.

The main reason we have two frontal eyes, binocular stereopsis.

Right.

And to really get this, we have to dig into the geometry a little bit.

Because it's really elegant.

The text introduces this with a scenario it calls Bob and the Crayons, shown in figure 6 .23.

It's a great simple way to visualize it.

So imagine you're looking at a scene.

There's a red crayon and a blue crayon.

You decide to look directly at the red crayon.

That's your fixation point.

OK, I'm fixated on the red crayon and my eyes are aimed right at it.

That means the image of the red crayon falls on the very center of your retina in both eyes.

This central spot is called the fovea.

So the red crayon's image is on the left fovea and the right fovea.

These two points are called corresponding retinal points.

They match up perfectly.

They match perfectly.

And because they match, the disparity is zero.

The brain says, aha, zero disparity.

This is the thing I'm looking at.

Now imagine an imaginary arc or a circle that passes through that red crayon in your two eyes.

This circle is called the Vieth Muller circle.

OK.

Any other object in the world that happens to lie on that circle will also cast its image onto corresponding points in your two retinas.

This entire surface of zero disparity is called the heropter.

The heropter.

So anything on the heropter is seen as a single object at the same distance as my fixation point.

Exactly.

But what about the blue crayon?

Let's say the blue crayon is closer to you than the red one.

It's inside the heropter.

Now it is not on the heropter, so its image will land on non -corresponding points.

And here's the cool part.

In your left eye, the image of that closer blue crayon will land a little bit to the right of your fovea.

In your right eye, its image will land a little bit to the left of your fovea.

Wait, let me trace that in my head.

Left eye sees it on the right.

Right eye sees it on the left.

The images are crossed relative to the fovea.

Exactly.

This is called cross disparity.

And cross disparity is the unambiguous neural signal for near.

It tells the brain, this object is in front of the thing you're looking at.

And I assume the opposite is true if the blue crayon is behind the red one, farther away.

Then you get uncrossed disparity.

In the left eye, the image is to the left of the fovea.

In the right eye, the image is to the right.

Uncrossed is the signal for far.

So the brain is constantly doing this calculation.

Is the disparity for this object crossed or uncrossed?

And by how much?

And that by how much, the magnitude of the disparity tells you the distance.

A really big cross disparity.

That object is right up in your face.

A tiny little bit of cross disparity.

It's just an inch or two in front of the object you're looking at.

But at some point, if the disparity gets too big, the system breaks down.

It does.

You get diplopia, which is just the technical term for a double vision.

If you hold your finger three inches from your nose, but you click looking at the wall across the room, you will see two transparent ghostly fingers.

The disparity is so huge that the brain just gives up trying to fuse them into one object.

But there's a little buffer zone around the heropter where we can still fuse them, even if the disparity isn't zero.

Yes, that's called Panam's fusional area.

It's this narrow zone in front of and behind the heropter where the brain can take two slightly different images and still merge them into a single solid 3D perception.

Step outside that zone, though, and you get diplopia.

Now, who figured all this stuff out?

It seems so fundamental, but the pack says it's a surprisingly recent discovery.

It really is.

The ancient Greeks, like Euclid, knew the geometry of vision, but they completely missed this.

They didn't realize that the difference between the two eyes' views was the secret ingredient for depth perception.

That discovery belongs to Sir Charles Wheatstone, an English scientist, in 1838.

He invented the stereoscope.

He did, and it's a great story of scientific insight and rivalry.

Wheatstone built this clunky but brilliant device using mirrors to present two slightly different drawings, one to each eye.

For instance, two drawings of a cube from slightly different perspectives.

And people saw a 3D cube.

They saw a 3D cube floating in space.

He proved for the first time that if you feed the brain retinal disparity, the brain will construct depth.

It was a scientific sensation.

But then his other guys, Sir David Brewster, came along.

Brewster was, let's say, a difficult personality.

He apparently couldn't stand Wheatstone.

He invented his own version of a stereoscope that used lenses instead of mirrors, which made it much smaller, cheaper, and easier to use.

He took it to the great exhibition of 1851, the Big World's Fair in London.

Showed it to Queen Victoria.

She was amazed by it.

And suddenly, stereoscopes became the hottest piece of technology.

It was the VR headset of the 19th century.

Absolutely.

Every middle -class Victorian parlor had a Holmes Brewster stereoscope and a box of stereo cards.

You could sit in your foggy London home and look at a stunning 3D view of the pyramids or Niagara Falls or a scene from the Boer War.

It was the first truly immersive mass media.

But you don't actually need a machine to do this if you can master the art of free fusion.

The poor man's stereoscope.

This is where you learn to cross your eyes or diverge them, intentionally to overlap two side -by -side images like the ones in figure 6 .33.

I find this so incredibly hard to do.

It is hard.

It's hard because you have to fight a deeply ingrained reflex.

Normally, when you cross your eyes to look at something close, that's convergence, your lens automatically gets fatter to focus up close, that's accommodation.

To free fuse, you have to decouple those two systems.

You have to cross your eyes as if looking at your nose, but keep your focus relaxed as if looking at the wall.

It feels like trying to pat your head and rub your stomach at the same time.

It's exactly that kind of motor control problem.

But some people can't do it at all, not because they can't learn the trick, but because their visual system is different.

And this is the topic of stereo blindness.

This brings us to one of the most compelling stories in the entire chapter, I think.

The story of Stereo Sue.

Dr.

Susan Barry.

It's an incredible story.

She's a neurobiologist, so she understood the science behind all of this.

And she had been cross -eyed.

The clinical term is strabismus, since she was a baby.

And when you have strabismus, your eyes are pointing in different directions.

So your brain is getting two wildly different images.

To avoid constant, confusing double vision, the brain does something drastic.

It just suppresses the input from one of the eyes.

It turns one of them off.

So she essentially viewed the world monocularly her whole life.

But she didn't know she was missing anything, right?

How could she?

The world looked normal to her.

It looked 3D.

She used all those monocular cues we talked about, accretion, motion parallax, relative size.

She was a successful scientist.

She could drive.

She could function perfectly well.

She just assumed that's how everyone saw the world.

But then, at the age of 48, she decided to try vision therapy.

Which flew in the face of all the scientific dogma at the time.

The dogma was there's a critical period for developing binocular vision.

If you don't fix a child's strabismus by the age of three or four, the neural pathways for stereopsis either die off or get permanently repurposed for something else.

You can't learn it as an adult.

The window has closed.

Sue proved them all wrong.

She did.

After months of really difficult grueling therapy to teach her eyes how to align and work together, one day, it just happened.

She was walking out to her car after a therapy session.

She looked at the steering wheel, and suddenly, it popped.

Describe that sensation.

What did she see?

She described the steering wheel as floating in space with a tangible volume.

But the most important thing, the thing that blew her mind, was that she could see the space between the steering wheel and the dashboard.

She called it the palpable void.

She could see the emptiness.

That's such a profound concept.

We always think of seeing things.

But stereopsis allows you to see nothingness.

It lets you see the volume of empty space.

Exactly.

The world suddenly had a new dimension.

She went back into her office, and the hanging light fixture looked like it was dripping from the ceiling.

She saw snowflakes falling in winter, and each one had its own little pocket of space.

She realized that for 48 years, she had been living in a beautifully rendered but ultimately flat world, and she had finally stepped into the real 3D one.

It's an amazing story about neural plasticity.

The adult brain is much more flexible than we used to think.

Absolutely.

The neurons for stereopsis were there all along.

They were just dormant.

Her story and others like it have really challenged that strict idea of a critical period.

So let's talk about those neurons.

What is physically happening in the brain that makes stereopsis work?

Okay, this takes us into the physiology of stereopsis.

The journey starts in V1, the primary visual cortex, right at the back of your head.

This is the first stop for all the visual data coming from the retinas.

It's the first stop where the data from the two eyes is combined.

Before V1, in the thalamus, the signals are kept in separate channels, left eye here, right eye here.

But in V1, we find the first binocular neurons.

These are cells that receive input from both eyes.

And these neurons are picky about what they respond to.

Extremely picky.

This is where the magic happens.

Neuroscientists have found that these cells are tuned to specific amounts of disparity.

You literally have neurons that will fire like crazy only when they see an image with zero disparity.

Those are the horopter neurons.

And there are others for near and far.

Exactly.

You have other populations of neurons that only fire for near disparity, for cross disparity.

And a different set that only fire for far disparity or uncross disparity.

As shown in figure 6 .41, they have what we call disparity tuning curves.

So right now, as I'm looking at you, I have a population of zero disparity neurons firing for your face.

And a population of far neurons firing for the wall behind you.

Yes.

And that pattern of activity, which neurons are firing and how strongly, is the brain's code for 3D space.

This information then gets funneled into two distinct processing pathways or streams in the brain.

The famous dorsal and ventral streams.

Right.

The dorsal stream is the where or how pathway.

It goes up from V1 into the parietal lobe.

It uses that precise stereoscopic information to guide your actions.

Where's that pen cap?

How do I need to shape my hand to grab it?

And it needs exact, metrical depth information.

And the ventral stream.

The ventral stream is the what pathway?

It goes down into the temporal lobe, which is involved in object recognition.

It uses depth information to help figure out what things are.

Is that a bump on the surface of this object or is it a hole?

It's more about the qualitative shape of things.

But before any of this fancy processing in the dorsal or ventral streams can happen, the brain has to solve a massive, and I mean, massive logical puzzle.

The book calls it the correspondence problem.

This is the part of vision that has stumped computer scientists for decades.

To understand it, imagine you are not looking at a simple scene with two crayons.

Imagine you are looking at a field of grass or a gravel driveway or a starry night.

OK, a very complex scene with many identical looking elements.

Exactly.

Your left eye sees a thousand little pebbles.

Your right eye sees a thousand little pebbles.

But they're all slightly shifted because of disparity.

The problem is this.

How does the brain know that pebble hashtag 432 in the left eye's image corresponds to pebble hashtag 432 in the right eye's image?

Why can't it just match them up based on what they look like?

Because they all look the same.

They're all just little gray blobs.

Why doesn't the brain accidentally match pebble hashtag 432 on the left with pebble hashtag 500 on the right?

If it makes a false match, the disparity calculation will be completely wrong and you'll see a phantom object at a crazy depth.

So how does it solve this seemingly impossible problem?

It uses a set of heuristics, built -in assumptions, or rules of thumb about the world.

One is the uniqueness constraint.

The brain assumes that a feature in the world will be represented once and only once in each retinal image.

So pebble hashtag 432 in the left eye can only be matched to one feature in the right eye.

That makes sense.

One pebble can't be in two places at once.

Another one is the continuity constraint.

The brain assumes that, for the most part, the world is made of smooth, continuous surfaces.

It assumes that disparity will change smoothly across an object, so it prefers matches that create smooth depth surfaces rather than matches that create a chaotic, random cloud of points.

And when this system fails, you get illusions.

You get the wallpaper illusion.

If you relax your eyes and stare at a repetitive pattern, like the floral wallpaper in a hotel,

your brain can get confused.

It might accidentally lock on to the wrong match.

It might match a flower in the left eye's view with the flower next to it in the right eye's view.

And suddenly the wall seems to warp and jump out at you or recede into a tunnel.

Exactly.

You've tricked the brain's correspondence solver.

This brings us to the scientist, Bella Jules.

He was the one who designed the ultimate test for the correspondence problem.

He wanted to strip away all the recognizable objects and see if the brain could solve this problem using only disparity itself.

This was the invention of the random dot stereogram, or RDS, and it was a complete revolution in vision science in the 1960s.

Before Jules, most scientists had a top -down view of this process.

They thought, first, you recognize the object in each eye.

Okay, that's a car.

Then you match the car in the left eye to the car in the right eye.

Then you calculate the depth of the car.

Right, recognition comes first.

Jules said, I bet the brain is smarter than that.

I bet pseriopsis happens at a much lower level.

So he created these images, which you can see in figure 6 .34, that are made of nothing but random static, thousands of black and white dots.

No shapes, no objects, no outlines.

To one eye, it just looks like the snow on an old TV screen.

It's pure noise.

But in one of the images, he secretly took a central square region of dots and shifted them all horizontally, just a few pixels to the side.

Then he filled in the gap that was left behind with more random dots.

So if you look at either the left or the right image with one eye?

You see absolutely nothing.

There is no monocular cue for the square.

It is perfectly camouflaged.

But when you put those two images in a stereoscope, so one eye sees one and one sees the other.

Pop.

A square floats in vivid 3D above the background.

This is what's called a cyclopean image.

Yes.

From the mythical Cyclops, who had one eye.

A cyclopean image is a shape that is invisible to either eye alone and is defined only by binocular disparity.

This was a bombshell.

It proved that stereopsis happens before object recognition.

It's a low -level bottom -up process.

It's a primitive mechanism for breaking camouflage.

Which brings us full circle back to the animals.

Because for a predator or prey, breaking camouflage is a matter of life and death.

Absolutely.

And this leads us to what is, without a doubt, my favorite experiment in the entire book.

The cuttlefish.

I cannot believe this is real science, but I love every part of it.

It's a study by Fjord and colleagues.

They wanted to know if cuttlefish, which are invertebrates, mollusks, a totally different evolutionary line from us,

have stereopsis.

So naturally, they decided to put 3D glasses on a cuttlefish.

I mean, can you just picture the lab meeting where someone pitched this idea?

They actually glued a tiny patch of Velcro to the cuttlefish's head.

Or mantle, I guess.

The cuttlefish was okay with this.

Apparently, they tolerated it quite well.

Then they attached a pair of very lightweight 3D glasses,

the old school anaglyph kind, with a red filter for one eye and a blue filter for the other.

The kind you'd get at a 50s horror movie.

Exactly.

Then they put the bespectacled cuttlefish in a tank in front of a computer screen and played it a video of a tasty looking shrimp, its natural prey.

And then they messed with the disparity of the shrimp image.

They shifted the red and blue images on the screen to create disparity, making the shrimp appear to be floating in the water in front of the screen.

A virtual shrimp hologram.

And the cuttlefish fell for it.

It struck at the shrimp.

It struck.

But here is the crucial, brilliant part.

It adjusted its strike perfectly to the virtual distance.

If the disparity cues said the shrimp was floating two inches from the screen, the cuttlefish shot its feeding tentacles out and stopped exactly two inches short of the screen.

It was aiming at the hologram.

That's incredible.

It proves, without a doubt, that it uses stereopsis to gauge distance.

And they did the same thing with praying mantises.

A study by Nichinanda and colleagues glued even tinier green and blue filters over the eyes of a praying mantis.

This is just amazing.

A mantis brain is minuscule.

They showed the mantis virtual bugs at different stereo depths and they found that the mantis would only strike when the disparity indicated the bug was within its catch range about two centimeters.

It proves that stereopsis isn't some high -level intellectual feat unique to primates.

It's a basic, fundamental survival tool that has evolved independently in multiple species.

So to pull all this together, we have this massive toolkit.

We have monocular cues like occlusion and size and perspective.

We have motion parallax.

We have accommodation and convergence.

And we have stereopsis.

How does the brain decide who to listen to when it's building our 3D world?

The text uses the metaphor of a committee, the perception committee.

I like that.

Imagine you're in a boardroom.

You've got the occlusion representative, the texture gradient rep, the linear perspective rep, and the big loud disparity rep.

And you're looking at a scene and they are all shouting out their estimates.

I think that object is five meters away.

No, my calculations say it's 10 meters.

Well, I can tell you it's definitely in front of that other thing.

So how does the committee reach a verdict?

How do they vote?

The modern view is that the brain uses a Bayesian approach, which sounds complicated, but the core idea is simple.

The brain weighs the input from each committee member based on two things.

How reliable that cue is in the current situation and what it already knows about the world.

That's the prior knowledge part, the prior.

Yes.

A prior is your built -in background belief about how the world usually works.

For example, you have a strong prior that pennies are round and flat and pennies are about two centimeters wide.

So if I'm looking at a penny from an angle, so its image on my retina is an ellipse.

Your brain doesn't conclude, wow, someone squashed that penny into an oval.

It says my prior is that pennies are round.

The visual data is an ellipse.

The most probable explanation is that I'm viewing a round penny from an angle.

The prior overrides the raw data.

Same thing if I see a penny that looks tiny.

The brain assumes it's far away, not that it's a miniature penny.

Exactly.

We combine the sensory evidence with our prior beliefs to arrive at the most probable perception.

But this is also why illusions work so well.

Illusions are just cases where our brain's best guess, based on its built -in rules, happens to be wrong.

The Ponzo Elysian from figure 6 .47 is the perfect example.

It is.

So you have a drawing of two horizontal lines.

One is at the top of the image, one is at the bottom.

These two lines are physically the exact same length on the page.

But they don't look at the top.

One looks much longer.

It looks way longer.

And that's because they're drawn over a background image of receding train tracks.

So the linear perspective committee member is screaming at the top of his lungs.

Those tracks are receding into the distance.

That means the top part of this image is far away.

So the brain says, OK, that top line is far away.

But it's still making a retinal image of this size.

For something to be that far away and still make an image this big, it must be a giant object in the real world.

So the brain scales up your perception of its size.

It's trying to be helpful by correcting for perceived distance.

It's applying the correct logic of a 3D world to a flat 2D drawing.

And in this artificial case, that logic leads to a perceptual error.

It's a guess gone wrong.

Before we close out, I just want to touch on some of the real world applications of this.

Because understanding stereo vision isn't just an academic exercise.

It has really practical uses.

Absolutely.

The book mentions aerial photography and cartography.

This is a big one.

If you're a spy plane or a mapping plane, you fly over terrain and take two photos of the ground, but separated by thousands of feet.

You look at those two photos in a stereoscope.

Yes.

And the mountains and valleys pop out in stunning 3D.

But here's the trick.

Because your interocular distance isn't six centimeters, but maybe 3 ,000 feet, you get what's called hyperstereo.

It exaggerates the depth.

Massively.

A gently rolling hill can look like Mount Everest.

It makes it much easier for cartographers to map the terrain's elevation.

And the other application was baggage screening.

Right.

If you're a TSA agent looking at a normal 2D x -ray of a cluttered suitcase, it's a nightmare.

Is that a knife or is it just the weird angle of a hairdryer and a comb overlapping?

But if you use a machine that takes two x -rays from slightly different angles and presents them in stereo.

The layers separate?

The layers separate.

You can see the knife floating in 3D space behind the laptop.

It makes spotting threats and clutter much, much easier.

So let's try to wrap this whole amazing topic up.

We started with a frantic bear chase.

We realized that to survive that chase, to find the path to the Jeep, our brain had to solve this incredibly complex inverse problem.

It had to take 2D curved warped upside down data from two separate sensors.

And from that build a stable reliable 3D model of a Euclidean world.

And it did it by assembling this committee of clues using shadows, using lines, using texture, using motion, and most powerfully by using the incredible geometry of two eyes triangulating on a target to create stereopsis.

It's incredibly humbling when you think about it.

We walk around feeling like we just passively see the world as it is.

But really our brain is actively constructing that world millisecond by millisecond from ambiguous and flawed data.

And that leads me to my final thought, a little provocation for you, the listener, to chew on.

Okay, lay it on me.

We spent a lot of time talking about how stereosue finally gained the ability to see the space between things, that palpable void.

But we also learned that our brain can be tricked into creating that sensation.

We can put on 3D glasses, look at a perfectly flat movie screen, and perceive that same palpable space.

We can look at a random dot stereogram and see a 3D square that literally isn't there in the image itself.

Right, the depth isn't on the screen, it's constructed in your head.

So if we can so easily trick the brain into creating a vivid sensation of space where there is none,

how do we know that the space we feel around us right now is fundamentally real?

How much of the distance between you and the wall is physical reality?

And how much is just a useful user interface that your neurons have created to keep you from bumping into things?

Is space a property of the universe?

Or is it just the brain's clever way of organizing sensory data?

That is the ultimate question, isn't it?

It might be that 3D space is just the desktop format for our brain's operating system.

A very useful illusion.

On that note, thank you so much for listening to this deep dive.

And a huge thank you to the Last Minute Lecture team for helping us unpack this really dense but fascinating chapter.

Keep your eyes open, preferably both of them.

We'll see you next time.