Mastodon

Your Eyes, Ears, and Brain Don’t Agree

Posted by Matt Birchler
— 2 min read

I was watching this excellent video from Ian Hink of Easy Allies and there are tons of good tip in here, but one thing really stood out to me around the 15 minute mark.

Tricking Your Brain

Ian talks about cutting video to music. As you likely know already, you want to cut on the beat to make the video and audio feel connected. That said, you can't just go into the timeline and put your cuts exactly on the beat you see in the audio waveform. No, the computer isn't wrong about where the beat is, your brain doesn't actually want to see the cut on the beat. Insted, you usually (but not always, because brains are weird) want to make your video cut one or two frames before the beat. This is technically out of sync, but your brain will make them feel in sync.

Why is this? I genuinely don't know. Maybe it's your brain anticipating the beat, but not knowing the next visual on screen, so it needs the extra 30-some miliseconds to process that a new image in there when the beat actually happens. That's just a guess, but the important thing is that it works, so us video editors can carry on using this trick that works, while scientists can figure out why it works.

Compression

Kind of unrelated, but also on my mind when thinking about this phenomenon, is the work that went into creating the MP3 audio format. This was covered pretty well in one of my favorite books last year, How Music Got Free, and the part that really got me was the scientific investigations into how human brains process audio signals.

A common thing music afficianados like to look at are waveforms. They'll show you the waveforms from the same song in MP3 format and in uncompressed audio file and will show how the MP3 loses so much complexity and detail from the uncompressed file. That's scientific evidence that uncompressed audio sounds way better than compressed MP3s.

While objectively the uncompressed file has more data, the interesting thing to me is that when taking in that data (aka listening to music), your brain is compressing that sound, so while you're seeing the added data in the waveform on your computer screen, your human brain is removing a good chunk of that data. Much of the research that went into creating the MP3 was understanding how our brains compress audio signals, and making the MP3 compression work similarly enough to that such that we would not notice the difference.

Super simplified version: our brains compress very complex audio anyway, so we're not even experiencing all that sweet, sweet AIFF goodness anyway.

Brains, Man

These are simple examples of our brains falling short of the pure rational operators we like to fancy them to be. Humans are not purely rational creatures, and the deeper into this you go, the less and less rational we seem. Maybe in an ironic twist, our brains are very good at convincing itself that its own irrational choices are rational.

Let me be clear, I am not saying that "truth is an illusion and who can say what is true or not," as I often see from some folks who go too far with this line of thinking. I just think that understanding some of the fundamental ways that our brains manipulate us into thinking some things are one way when in fact they are another, is essential to understanding humanity beyond film editing tricks and compressed audio.