Intermittent Mechanism

Ian here—

Spending one week on sound in an Introduction to Film course can be a daunting task. So much vocabulary, and so many new issues to discuss, with only a class or two to dwell on them. What to do?

I like to take this week to introduce the concept of synesthesia—the “bleeding” of one sense into another that results in sensations in one sensory modality being interpreted as impressions in another. It’s a phenomenon that has been studied from the era of classical Greek philosophy up through modern neuroscience, and it has provided inspiration to artists for nearly as long. For understandable reasons, it was something that was on a lot of filmmaker’s minds during the transition to sound cinema. Turning to this topic allows us to rope in stalwarts of classical film theory such as Eisenstein and Vertov, and to freely intermingle experiments in feature filmmaking with more radical experiments in the avant-garde.

In this particular lesson, I focus on music. If one’s spending two days on sound, this leaves another class for sound effects, if one so desires.

Music and moving images: Time, space, and other dimensions

Moving images are two-dimensional, meaning that they have an x-axis and a y-axis. In addition to this, they also transform in time. Music is similarly time-based, but otherwise it possesses nothing like the two dimensions of images. Instead, it possesses pitch, volume, timbre, and a whole host of other qualities.

This raises a question: If a filmmaker wants to “translate” a cinematic image into a cinematic score (or vice versa), creating a synesthetic link between the two, how might they go about mapping these differing dimensions onto one another?

In his essay “Form and Content: Practice,” printed in The Film Sense, Sergei Eisenstein lays out the approach he used when collaborating with Sergei Prokofiev on Alexander Nevsky (USSR, 1938), using twelve shots from the “Battle on the Ice” sequence as his example. Here is Eisenstein’s graph of image-sound relations in shots VII-XII:

And here is the entire twelve-shot sequence:

Writing about image-score relationships in this sections, Eisenstein acknowledges that his methods were diverse. Sometimes, he was working off of little more than his own vague synesthetic feelings. For instance, he writes (and illustrates) of the musical cue accompanying shots VI and VII:

Here are just shots VI and VII, for reference. Pay attention to the score:

Do you hear, and see, what Eisenstein described? I give students plenty of time to discuss their own reactions (if, indeed, they have any at all!). Eisenstein admitting that he didn’t check his own impressions against those of others is a good acknowledgement of the fact that synesthetic responses can often be highly personal. (Personally, I have to admit that I don’t get the same impression from measure 10. The fact that the bass line here is actually ascending, rather than descending, kills any feeling of a heavy mass of sound “rolling down” for me.)

Elsewhere, Eisenstein doesn’t rely on vague personal impressions at all, but instead attempts to visually represent what is actually happening on the staff of the film’s written score notation. Eisenstein provides a handy graph of this type of correspondence in shot VII:

Here’s the shot in time, with the score actually playing:

Eisenstein is fairly confident that this technique works. Here are a few scattered quotes from “Form and Content: Practice,” where he lays out his methods of drawing the spectator’s attention from left to right (much as if they were reading a musical score), and defends the structural integrity of what he has put together:

A motionless picture exists spatially, that is, simultaneously, and neither its left, nor its right, nor its center can be thought of as occupying any order in time, while the musical staff contains a definite order moving in time. In the staff the left always signifies “before,” while the right signifies “after.”[i]

The art of plastic composition consists in leading the spectator’s attention through the exact path and with the exact sequence prescribed by the author of the composition.[ii]

In the twelve shots we are analyzing … this movement proceeds precisely from left to right through each of the twelve shots identically, and fully corresponds in its pictorial character with the character of the music’s movement.[iii]

Here, I open things up to comments. Do students think this section is as successful as Eisenstein claims it is?

Usually, the general response is “no.” Students are often befuddled by the connections Eisenstein is attempting to make in this section. It’s a valiant effort, perhaps—Eisenstein has surely put no small amount of thought into his process—but the results are less than consistent. Students who don’t know how to read music, especially, shed doubt on the synesthetic efficacy of this passage. They don’t know what they’re supposed to be looking for, and, as a result, the types of direction of attention Eisenstein was attempting falls completely flat.

This is a good chance to point out that although Eisenstein was extremely erudite, sometimes his erudition becomes a liability. Across his writings, he displays admirable knowledge of art history, comparative cultural studies, literature, and music (including the latter’s composition and notation). But sometimes he loses sight of the fact that other people don’t know those things, and so the “universal” gestures he hopes to hone in on are, in fact, less universal than he suspects. Here, in particular, we have an attempt to craft sound-image correlations that relay too much on a working knowledge of Western musical notation.

Other filmmakers have tried different approaches—perhaps, to greater success. Let’s take a look at some!

Alternatives to muscial notation

Let’s back up some, from the sound era into the silent era—and, in particular, to the interwar European avant-garde. In 1921, Walther Ruttmann premiered his landmark contribution to abstract animation and “visual music,” Lichtspiel Opus I. Featuring hand-coloring and a specially commissioned score from Max Butting, the film has long since become a mainstay in film history courses and overviews of avant-garde cinema. Let’s watch a few seconds:

Now, let’s get a few things out of the way. Opus I is a silent film. Butting’s score would have accompanied the film during most of its screenings, but the very precise synchronization between imagery and music that we are able to enjoy on modern visual copies is something that just didn’t exist at that point. Unlike Eisenstein, there was no belabored process of editing images to sound. Also unlike Eisenstein, Butting’s score was added to Opus I after its visuals were completed. Prokofiev’s score for Alexander Nevsky, according to Eisenstein, was at least partially written and recorded before the film was edited. So each film involved a very different crafting of image-sound relationships.

But beyond these basic differences, Ruttmann’s approach drastically strays from Eisenstein’s technique. Ruttmann eschews any reference to traditional musical notation. Rather, Opus I itself, at moments, looks as if it’s trying to create an entirely new mode of musical notation, one that is based around the real-time manipulation of shapes along two axes. Particularly notable is the way it handles the dynamics of attack and decay (in layman’s terms, the volume of the beginnings and endings of notes and phrases). At the 13-second mark of the above clip, a crescendo-decrescendo-crescendo (that is, a phrase where the strings gradually get louder, then softer, then gradually louder again) is represented by a blue rounded blob growing up from the bottom of the screen, and then shrinking again. At the 21-second mark, accented bursts of string are accompanied by knife-like protrusions in from the edge of the frame.

We’re very far from attempts to translate written musical notation into imagery here. In the case of the “big blue blob,” in particular, we’re actually closer to something like waveform representation of music. The waveform representation of sound is conventional to some degree, but because of its reference to the physical properties of mechanical waves, it is generally much more intuitive to “read” than traditional Western musical notation is. Just as an example, look at the waveform below. I’ll ask you what I ask my students: can you offer a guess on what the sound represented will sound like? Click play once you have an idea.

The drum track here has an additional effect applied to it—what’s known as a “gate” effect, which entirely erases all sound that dips below a certain decibel level. As a result, it is something of an ideal example for the principle of staccato, bursts of sound with short duration, and with sudden attacks and decays. You can visibly see this in the waveform, especially on the attack side of things. Do you see how each drum beat starts as a sudden burst of vertical noise? That’s staccato articulation, in visible form.

Okay, back to Opus I.  Let’s look at another really quick clip. Pay special attention to the swell of strings in the opening four seconds of the clip, accompanied by another blue rounded blob:

You’d be hard pressed to find a passage of music that is less like the staccato drums we heard just above. And, correspondingly, its waveform also looks quite different. Here, you can see that the crescendo that opens the phrase and decrescendo that closes it result in a visibly tapered waveform. It is smooth and rounded down: it starts small, gradually builds to a hump in the middle, and then tapers away.

Ruttmann and Butting didn’t create a film in which the visuals were an exact waveform representation of the score. But you can see some similar principles at work. I’ve overlaid the score’s waveform onto the film’s visuals in the following video, so that you can appreciate the way in which both visually represent “growing” and “shrinking” as the score’s dynamics do the same. (I’ve repeated this short section and re-mixed the visuals a bit, so that you get a variety of senses of how this works.)

Alright: Next, we fast forward 92 years, to 2013, for a third example. I hope, though, that you’ll keep Opus I in mind.

Gravity, Point of Audition, and Score as Sound Effect

Alfonso Cuarón’s 2013 film Gravity is a wonderful film to use as an example of the concept of point of audition—the sonic equivalent of point of view, which attempts to deposit spectators into the subjective auditory realm of a given character. The following clip is an especially good example of the concept in action, honing in on the sonic experience of Sandra Bullock’s character, Dr. Ryan Stone:

Cuarón and his sound design team set some very particular constraints for the soundtrack of Gravity. Following in the footsteps of 2001: A Space Odyssey (Stanley Kubrick, 1969), Gravity strenuously commits to the scientific fact that, as a mechanical wave, sound cannot propagate in the vacuum of space. All of the dialogue and sound effects that we hear in the film, then, fall into one of three categories: 1) radio chatter, 2) Stone’s own voice (which she would be able to hear given the air inside her helmet), or 3) the soft thud of objects that make direct physical contact with her suit, which itself acts as a propagation medium for sound waves.

This isn’t much of a palette for possible sounds, but Cuarón and company make the most of it. For instance, the above clip takes place after a section of the film in which we have only heard Stone’s voice, and her fellow astronaut Kowalski’s voice on the radio. As we enter her helmet, however (an effect made possible by the fact that nearly everything we see in the film beyond the actors’ faces is CGI), we transition to a much more literal interpretation of point of audition, suddenly privy to the specific reverb of her breathing and speech inside the constrained space of her helmet, as well as as subtle radio noise.

Adopting these strict scientific rules creates a conundrum for Gravity. How do you make intense action scenes, filled with devastating destruction, exciting without recourse to filmmaking’s usual suite of sound effects? This wasn’t a problem as much for 2001, as cold detachment was a key part of Stanley Kubrick’s directorial stamp, and anyway the film is much more of an art film-science fiction hybrid than an “action” film, per se. But Gravity is very much an action film. How does Cuarón maintain the type of sensory bombardment so typical of 21st-century action cinema while still obeying his self-imposed restrictions on sound effects?

Well, he uses composer Steven Price’s score to cheat. Take a look at and a listen to the following clip:

As the space shuttle arm that Dr. Stone is attached to whips toward us in its terrifyingly fast arc, there is a distinctive screaming sound on the soundtrack. This is not, in fact, a sound effect. It is an element of Price’s score. It’s clear, though, that Cuarón is using Price’s score as a pseudo-sound effect, closely linking auditory and visual effects to attach a sense of urgency of something that would actually be playing out in silence. The dopler effect-like whizzing of Price’s unusual electronic instrumentation here makes the distinction between score and sound effect seem academic.

The Sound of Angular Momentum

Superstar astrophysicist Neil deGrasse Tyson went couldn’t stay away from Twitter following Gravity‘s release. Although he copped to “enjoying the film very much,” he also poked holes in the film’s plot, visual effects, and even its marketing. One bon mot he dropped at the time was this:

As Tyson later explained to non-physicists, angular momentum is “the tendency, once set rotating, to keep rotating, unless another force acts to slow or stop it.” And Tyson’s right: many of the film’s most memorable moments, and most alarming visuals, center on the stomach-churning spin that Dr. Stone sets off in after the shuttle arm she’s attached to is struck by debris and severed.

But this theme of spinning out of control while hurtling away from Earth isn’t just an arresting visual motif of the film. I also argue that it’s a crucial component of the film’s score. Here’s a few seconds from “Debris,” a track from the film’s soundtrack album that plays over the scene of the initial impact, and Stone’s subsequent uncontrolled spin:

Here, I do something that might embarrass certain students: I ask them to sing the main motif of “Debris.” Go ahead and do it yourself. Really! WIEERMP-WOMP. WIEERMP-WOMP.

Pay attention to what your lips are doing as you make those noises. What does it feel like? To me, it feels as if I’m pulling two golf balls out of my mouth, in rapid succession. My lips form a tight circle, open up, close again, open again, close again. WIEERMP-WOMP. WIEERMP-WOMP. I don’t think we should discount this bodily sensation. There is something distinctly round about the main “Debris” theme.

Don’t believe me? Here is a waveform:

Okay, so it’s not as if the audio looks like two completely separate circles. But you’ve definitely got two “mounds” happening, with fade-in attacks and fade-out decays that give the waveform a distinctive “rounded” appearance.

Just for fun, let’s see what flipping the x and y axis does, and setting the waveform itself in motion, rather than just sending a mobile cursor through it:

Does it remind you of anything?

No? Okay. Let’s try this. In the following clip, pay attention to Earth, both as it shows up behind Stone as she experiences unbroken angular momentum, and how it reflects in her helmet. This image is all curves and circles, as the outline of the sphere of our planet slides behind and is reflected by the glass sphere keeping Stone alive. I’ve added an overlay here to aid in grasping how the music fits in with all of this.

In the end, synesthesia is in the eye of the beholder. (Or, rather, it’s in whatever two or more sensory organs of the beholder’s that are being linked.) To me, though, the overlay of the waveform clinches it: Cuarón and Price were attempting to construct a synesthetic link, attempting to create “circular” or “spherical” musical motifs to increase our bodily reaction to the spinning views of spheres onscreen. And, arguably, they were more effective than Eisenstein and Prokofiev were in establishing this sort of synesthetic link—in large part, I think, because they forewent any appeal to traditional musical location, and instead focused more on the feel of sound, a “feel” that can be at least partially represented with recourse to a waveform.

It’s no accident, I think, that the theme Price settled on was the same sort of crescendo-decrescendo burst that accompanied the “blue rounded blob” object in Opus I. He was, after all, trying to sonically represent a different sort of big, blue, round object.

[i]. Eisenstein, Sergei. “Form and Content: Practice.” In The Film Sense. Edited and Translated by Jay Leyda. New York: Harcourt Brace & Company, 1942. Pp 189-190.

[ii]. Eisenstein, “Form and Content: Practice,” pg 190

[iii]. Eisenstein, “Form and Content: Practice,” pg 194