A Synesthetic Experience (Documentation)

A Synesthetic Experience is a project to design to capture the sensations of synesthesia. This goal is achieved through the use of 3D animation set to classical music. The equipment used for this project are a stereoscopic 3D viewer and wireless headphones allowing for a more personal experience for the viewer.

Inspiration and Research
The inspiration for this projected started with the synesthetic painter Wassily Kandinsky. While painting to the music he heard, Kandinsky painted in a vibrant and geometric style that conveyed a lot of emotion. A simplified form of Kandinsky's painting style would be the guide the overall direction of the animation.

Wagner's Lohengrin, which had stirred Kandinsky to devote his life to art, had convinced him of the emotional powers of music."
‑Excerpted from "Kandinsky: Compositions", by Magdalena Dabrowski

"I saw all my colors in my mind; they stood before my eyes. Wild, almost crazy lines were sketched in front of me. I did not dare use the expression that Wagner had painted 'my hour' musically."
-- Wassily Kandinsky

The music implemented for the animation is Richard Wagner's Lohengrin: Prelude to Act I.
In order to realize the goal of experiencing virtual synesthesia, the project needed a structure to link the music to the animation.
Alexander Scaibin, a composer who Kandinsky respected, produced a Color-Music theory in which he linked the musical notes on a scale to corresponding colors on a color wheel.

The Animation Process
To begin the animation process I listened to the music and created the initial storyboard based on my interpretation of the music. I divided the music into three components: Horns (represented by circles), Strings represented by lines, and bass representing the backdrop of the scene.
Then, I analyzed the sheet music and determined the timing of the notes played in the audio track used for the animation.
Next, I animated the colored shapes to their corresponding notes in the music.
Lastly, I rendered two versions of the animation, one for the left and one for the right eye. I then rendered both views into a single video.

Equipment
My initial designs for the equipment involved the use of a head mounted display in conjunction with wireless headphones. Due to the fact that the head mounted display available had very low-resolution screens, I opted to try an alternative method. I then switched to a Hasbro's MY3D viewer. This device would allow me to use an iPhone 4 as a stereoscopic 3D display screen. With the iPhones Retina Display I could use a screen that contained a pixel density of 326 pixels per inch, allowing for a much clearer viewing experience.
User Experience
Most users were very impressed with the use of 3D and animation in combination with music. Some viewers even exclaimed that they were very pleased with design to make the project a personal experience. One viewer did point out that they were not satisfied with how the 3D clipping was handled in the animation.

Future Plans
Given more time and funding I would like to experiment more with the use of the 3D effect in the animation. I would also like to use more representational shapes showing the other the instruments featured in the music. Lastly, I would want to be able to modify my 3D viewer to prevent light leaking, therefore creating a more immersive environment.

With the amount of positive feed back that I received from my audience, I can say that my project achieved it's goal to emulate a personal and virtual synesthetic experience.

[cutiepie]

I just cry when I watch the color I don't know why

[LoganJinks]

I do not have synesthesia. I was inspired by the artist Wassily Kandinsky. If you watch the video, I feature one of his composition that helped in spire the shapes. I then used the Color-Music theory developed by Alexander Scraibin to determine the the correct corresponding colors to each note.

[sianaa.degitz]

it's awesome what you did. with synesthesia, the colors and shapes blend and merge and dance and shift form. i love how youre able to bring this to life though. amazing!

[nachtorchis]

I see 3d fractal like, mandala like structures in flashing and changing colors. I would love to program it like you did. Must be possible.

[asmrgem]

I really wish I had synesthesia. But there's no way to learn it because someone is born with it.

[Oxyaena]

This is really cool! It's not really anything like having musical synesthesia, but it's a very interesting form of art! I love it! It must have been a lot of work to simulate this! For a non-synesthete, you did a great job of grasping what it might be like. Synesthesia is less specific colors, and more patterns and shades blended together smoothly, but this is a cool way to get the general idea across. Nice work, man.