I asked AI to make a Music Video... the results are trippy

In this video, I utilized artificial intelligence to generate an animated music video for the song Canvas by Resonate. This tool allows anyone to generate beautiful images using only text as the input. My question was, what if I used song lyrics as input to the AI, can I make perfect music synchronized videos automatically with the push of a button? Let me know how you think the AI did in this visual interpretation of the song.

After getting caught up in the excitement around DALL·E2 (latest and greatest AI system, it's INSANE), I searched for any way I could use similar image generation for music synchronization. Since DALL·E2 is not available to the public yet, my search led me to VQGAN + CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training), before settling more specifically on Disco Diffusion V5.2 Turbo. If you don't know what any of these words or acronyms mean, don't worry, I was just as confused when I first started learning about this technology. I believe we're reaching a turning point where entire industries are about to shift in reaction to this new process (which is essentially magic!).

Important note:
While this AI is impressive, it still required additional input beyond just the song lyrics to achieve the music video I was looking for. For example, I added keyframes for camera motion throughout the generated world. These keyframes were manually synchronized to the beat by me. I also specified changes to the art style at different moments of the song. Since many of the lyrics are quite non-specific, even a human illustrator would have a hard time making visual representations. To make the lyrics more digestible by the AI, I sometimes modified the phrase to be more coherent, such as specifying a setting or atmosphere.

This was my first time working with DDV5, and I'm very happy with the results! There were many times where my jaw dropped upon seeing what the AI came up with. I haven't felt this sense of wonder from technology since I first experienced a HD videogame as a child.

If you would like to learn more about how this video was made, try this yourself, or ask me any questions, I'll post a more detailed explanation of how to get started on Patreon (link below). The post is free to the public, no need to pay. If you do want to support me and become a member that would be much appreciated, you'll also automatically be entered into the end screen minigames where you earn points on each video and move up the leaderboard!

My social media:

While Disco Diffusion is based on the contributions of many, show some love to the two most prominent contributors:

Music:

[danielo]

The seamless transitions from realistic images, sketches, and digital art are mind-boggling

[sakuna]

This looks like the things you'd see in a dream. Where everything doesn't makes sense but they just fit together nicely.

[bayfourdu]

came back here after 2 years because I realized that this is where AI peaked. this may look “more ai generated” than a lot of stuff being created these days, but this isn’t trying to hide what it is and it’s so much better for it. honestly I wish video AI never improved past here.

[JesusofNazareth.]

It's amazing to think that if this came out just a few years ago before AI art it would win like ever single award.

[xereeto]

This is absolutely incredible, but the thing that stood out to me was the AI taking the strings of text that you presumably overlaid, and turning them into words relating to art. I definitely didn't expect that.

[timbomb]

I love how my brain is constantly trying to identify objects and understand the 3d space but can't hold on to a single perception for more that a couple of seconds.

[hugojj]

I just thought that maybe, the reason our reaction to this kind of art is so strange, is because unlike humans, maybe this AI doesn't have a point of focus like us, So unlike human art or music videos, they're designed with the same understanding of drawing our attention to particular parts. But with this art, it's similar to meditation or psychedelics because you're in an open or free state where there's no focal point because you are still and free with no intention or in psychedelics case, ability to look at something with out it changing to quick to focus on, where you're kind of forced to go with. Because the AI's in the same state.

[alkhmyst]

This is really well done. I really liked how extreme closeups marked the transitions between the different scenes, rather than using fades, dissolves, or cuts. It gives a feeling that the viewer is traveling along a road, instead of passively sitting and watching.

[goldcherries]

I feel like this video really reflects how our own dreams are built, taking pieces of words/thoughts and generating them into images that are an abstraction of them, still somewhat recognizable, but not fully graspable.

[lurker]

The best representation of "Don't let others know your next move.".

The AI absolutely nailed it, dang im amazed.

[esckey]

IT feels like those AI generated images where you think you know whats going on then you look closer than you realise its not what it seems

[elidah]

Played in x 2 speed, this video actually represents really well how it was inside my brain before I got ADHD medication; just a constant stream of "pictures" or visual thoughts and "verbal" thought that came and went, involuntarily, so quickly that you don't really have the time to properly "see" the thought before it is lost, so everything kind of morphs tougher to a meaningless soup haha. And you need to focus really hard to get the important thoughts through the soup without them getting lost haha.

[anthdimensionalbeing]

the style of this is like something between fantasy, cyberpunk, and paintings. its really interesting to see new landscapes appear out of the infinitely zooming camera view.

[CatchThesePaws]

I don’t think AI art is going to destroy the need of artists as much as it will create more artists by giving them the tools to make beautiful things. I already use generated images to inspire my art!

[dtniland]

1:36 is definitely my favorite part, just the way the camera angle is turning and zooming in and changing, and how the music pairs with the visuals... incredible

[geniusworldlive]

This is EXACTLY how dreams look! Which is why it’s often difficult to remember and explain them

[TheRunningChopsticks]

As a 3D artist, Graphic Designer, Painter, and draftsman seeing this just reminds me…

It’s an incredible age to be alive.

[ENDITFAUST]

I really like how at 4:00 the lyrics on the video say 'it's quite complex' but then it completely morphs to 'art's quite complex', such a cool touch

[mikeyvee]

The art of this is creation and how it follows a pattern that is natural and influenced. Like Earth, Humans, Creatures witnessing and understanding the beauty of nature around them.Then the AI clevery shows its appreciation for the video towards the end of it. Very heart warming stuff.

[grandpadavid]

Awesome! I can't help but feel a bit disoriented after watching this. Almost like a small flashback from past entheogen use. Trippy AF!!