Why AI is Doomed to Fail the Musical Turing Test

AI will get vibed at the jam session, and there's nothing that it can about it.

0:00 Intro
3:54 Part I - Musical Turing Tests
10:56 Part II - Thinking Like a Human
20:17 Part III - "Not music"

Sources:

Valerio Velardo's channel on AI music.

(⌐■_■)

⦿ Adam Neely T-shirts! ⦿

⦿ SUPPORT ME ON PATREON ⦿

⦿ FOLLOW ME ON THE INTERNETS ⦿

⦿ Check out some of my music ⦿

Peace,
Adam

[DeGuerre]

I am a computer scientist, and the category error problem constantly annoys me. We find a problem that requires a lot of intelligence in humans, like playing chess or go at a grandmaster level, and declare that AI is therefore "intelligent".

For some reason, it's only AI that we use this kind of language about. The best human weightlifter is easily outcompeted by a small forklift, but we don't call the forklift "strong". The best human sprinter is outcompeted by a locomotive, but we don't call the train "fit". Hell, computers have been beating humans at mental arithmetic for ages, and that's even a marker of human intelligence.

To quote the great computer scientist Edsger Dijkstra, "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."

One of the under-appreciated aspects of the Turing test is that it was an activity that humans should find easy, not something like playing chess which humans find difficult. It's these "easy for us" problems where AI tends to fail, partly because they are the problems that machines find very hard, and partly because you can't get money to solve unspectacular problems.

I want a machine to do things that I find easy but tedious, like cleaning my bathroom.

[pepkin]

0:00 "What would it take for a machine to jam?"
Very little, actually, my printer jams all the time.

[Bleepbleepblorbus]

Imagine having to prove you're not a robot by playing free bird on a fucking hurdy gurdy

[ZaphodBeeblebrox]

I really very much like the concept of "musicking", it reminds me of a quote from someone I very much respect a few years back, they were talking about NFTs and the commoditisation of the visual medium, but this hit me so much I don't think I'll ever forget it:

"Art is a verb. It's a process. It's an act of communion. What hangs on the wall is a fucking collectible. What you and the artist communicate across centuries is the art."

Obviously Small's book and concept well predate this quote, but it honestly changed my perspective on why art and music mean so much to me, and what's truly valuable and meaningful to me as someone who appreciates these things.

[JacobGeller]

Adam!! Drive-by praise during the sponsor read, I was not prepared!

[Pomplamoose]

This is so interesting. And such an honor to be featured! I feel like you and Jack could nerd out for days on AI (and music).

[theomnitorium]

I gotta say: I'm glad no human being was forced to record those Red Lobster lines...

[mariellestoelinga]

Professor of computer science & amateur musician from the Netherlands here. First, thanks for all the thorough and well-researched videos, always a joy to watch.

"AI cannot do X" arguments are, in my opinion, always tricky: AI has surprised all of us, even researchers in the field, with its incredible progress. In particular, I am not convinced about the interaction argument. Reinforcement learning is branch of AI that is specifically tailored to interacting agents learning behavior in a dynamic environment.

Amazing progress by companies like Boston Dynamics has enabled robots run and do back flips. I see no reason why in several years this technology would not be able AIs to play in jam sessions. Sure, there are challenges, like there were in AI before. So the real question is how should we musicians, writers, scientists and all other relate ourselves to AI? This video makes a good contribution to that debate and the various aspects.

[bardofhighrenown]

I think the Red Lobster thing isn't proof that AI is now good at producing music, but rather an indictment of how bad music in advertising is.

[azaleajanemusic]

Ever heard of David Cope? He was creating new AI-generated classical pieces in various composers' styles over 20 years ago with a program he wrote himself in Common LISP. I took a course with him called "Artificial Intelligence in Music" in like 2005, though I don't remember much except learning a bunch of LISP.

[jaspertucker]

Seriously great video Adam, I'm a grad student in computer science and musician/composer doing my thesis on generative music right now. I think an important thing to bring up too is the difference between generative systems being built today and those built by the original pioneers of 'generative music' like Brian Eno and David Cope from the 1960s-1980s.

I'd argue that a lot of those older processes and systems thought more 'human-like, ' in that Eno, Cope and others tried to make systems that extended their own compositional thought process (Music for Airports from Eno, It's Gonna Rain from Steve Reich, EMI, etc...). But today, that compositional curiosity is gone, and companies like Suno, Meta, Google, and others are looking at music like the next 'check box' for AI and trying to achieve greater technical prowess.

[janmelantu]

Things that will stop AI at jam sessions: “the usual key”

[Buckleupbucko]

I’m about to start randomly yelling out “BLUES IN E FLAT” At the jam from now on.

I’m sure everyone will love that.

[Kuukunen]

I feel like I need to write a book or something at this point, with how many easily countered AI takes there are.

Usually I do find Adam's videos very insightful. This time the main argument seems to be "AI is doomed to fail the musical Turing test, because the companies just don't care enough." The other things mentioned seem to be things that AI is already doing, like making mistakes and forgetting things. There's also already been instrument playing robots for decades and there's no reason to not expect them to not get rapidly better with ML. (Check what Nvidia has been doing with robots for example.)

There's also the weird directive/output division. I think the point wasn't about the technical aspects, because you can easily train an AI that takes as input all the sound it has heard and produced thus far and then ten times per second produces the next 100ms of output, thus transforming the problem into an output problem.

Going back to the business case, I would say there are three counter points. First, I could easily see some product potential in an AI music teacher/jam partner or something similar. Or even an AI based autotune that not only fixes vocals, but also instruments. Second, companies have already spent a ton of effort on non-profitable endeavours like how five years ago OpenAI made Dota 2 bots just because the problem was interesting enough.

Thirdly, and most importantly, you don't have to train an entire massive model. Companies are working on multi-modal models like the just announced GPT-4o that are much more flexible than purely text-based models. And once you have trained the base model, you can fine tune it for different purposes. If you want to see examples, check how making Stable Diffusion checkpoints works. If Hallmark wants to make a model that generates card art, they don't have to spend skrillion watts to train a brand new model, they can just show the model couple of hundred examples and in a few hours you have a new model that specializes in card art. So you could take some jam sessions, separate one instrument and fine tune a model on that.

Lastly, I'm as annoyed as anyone else when creative people lose jobs and "prompt engineers" call themselves "artists". But it feels like 99% of people who feel like they simply must give their opinion cannot separate the "what should be done?" question from the "what can be done?" question, even though they have absolutely nothing to do with each other. Everyone has such high-strung emotions about the subject it feels like every take I see is either "AI will solve every single problem in a year, just you see!" or "AI is crap, it's always been crap and always will be crap! lol look at this stupid AI that can't even draw fingers or text and it never will" (they can now) And to be honest, this video seems to fall in the latter category.

The way I look at visual AI art for example is comparing it to a very technically talented idiot savant dude in Fiverr from some fourth world country willing to work for peanuts. Let's call him Albert, or Al for short. It might take a lot of effort and iteration to make him understand what I actually want, but no matter how detailed of a prompt I write, I wouldn't call myself an "artist". But if I for example make a simplified sketch of a character and ask Al to finish it based on my detailed description, I might credit myself for the character concept. Or what if I ask Al make a bunch of different images and then I Photoshop them together, along with some custom edits. At some point it starts approaching some form artistic collaboration from my side as well. Like... can photo bashing ever be art? I think so.

I don't know why I'm spending so much time writing this, though. I didn't write this in the first nanosecond of the video being out, so the chances more than a few people will ever read this or care are quite minimal. *sigh*

[ducksies]

Transformers are really good at one thing and that's pretending to be something. I'm convinced that if ample training data of jam sessions, with specific instruments removed and added to the mix (using simple music editing programs) is provided to an AI, we could have a real time musical jam AI, similar to how we have real time voice changing AI now.

[meganechan]

I always forget that the guy from pomplamoose is the one who invented patreon, and every time I'm reminded of it I'm blown away all over again

[rueburch]

As a researcher on social interaction (including social and distributed cognition), I love that you did a deep dive into cognition beyond the computational model that is so prevalent in everyone's imagination at this point. I hope folks learned something from it.

(I'm also a little jealous that you explained so clearly in a few minutes what took me years of PhD study to understand. Bravo!)

[MatthewGraham-xiio]

I am a researcher at the oxford university robotics institute, I wrote my thesis on a robotic piano that composes its own music to pass a turing test. It is an interesting video however there are lots of researchers solving these problems purely for interest. My piano can play and compose music to a level where people couldn't distinguish it from a human. Also the nature of the transformer algorithm underlining it allowed it to write harmonies to tunes people played on the piano live. It would be interesting to talk further however I doubt this comment will be found!

[rojopantalones]

I miss taking part in jam sessions. When I was stationed out in California, I had a bunch of musician friends, and we'd all rent a practice booth at the rec center, cram ourselves inside however we could, and just play. There wasn't a ton of vocal communication, only really asking what sort of vibe we were going for, never recorded anything, and a session would just start with one person just... Playing. Usually it was me as the drummer giving the rest of the group a beat, sometimes it was one of the guitarists (of which there were three) showing off a riff they liked. Music, itself, is a language that we spoke to one another in. Despite the rather eclectic collection of instruments (2 electric guitars, one acoustic guitar, an upright bass, a trumpet, a ukulele, a clarinet, and drums), it all worked cause we knew what things sound like and how they worked together. Yes, that is too many musicians to shove into a practice room, but we got by by having three people sitting on the piano in different ways (one on the bench, one on the key cover, and one on top), with the rest of the people stuck standing around where they could find room.
Can't play drums much anymore due to chronic pain in my knees and lower back, but that little jam band I was part of will live forever in my heart cause it was something we did and made together. I guess the secret to making it all work was that we had something as the root, whether that be a guitar or the drums or the bass, and all of us taking the time to either agree on what it should sound like or disagreeing through music by pushing against what the root was trying to establish, forcing it to change to fit the new sound everyone else was creating. The specific arrangement limited what sorts of vibes we could create, as it's not the easiest task to make a ukulele sound dark or a trumpet sound atmospheric, so we always ended up with a bit of a brighter sound.
I do agree that that's something an AI model can't recreate: The process by which people create. The how and why, merely producing an attempt at a finished product. The creative process is just as, if not more important than the end product, despite most people only really seeing the end product.
However, when one looks at a piece of art, we wonder what it means, what was going through the artist's mind as they created, their reason for making it, the curiosity and analytical parts of the mind trying to break it down and make sense of it. We have always cared about the journey more than the destination.

[ScienceEtonnante]

Hi Adam, this is one of the best "AI will never be able to do <X>" argument I've ever seen !

For context, my current job *is* about GenAI (for instance I know exactly the ElevenLabs settings you used for Xenakis), and I started doing machine learning 20 years ago. So I've heard my share of silly "AI will never be able to do <X>" arguments, that proved to be wrong in the end. I'm also an amateur musician who knows what jamming is, and the very subtle body language communication it can involve. I was very skeptical at the beginning of the video, but this is maybe the first time I hear such a compelling argument against AI taking the place of humans.

Two thoughts to slightly moderate your argument :

* I do think that machines will be at some point able to pass your musical Turing test, but it might require a big paradigm shift in architecture, training procedure, etc. (a shift that will not likely happen soon for the reasons you describe around economic incentives).

* In a sense I feel that that Turing test is a bit unfair in the sense that it involves doing a thing well "with humans", and not "between AIs". It is like saying "Oh but humans will never pass the spider test which consists in being able to get offsprings with spiders". I'm sure that AIs will be able to jam "between them" because they will find digital ways to replace the subtle body language and so on. (It reminds me the "Hanabi" card game test : it is a cooperative game that AI can do very well when they are playing together, but do badly if they try to play with humans.)

Keep up the good work !