Blind Audio Listening Test - Can we really hear subtle differences?

The results from and discussion of a blind listening test I conducted at the 2022 StereoNet Melbourne HiFi Show. Thank you to @StereoNET Australia for creating such a great event and for everyone who came to visit and participate.

00:00 Introduction
02:09 Why I ran this test
03:18 Blind listening test setup
04:25 Instructions to listeners
05:26 Results
06:52 Additional data
09:32 Listening preferences
14:42 Listener predictions
15:59 Quick recap of results
16:42 Reveal video

SUPPORT THE CHANNEL
Or put one of these links in your bookmarks for all of your future purchases of ANYTHING:

PASSION FOR SOUND RECOMMENDS
Check out this link for an ever-evolving list of the products I recommend, including links to make your research and purchasing easier!

(Affiliate links help to support the channel - thank you!)

Reviews filmed using:

#audiophile #audioreview #blindtest

[Foobymaster]

I would be curious to see if, leaving no change at all, people would say they hear a difference.

[SuperReview]

Very surprising results! Love the test.

As a cable skeptic, I'm surprised that 80%+ of listeners reported hearing a difference, BUT I can rationalize it away as... listeners are motivated to say that they hear a difference as a sort of hobbyist cred. In other words, if I answer no, it must mean my ears are not very good, ergo I am motivated to answer yes, I heard a difference.

The second data point about preference seems to back this up. When asked which one was better, the answers were 50/50, a coin toss. So while listeners could placebo themselves into hearing a difference, they couldn't consistently identify the "better" one. In a world where there _is_ one "better" one, this would be pretty damning, but as you rightly point out, listener preference confounds that rationale. In other words, a 50/50 result, given a random distribution of listener preferences, is certainly plausible if indeed the setups sound different.

At this point, game theoretically we're at... it's still plausible the difference is "real, " but also just as likely it is placebo.

The third level of data is the strongest support for the "reality" of different sounds of the setups: People used similar words to describe the each setup, and (presumably from here) without a high degree of overlap between Setup A and B. This is a result that could not be easily explained by placebo rationale and is indeed the data point that forces me as a skeptic to sit back... and open my mind a bit. This data point adds credibility to the prior two.

I am not ready to replace USB cables in my setup, but I really appreciate the effort you've put into this test -- it's well-designed and something I'd love to see more of!

[andyreichert]

From a process standpoint, I would swap the cables part way through the day to make sure that there wasn't other equipment variance that could account for the differences. It would also be interesting if there was a control setup that matched one of the other setups and we could see how often people could identify the identical systems.

[Seconds]

First, Test still has an issue because you need an X example. ABX. Second, different headphones were used which could have had slight differences in drivers and therefore sound. Also, you told people before you started that there’s is a variation.

You shouldn’t have mentioned a variation. Then, ask them whether there was or wasn’t a difference without them knowing that there was before the test started. You created bias right from the start. With ABX, you could have had a variation and then asked the participants which one was different and which 2 were the same.

[PassionforSound]

I've been looking at some of the comments and want to address a few things in one go here to save me typing it over and over:
1) the variation of performance between two products (headphones or DAC/amp) from reputable products would not produce the types of results shown in this test unless one was faulty (which they were not)
2) fit and positioning of a headphone can alter its sound slightly by shifting the size and exact frequency of some spikes in treble, etc. but not by a huge margin and, more importantly, it will not influence the types of factors people cited in the test (e.g. presentation of detail, spatial qualities and dynamics)
3) swapping cables on these devices takes time because the playback software has to be stopped and restarted to prevent crashes. This introduces a very challenging variable which is the reliance on medium-to-long term auditory memory and that is highly innacurate
4) given the budget and space, I would have loved to have an identical setup with no variables as a control to see the percentage of people who heard differences between two identical setups. I hope to build this in to future tests.
5) participants were told that there was a variable in the setups and asked to decide if it made a difference. That can lead to two thought processes/expectations - the expectation of a difference AND the expectation of no difference so some respondents may have also been biased towards a "no difference" response.

[TheOriginalBagelstein]

My dude, if you give people an A and B setup tell them there might or might not be a difference, most people will ABSOLUTELY tell you there is indeed a difference. If you wanted to do this with actual scientific methodology you would have a setup with either A + B + Control (Same as B), and determine if people could detect a difference from A vs. Control against B vs. Control. Alternatively you Could do 2 setups, A + B with a difference, or a control A + B without difference and determine the first set has a larger amount of people claiming a difference over the control. In short, you didn't use a control and you introduced bias by implying there is likely a difference.

[matthewweflen]

As far as this test, I think there would best be a third system, to mirror an "ABX" blind test. That way you can get at "which of these is different" and really drill into whether people who cite differences are doing so because they expect it or because they actually hear it.

For future tests, I would love to see a volume matched test with different amplifiers as the variable, specifically a high powered vs. a lower powered amp. This would investigate claims that "some headphones need lots of power to be 'driven properly.'" It makes no sense to me from a physics standpoint why an amplifier with more headroom would "drive a power hungry planar" better *at the same volume* as a lower powered amp with less headroom.

[shreddherring]

I'd just like to say that I appreciate the efforts to make this a controlled test with as little bias as possible, hats off Sir!

[AlexKouznetsov]

I had this revelation about 2 years ago, while testing a DAC that I was planing to buy, I was suggested to try out a "Good" usb cable, I was curtain that I won't hear any difference, but I did, then I asked to give me a blind test. 10 tests with random cable each time, I got 10/10 right. I brought the cable strait away and continued to test the DAC.

[hendrikoffermans]

Yes, shocking for me.. until the end I was convinced that the difference must be the power supply.. I always thought usb cables make the least difference but here we are... it is time to hear this for myself..

[woopygoman]

I'm not a high-end cable user but I have always kept an open mind when it comes to cables and I'm honestly not surprised.

[martinhiggins]

The thing which has made a consistent improvement to my system over the years has been a good malt whisky. 🙂

[drewmorg.]

I have driven myself crazy with A-B testing all the new (expensive to me) gear I just bought. Going from gaming headsets traditionally to Edition XS and a $300 Schiit Stack is a leap for most of us. I got each item in isolation and tested for a week and kept adding finally the amp and then the Modi dac. All I can confirm is yes it all does sound very good and each piece added some kind of additional level of importance and well roundedness to the equation. For those on the fence, who are really tight on the money and are new to this game, my simple advice is once you get any kind of higher ohm or low sensitivity headphone you WILL need at least an amp. I think the best bang for buck here so you get the DAC too is Fiio K5 it seems like. I think I would have been happy with that over my Schiit stack but whatever, my XS eats all the energy you throw at it. I will say the DAC added a presence particularly in sound stage and the ability for my amp not to work so hard since its pushing 2v instead of 1v like most other onboard DAC's or XLR interfaces. The difference is not at all night and day for me. I still keep plugging my XS into my $40 UMC22 interface to see if I can tell the difference and it's incredibly subtle. I just know that the Modi/Magni sounds a bit more spacious and "interesting", hard to put my finger on it. Small, small differences. The biggest difference is that the UMC22 cannot drive any of my headphones to reasonable high volumes without distorting so this is where you are forced to make the leap into something with more power.

[michaelshulman]

This is a fascinating test; thanks for doing it. I can maybe see a couple of possible improvements: (1) Have 3 configurations, 2 of them with cable A, and 1 of them with cable B. See how many people can identify the two configurations that are the same. (2) Swap which USB port each cable is plugged into, halfway through the test. This could identify if there is a random difference in the amount of noise leaking into one USB port, and the amount of noise leaking into the other USB port.

[AccuphaseMan]

I was lucky enough to be there and I'm honestly impressed. As for the 50/50 results, I'm not surprised, they both sounded very good but offered different presentations.

[itsabodh]

It’s interested that if people can hear a difference in such a loud environment then it should be an obvious difference when listening at home and there shouldn’t be any debate on it. Which can only mean one thing : most of audiophiles(arguing types) don’t have any hands on experience with the product they are talking/writing about.

[gedfield]

If you told people that was a variation before the test, why are you surprised the 85% can hear a difference?. Particularly with a selective group of people at a Hi-Fi show where nobody wants to admit to being deaf to nuances. Your description of experienced listeners and non-experienced listeners was a little too loaded for my liking. These are my initial thoughts and undoubtedly as heavily biased as anybody else's. I really enjoyed the video and really appreciate your work, it's a fascinating subject. Thanks.

[taidee]

I like this, after all a blind test is what is necessary for finding out if there is a difference that a certain piece of equipment makes.

[RyzaJ]

As a participant, there was a discernible difference; I'm just clarifying this here because there are a lot of...opinions.

[davidm]

My hypothesis is the Biggest difference is variation in headphones.