Why you can't trust audio measurements

In this video I show various ways in which measurements can be unreliable, changed, misleading, and why you should always take them with a pinch of salt.
Quite a long one but hopefully will be interesting!

0:00 - Intro
1:56 - Disclaimers
3:08 - Measurements are not exact
4:32 - Measurements can be altered by test configuration
7:21 - Unweighted vs A-Weighted
8:22 - ADC/Analyzer distortion adds to the result
9:30 - Amps must be fed a clean signal to test properly
10:42 - How do you know how the analyzer was configured?
11:40 - Amps: Loads
13:01 - Amps: Measuring at unrealistic levels
13:31 - Amps: Volume control makes a big difference
15:06 - DACs: Standard 4v outputs vs 'Hot' outputs
16:04 - DACs: Noise from PC
17:29 - DACs: Internal settings
18:45 - DACs: Jitter from digital source
19:35 - DACs: Sample rate affects performance
20:31 - Manufacturers: SINAD doesn't tell the whole story
21:30 - Manufacturers: How they optimise for test conditions
23:44 - Why focusing on SINAD can be a problem
27:29 - High SINAD does not mean 'transparent'
27:45 - Outro

The GoldenSound youtube channel, and all content is created and owned by GoldenSound LTD

[ErinsAudioCorner]

I'm on board with the intent of this video. In any engineering test you document the procedures and the findings. That way someone who wants to repeat the test can do so and they should then arrive at the same result. Where there are differences are almost always attributed to the test procedure itself and not product variation. This is standard stuff in every field. Not just audio components. Anyone who takes issue with your encouragement for that are simply doing so out of spite and letting their emotions override logic.

So, yes, document the procedures and publish the results. Simple.

I also agree that often too much importance is placed on a single value metric (such as SINAD) and the other parts of the whole are tossed by the wayside.

However, you lost me at 27:30 with the statement that "two devices with the same SINAD levels can sound very different". While I agree this is certainly possible, you provided a generic statement with no footing. You spent the better part of 25 minutes making many valid points with examples. And then you dropped that statement as if it were an afterthought. I wish you had taken steps to provide an example of when you have seen this and then provide reasons why that was the cause (i.e., frequency response, different voltage levels needed to reach this SINAD match, etc). This, to me, is much more helpful in providing rationale for a) why measurements can be misleading and also b) how proper measurements can help us learn *why* Product A sounds different than Product B. In my opinion, that should be the goal of a reviewer using data to help supplement their review. I don't believe it serves much purpose to say "here is data, and by the way, this is what I heard" without at least attempting to reconcile those two things.

At any rate, I hope you understand where I'm coming from and don't take this personally. Overall, I'm in complete agreement. I just think this video would have benefited the community if you had taken the extra step to explain your ending statement (or you had simply left that out). Maybe that's a topic for a later video...

Take care,
Erin

[drewwilson]

As a retired measurement engineer with 42 years of experience I fully agree with this video. You have just scratched the surface. But the real take home on this subject is if the measurement is “excellent” then the likely hood of you hearing that difference is minimal given all the other compromises in a real word system. If there is a difference heard then it will be associated with a difference not covered by that particular measurement. Audio reproduction is complex and one must look at perhaps 20-50 different specific measurements to make a valid comparison. Keep up the excellent work.

[audiorick]

You have this ease of putting complex and specialized subjects in a language that the layman enthusiast or semi expert can understand. I may have gotten about 60% of what you were talking about but I completely got the point of what you were trying to demonstrate. Huge takeaway for me and major props to you 👊🏻

[MrEliwankenobi]

This has got to be one of the most important videos about product reviews (in general) EVER! One always has to have discernment in product reviews, regardless of who reviewed it and HOW the product was reviewed! Good job @GoldenSound! This is gold indeed! I’m gonna share this to all my friends!

[PiercingSight]

"May your air be most wiggly, " is the best sign-off ever.

[gioponti]

Thanks for great work, am glad you’re doing what you do, as some people on youtube etc seem to believe their test regime plus superficial listening test constitutes a complete metric for judgement on the quality of reproduction of recorded music, and as such replaces a longterm listening test.

[metal]

I hope the folks from AP see this. I am always pleased to see another unparalleled and highly technical GoldenSound video. Superb.

[jimfarrell]

Fantastic video. An excellent challenge to those in both the objectivist and subjective camps that not everything is black and white. Audio, like life, is nuanced and open to interpretation.

[hllwmn]

(trying to post this comment now for the 4th time, youtube is having issues or this channel is filtering comments .... anyway ... ) Excellent -- you get the GOLD medal for this legwork, GOLDENSOUND. Thanks!
BTW ... in your opening, you forgot to mention the other heavyweight, longtime-running OBJECTIVIST forum, hygrogenaudio. They do some in-house measurements but it is mostly a discussion forum of think-tanks.
BTW2 ... If Amir is reading this comment, then I highly encourage him to make a response video as he does with PS Audio.

[EthanWiner]

This is excellent, but with one complaint and a few comments. My complaint is that near the end around 27:40 you claim that two DACs with very high Sinad levels can sound very different. No, they can't. That tells me that the listening comparison was flawed. If the response of an audio device is flat within, say, 1/4 dB across the full range, and the sum of all artifacts is 90+ dB below the music, then the device is considered to be audibly transparent. Any coloration that is added can be measured, but nobody will ever hear it. So by definition all transparent devices sound alike. This then leads to my comment: Again, this video is excellent, and it's valuable to anyone who truly cares about audio. But I fear the sensationalist headline will give ammunition to the Luddite crowd who dismiss measurements and believe "just listen" is superior. Measuring beats listening every day of the week for many reasons. Though, as you point out, measurements can be skewed. (And some vendors just lie and make stuff up. I've seen that with acoustic treatment products.) So it's important for people to understand that once a device is clean enough to be audibly transparent, anything better doesn't matter. Nobody can hear the difference between distortion at 0.003 and 0.006 percent. And the real point is that it's not worth paying ten times more for something that measures better but doesn't actually sound better.

[hdmoviesource]

Excellent video, this will be bookmarked, watched, and rewatched. This is very good information which I think the public deserved to hear. Thank you.

[Neopheus]

We needed this video so much. This is part of why I never really paid attention to measurements. I give more credit to subjective feedback, especially when I could validate their impression by testing a piece of equipment myself. It worked for me and I am happy with what I have.

[Frequincy]

You continually show that you're one of the best audio gear reviewers! I'd love a deep dive into oversampling/NOS!

[ericharrelson]

This kind of information and your style of video presentation are exactly why joined your Patreon; informative indeed. Thanks

[TheGamerUnknown]

Excellent. It is so nice to see something informative like this put up. I think many consumers jump to conclusions regarding measurements without knowing the full picture. I hope this will change some minds.

[AntPDC]

Thank you Cameron. I would also add that while competently undertaken measurements provide a strong foundation and are crucial for understanding the technical aspects of audio equipment, they can only partially predict the subjective listening experience - some argue within a range of 50 - 70% accuracy. Therefore, while measurements are highly valuable, listening tests remain essential to gain the true picture of sonic performance.

[truepaulesko]

Greatwork, it shows that, as almost everything in life, one can not reduce things to a hand full of numbers. This is also applicable to cars for example.

[ResolveReviews]

Very interesting video! Thanks for doing this.

[jsharp]

Alternative title: "How to interpret and understand measurement settings".

[TheRacingSlick]

Really interesting, educational and helpful, thanks so much for this thorough walk through, it’s clear that you’ve invested many hours in putting this together!