You guys need to de-emphasize Frequency Response Consistency (FRC) when calculating overall scores. Yesterday.

While it’s a useful metric for people wanting to know how Frequency Response might change with positioning, it bears by far the least weight in determining how “neutral” a headphones presentation of music is/can be.

The fact that it’s weighted as highly as Treble Accuracy is, frankly, ridiculous for numerous reasons. Your weighting for Neutral Sound is all in need of serious retuning. Let’s start with FRC though:

1) FRC is something that, inherently, is a measure of potential, not present, results. Many people simply will not have the issues with FRC that the test specifically measures because most people will be putting the headphones on correctly. Which brings me to…

2) You do not share how you’re positioning the headphones, therefore the tests do not illustrate how the sound changes when a specific change is made (for example, like if yr using a pair of glasses to break the seal, a common occurrence for many in the audiophile hobby). This renders the usefulness for those who want to know how positioning changes tonal balance nil. For all we know you could be lifting the headphone away from the ear to completely break the seal, rotating it forward/backward on the X-axis, moving it up or down, etc. Which, yes, would change the tonal balance, but is not relevant to the experience of the end-user because no one is doing those things when putting headphones on. The point is we don’t know what you’re doing, how dramatically your positioning diverges from the normal position, and I’d argue having a test without indicating what change is correlated to what position is less valuable than having no test at all.

Now on to the other big problems:

3) The fact that Imaging is the lowest weighted metric, a measly 4% is transparently atrocious, especially because within this metric lies the sub-metric of channel amplitude/frequency matching (the most important element in someone’s perception of Left/Right positioning). Not only should this be much, MUCH higher than FRC in weight, but it should also be higher than Soundstage and Distortion. This quality you’ve deemed worthy of being only a sub-metric within Imaging is actually both more easily/consistently measurable, as well as very easily discernible to the end-user. Distortion is similarly measurable, but not easily discernible to most listeners (especially 2nd order harmonics that result from distortion, which is what I assume you are testing). Soundstage is discernible, but, while you’ve endeavored to great lengths to make it so (and I applaud you for it; not many have the cojones to attempt such a feat), it is not yet consistently measurable. And the simple fact is: Soundstage never extends more than a few inches our from your head with any headphone (with conventional non-binaural recordings), therefore weighing it so highly unnecessarily gimps ALL of your headphones scores. Especially IEMS. Because neither are as important to the end-user’s perception of “detail” or “neutrality,” I’d say it is absolutely critical that these things are re-weighted slightly or, preferably, massively, in addition to FRC.

4) Weighting FRC so high has the effect of rating headphones that don’t deserve a high rating, highly; and rating better headphones poorly. This is a massive problem, by my estimation, simply because the point of this site is purported to be a consumer aid to point us in the direction of the best things we can get for our money. Calling Audio-Technica’s M50x “The Best Closed Back for Audiophiles” is woefully incorrect, and has been for a few years. It’s terrible comfort (important for audiophiles, who listen for long periods), weird midrange presentation (wide boost between 100-200Hz, massive dip @300Hz, nasty sharp peak @4.3kHz), and poor channel matching have made it a joke in audiophile circles, who for years have dethroned it as the “default recommendation for a closed back headphone,” in favor of AKG K371

5) Speaking of which, AKG K371 has better Bass Accuracy and Mid Accuracy scores than M50x and somehow has a LOWER Neutral Sound score than ATH-M50x, which is by no metric a “neutral” headphone?! How is this possible?! Maybe the answer is somewhere in this post? ;-)

Anyways, knock Bass/Mid Accuracy to 25% weight each, Treble down to 15% (due to it being naturally difficult to measure and being more prone to deviation from person to person based on individual anatomy), Imaging up to 15% (because it is critically important for the reasons mentioned above), Distortion to 10%, Soundstage to 5% and either remove FRC completely and make Soundstage 10% or give it the last 5%. Your ratings will be more accurate to the Neutral Sound criteria you are purporting to try to meet, and will stop people from thinking HD800S is the most neutral open headphone (spoilier: it isn’t, it’s a treble detail cannon) and thinking ATH-M50x is the most neutral closed back (it isn’t).