Scoring System Overhaul

Before

After

Problem

Our scoring system was flawed:

• The overall score promoted a jack of all trades, master of none. Our headphone reviews exposed that problem. For example, the Sennheiser HD 800 S scored the best for sound quality but got a mediocre overall score because it leaks a lot. These headphones were specifically designed to leak and not isolate, with its open-back design.
• The subscores were too limited. For example, the HDR score only included three things: color gamut, peak brightness, and gradient. However, a lot of other stuff matters for HDR, like contrast or local dimming.
• Not everyone agrees with the weights and what's included. For example, sound quality was included in the overall score for TVs even though some people don’t care about this since they have an external sound system.

Solution

The solution that we have arrived at has multiple parts:

1. Get rid of the overall score. A perfect pair of headphones for every use doesn’t exist.
2. Restructure the reviews around the tests, not the usages. No more ‘Movies’ or ‘Living Room’ sections, but a ‘Picture Quality’ section instead. This removes redundancies for elements that are important in multiple usages.
3. Create multiple ratings. With the test results more isolated, it's now possible to completely separate the scoring from the tests. Each usage can now be scored with only what matters for that usage, with its own weights.

Custom Ratings

With this change, we're now able to add a new feature that has been requested by a lot of you for a long time: custom ratings. You can try it out here.

You can pick and choose from any of our scored test results, and assign your own weight to the things that you care about. This will produce a table that you can sort based on your own rating. For now, that tool is pretty simple, but we will add more options in the future.

Custom Ratings

Future

This opens up a lot of possibilities. However, the biggest drawback of this change is the increase in complexity. A lot of you are already overwhelmed by all the info present on our website, which causes analysis paralysis. This won’t help at first, but this change creates a solid foundation for what's coming in the future. In the next few months, we will make a lot of improvements to make your buying decision even easier than it was before.

Also, this places more importance on the accuracy of our measurements. Therefore, in the coming months, we'll continue to improve our testing and retest previously reviewed products. We’ve recently updated a few tests like peak brightness or local dimming, and a few other ones are already planned.

Feedback

As always, let us know what you think. We wouldn’t be here without your help. We are far from our goal of a perfect review website, so any suggestion can help.

You can send an email directly to the TV reviewers at tv@rtings.com, or to the headphones reviewers at headphones@rtings.com.

We're also hiring. If you like where we're going, check the job postings here.