Test Bench 1.9: Changelog

Goal

The primary goal of this test methodology update is to improve our contrast testing so that it better reflects the real-world user experience. To achieve this, we're updating our contrast methodology for the first time in over five years. We're replacing our local dimming test with four separate tests that better showcase the impact on picture quality. Our current local dimming test aimed to test local dimming as a feature of the TV—but going forward, we want to move away from testing features and instead focus on the ways those features impact your usage of the TV.

We also made a change in the settings used for our contrast measurements and all related tests. Previously, once a TV had a local dimming feature, all of our measurements would be taken with the best local dimming setting on that TV. We've nuanced that a bit now, and if a TV's local dimming feature is too distracting, and we expect most users to keep it disabled (like with a bad edge-lit local dimming feature), our recommended setting can now be "Off". The exception to this is a TV that doesn't normally allow users to disable local dimming. If it can't be disabled without going into a service menu, we'll always test the TV with it on, even if it's bad. If we decide to test a TV with local dimming disabled—which should be rare—we'll also include photos with the local dimming feature enabled.

The changes we've made to the Contrast and Local Dimming tests are some of the most significant changes we've ever made to our TV test methodology. Unfortunately, this makes it more difficult to compare current reviews with older TVs. We were aware that the local dimming changes would break comparability with previous reviews. It's something that can't be avoided, and we had to do it at one point. Most TVs we've tested over the last two years will be updated to this methodology, and they'll be comparable to all future reviews.

We also made a few other minor changes to our TV test methodology, including a few minor quality-of-life changes. We've removed the Temporary Image Retention and Permanent Burn-In Risk boxes, as that information is now covered in our Accelerate Longevity Test, and we removed a few minor tests that were no longer relevant to TVs. Finally, with all of these changes to tests, we made a few changes to our usage scores.

Summary of Changes

Below is a brief summary of the changes we've made in this update. More detailed information on the changes made to each test is available below this table.

Test Group	Change
Contrast	Replaced the checkerboard pattern with a modified real scene Shifted weight from native contrast to contrast
Local Dimming	Removed Split into Blooming, Black Crush, and Lighting Zone Transitions
Blooming	New test, replaces Local Dimming
Black Crush	New test, replaces Local Dimming
Lighting Zone Transitions	New test, replaces Local Dimming
Contrast And Dark Details In Game Mode	Renamed, was Local Dimming In Game Mode If the TV supports it, we now test this with the TV's variable refresh rate feature enabled Removed the local dimming real scene video reel Removed duplicate local dimming and backlight test results Adding blooming picture in game mode Added a video running in SmoothFrog to validate if VRR actually works with local dimming
Black Uniformity	Shifted weight from native uniformity to uniformity with local dimming
Temporary Image Retention	Test removed
Permanent Burn-In Risk	Test removed
Variable Refresh Rate	Removed VRR Supported Connectors Added a VRR + Local Dimming test
Total Inputs	Removed SD/SDHC

Details of Changes

Contrast - New Methodology

In the contrast box, we replaced the existing checkerboard pattern with a modified real-scene test. Even though the checkerboard pattern is an industry standard for contrast testing, it's become clear over the years that it's too simple of a scene to accurately represent the contrast of a TV. We've also increased the brightness we use for testing from 100 to 200 cd/m².

Like before, we take two sets of measurements for this test. The Contrast measurement represents the best user experience, with our recommended local dimming setting for that TV. This is done using the new real-scene photo. We measure the black level of the four corners of the test image and take the average. We combine this with the white level in the center of the white area to arrive at the final measurement. Many of you have asked us to include the Black Level in our reviews. We don't currently have any plans to include this number, but you can easily calculate it by dividing 200 by the Contrast measurement. The Native Contrast measurement is done the same way, but this time we take it with the TV's local dimming feature fully disabled. Unlike our Contrast measurement, this is always done with local dimming disabled, even if we have to go into the service menu to do so.

We also made a significant change in the way we score this box. When we first came up with our contrast scoring, local dimming features on TVs were very basic, with only a few zones, and they weren't very effective. At the same time, our usage scores gave an unfair disadvantage to TVs that didn't have a local dimming feature. Given two TVs that performed similarly in the dark, one with local dimming and one without, the one with local dimming would automatically score significantly higher. We were essentially giving bonus points to TVs just for having a local dimming feature—even if it was a bad one. Since local dimming technology has changed considerably since our last update to this test, we flipped the weights. 75% of the score is now based on the Contrast measurement with the recommended local dimming setting. Only 25% of the score comes from the Native Contrast, as we expect most users to use the recommended local dimming setting. We've also made changes to the scoring curves for each test, and our minimum "good" value for contrast has increased to better reflect the current TV market.

Before:

After:

Local dimming - Removed

As mentioned above, we decided to remove the local dimming box entirely. Instead of testing a specific feature of a TV, we've decided to split it into three separate tests: Blooming, Black Crush, and Zone Transitions, as these three picture quality tests showcase the different aspects of picture quality that are affected by local dimming. Combined, these three tests are approximately equal to the things we looked at in the old test, but they now better showcase the real-world performance of the TV, and they make it easier to compare TVs. It also helps to showcase that OLED displays aren't perfect, thanks to the new Black Crush test, whereas they used to always score a perfect 10 automatically for the old test.

Blooming - New Test

For the first of our new local dimming breakout tests, we're looking for blooming around bright areas of the screen with a dark background. We evaluate the TV using a real scene with hardcoded subtitles in SDR, with our recommended settings, in a completely dark room. Our testers assign a score subjectively based on how much blooming we see around the subtitles and around the lights on the top of the garage, and along the wall.

This test only looks for blooming, not other issues like raised blacks or low contrast. This means that TVs with no local dimming, or where our recommended local dimming setting is "Off", could score very high on this test, which can seem a bit counterintuitive.

Samsung QN900B QLED - Normal blacks levels, this image is close to the creator's intent.

LG NANO90 2021 - Bad blooming around subtitles and bright lights.

Black Crush - New Test

Processed image from the reference monitor

The black crush test isn't directly related to local dimming, but it's an important factor in overall picture quality that's closely related to contrast and local dimming. For this test, we display a real scene image in SDR on the screen in a completely dark room and look for fine details in certain areas of the scene. Like our previous tests, this is done with our recommended settings, including the recommended local dimming setting. Our testers have access to a reference monitor result, so they can compare the photo on the TV with the reference image. They assign a score subjectively based on how well fine shadow details are displayed and how closely it resembles the reference image. We take a photo of the image and slightly alter it to raise blacks a bit, so it's easier for the reader to see the nuances between TVs.

It's a difficult test to evaluate in an individual review, as the display you're using to view the photo could also be crushing blacks. The image above shows the resulting processed image taken from our Canon DP-V2411 reference display. Below you can see two examples, the one on the left isn't crushing blacks at all, and fine details in shadows are easy to make out. The TV on the right shows severely crushed blacks, and fine details are lost. Even if your display is also crushing blacks, by displaying these two images side-by-side, you can easily see the difference between them.

The results of this test so far have been interesting. OLED TVs like the LG A2, B2, and C2 crush blacks more than many TVs with LED backlights, something our previous contrast testing didn't expose. It's also important to remember that, like the blooming test, this test only looks at crushed blacks, so if a TV is raising blacks and looks washed out, it won't be penalized in this test.

Amazon Fire TV 4-Series - Normal blacks levels, this image is close to the creator's intent. Fine details in shadows are clear and easy to make out.

LG A2 OLED - Crushed blacks—the entire image is darker than it should be, and fine details are lost in shadows.

Zone Transitions - New Test

Although this box is new, the tests in it aren't really. The video is the same one we used to have in the Local Dimming box, but it's now scored subjectively based on how noticeable the zone transitions are. We look at both slow and fast content to see how quickly the zones light up. The backlight zones should light up as close as possible to the bright highlights. If they're too slow to change, there'll be a leading dark edge and a bright trail behind fast-moving objects.

By popular request, we've also started including the zone counts for the size of the TV we've tested. As most manufacturers don't provide this information, we use our zone transitions video, with the smallest highlight, to count the zones. It's usually accurate, but with new Mini LED TVs that have thousands of zones, we might occasionally be off by a bit. This doesn't impact our scoring, as more zones don't necessarily mean that the local dimming feature will be better.

As for the other sizes, we'll usually include this information in the text, as well as in the Differences Between Sizes And Variants section of our reviews.

Contrast And Dark Details In Game Mode - New Test

Some TVs degrade their local dimming and overall dark scene performance when they're in game mode. We repeat all of the above dark scene tests, but this time in game mode, with the recommended local dimming setting.
We look at all aspects of dark scene performance in game mode, including contrast, blooming, black crush, black uniformity, and zone transitions, and assign a subjective score based on the same subjective criteria we use to score the above tests. We use an internally weighted average to calculate a final score. Since we're using a weighted average of the above tests, the score for this box isn't directly comparable to any of the above tests. It's also important to keep in mind that the score represents the absolute performance of the TV; it's not relative to the performance out of game mode.

We take one photo and two videos to showcase the dark scene performance in game mode. The first photo looks for blooming, as some TVs adjust the number of virtual dimming zones in Game mode, so there can be more blooming as the zones are larger. We also film the zone transition video. Finally, if the TV supports a variable refresh rate, we use SmoothFrog with a custom pattern to look for tearing in game mode with local dimming enabled.

Black Uniformity - Scoring changes

Consistent with our changes to the Contrast scoring, we've changed the weights for this test, putting more emphasis on the Black Uniformity With Local Dimming results. We've also changed the settings used, and this test uses the same settings as the Contrast test, including the possibility of leaving local dimming disabled if it degrades the overall picture quality. No changes were made to the way we measure this, though, and the scoring curves for each test are the same, so although the score for the entire box is no longer comparable with our previous test methodology, the individual results for the Std. Dev. and the Native Std. Dev. are still comparable.

Temporary Image Retention - removed

We decided to remove this test, as we're indirectly testing for this as part of our long-term accelerated longevity test.

Permanent Burn-In Risk - removed

We decided to remove this test. We have a good understanding of the types of content that are most likely to cause burn-in from our Real-Life OLED Burn-In Test, which involved 6 LG OLED TVs and ran for two years. OLED technology has changed a lot since then, though, and with new heat reduction technologies, burn-in compensation mechanisms, and new panel types like QD-OLED, we just don't know how burn-in is affected by these new panels. We're running a new longevity test that indirectly tests for this. Although it's not one of the primary goals of this test, we're hoping to better understand how newer OLED panels compare to the older generation of OLEDs. It's generally accepted that burn-in isn't as much of an issue as it used to be, but it's unclear just how much better the newer OLED TVs are.

Variable Refresh Rate - Minor Changes

We made a few quality-of-life changes to the variable refresh rate box. We removed the VRR Supported Connectors box, as it's simply not relevant to TVs. This test is only relevant to monitors, which typically support HDMI and DisplayPort connections. TVs don't support DisplayPort, so this test is meaningless. Next, we added a test for VRR + Local Dimming. Some TVs disable their local dimming feature when the variable refresh rate feature is enabled, so we check to make sure that you can use both at the same time, and that there's no decrease in local dimming performance with VRR enabled.

Total Inputs - Minor Changes

Finally, we removed the SD/SDHC test from our inputs section. A few TVs allowed you to connect SD cards to view photos or videos from your camera directly on your TV, but it's been over 5 years since we last tested a TV that supported this.

63 TVs Updated

We have retested popular models. The test results for the following models have been converted to the new testing methodology. However, the text might be inconsistent with the new results.