Testing The Capabilities Of ANC Headphones

Bridging The Gap To Earmuffs

In this article, we'll explore the impact of headphone design on their noise cancelling capabilities and look at how headphone manufacturers approach the seemingly impossible task of blocking out wind noise. Check out the changelog if you want to skip the details and see what's changed in our tests.

The Difference Between ANC Headphones And Earmuffs

Our new and improved noise isolation tests developed for test bench v1.7 allow us to compare the performance of ANC headphones in a more fair and approachable way. As ANC algorithms change how they handle background noise, a simple sine sweep wasn't sufficient to evaluate the real-life performance of the headphones. Instead, we've updated our tests to use both pink noise signals and actual background scenarios to better test the capabilities of ANC headphones. We took this test bench update as an opportunity to put a pair of 3M Peltor X5A earmuffs through our new noise isolation tests to see how they fare against some of the best ANC headphones on the market.

ANC headphones perform quite well in the bass range, with the top performers attenuating more bass than the earmuffs. However, even the best ANC headphones come up short in the mid and treble range due to the incredible passive isolation of the earmuffs.

It should be noted that, unlike headphones, earmuffs are designed and certified to attenuate very loud noises. Even the language used to describe earmuffs varies quite a bit from that used to describe ANC headphones. Typically, earmuffs are advertised as hearing "protection" or "conservation" for use in loud work environments. On the other hand, ANC headphones are typically advertised to "block out" or "tune out" background noise, indicating a more casual use. Because of this, regardless of how ANC headphones perform, they aren't suitable as a replacement for earmuffs as hearing protection.

ANC vs. Passive Attenuation

Before comparing earmuffs and ANC headphones, we must first go over how these two different products attenuate noise. In the case of earmuffs (and any headphones that don't have ANC), they rely on what's called passive attenuation. Simply put, the materials, shape, size, and seal of the earmuff or headphones are what block background noise from entering the ear.

On the other hand, headphones with ANC, like the Sony WH-1000XM4 Wireless shown above, use microphones to listen to the background noise along with noise cancelling algorithms to play an "anti-noise" in the headphone. This anti-noise is an inverted version of the background noise, which is designed to cancel out the background sound waves through destructive interference before they ever reach your ear. In theory, this should perfectly cancel out the background noise; however, in practice, it works to varying degrees of success.

Although ANC can do a lot of noise attenuation, even ANC headphones rely on good passive noise attenuation to combine the effects and isolate the wearer as much as possible. For this reason, we test both passive and active noise isolation so that you can get a good idea of how the headphones achieve their isolation.

Comparing Performance

The following graph presents the attenuation performance of some of the most popular ANC headphones we've tested so far on our new test bench. We use a full-range pink noise signal because it allows us to account for the ANC response time that some headphones exhibit (but more on that later!). The result is measured as an audio volume reduction in decibels. The headphones tested perform quite well in the bass range with ANC enabled. Noises in this range would include airplane noise, a car engine's rumble, or an AC unit's low hum. However, once we reach an audio frequency of about 1 kHz, the difference between attenuation when ANC is on and when ANC is off becomes much less noticeable. Mid-to-high frequency noises like a person talking, dogs barking, or the screech of car brakes will be attenuated similarly whether ANC is on or off.

So how do the 3M Peltor X5A earmuffs, top-of-the-line earmuffs with an NRR (noise reduction rating) of 31dB, stack up against the attenuation of ANC headphones? When we compare the results of the same headphones above to the earmuffs, we begin to see where their strengths and weaknesses lie. The ANC headphones attenuate quite well in the bass range, with all of them attenuating more than the earmuffs. The earmuffs, however, perform noticeably better in the mid and treble range due to their superior passive attenuation.

There's such a dramatic difference in the volume reduction performance within the bass range because bass frequencies are among the most difficult to attenuate passively. The longer waves in the bass range pass more easily through materials and are, therefore, challenging to absorb. This is also the most effective range for ANC algorithms to attenuate. Looking at all our noise isolation results, ANC seems to consistently outperform passive isolation below 1 kHz. The result is that ANC headphones can use their ANC to attenuate noise in the bass range while relying on passive isolation to account for noises in the high-mids and treble.

Earmuffs don't have ANC to lean on when attenuating noise in the bass range (though there are indeed ANC earmuffs on the market). The 3M Peltor X5A earmuffs must make use of their physical characteristics to attenuate noise across the entire audible frequency range. Factors that influence an earmuff's performance include ear cup depth, clamping force, insulating materials, seal, and more (sources: Canadian Centre for Occupational Health and Safety and Paakkonen, 1991).

The Peltor X5A have very deep ear cups, strong clamping force, thick foam padding, and very well-sealing ear cups. All of these combined factors make the passive noise isolation performance so strong.

Why Don't Headphones Copy Earmuffs?

If earmuffs are so much better at passive attenuation than headphones, it's logical to ask why headphone manufacturers don't simply copy the design principles of earmuffs. The answer is earmuffs are generally less comfortable, especially for long durations.

High clamping force, usually required to keep a strong seal, is one of the main factors that make earmuffs less comfortable than most headphones. Among the 24 headphones tested so far on our new test bench, the average clamping force is about 1 lb. The 3M Peltor X5A earmuffs have a clamping force of about 2 lbs, which is 0.4 lbs more than the highest headphone clamping force among the 780 headphones we've tested so far. High clamping forces can be irritating if the product is used for long periods, something headphone wearers do regularly.

Earmuffs also make use of large ear cup depth and thick padding to help improve attenuation. This simply isn't feasible for modern headphones due to a demand for lighter, more portable options. Headphone wearers don't want to lug around heavy and cumbersome headphones, so manufacturers need to make the trade-off between size, weight, and passive noise isolation performance.

For these reasons, headphone manufacturers are unlikely to copy too much from the design of earmuffs to keep headphones comfortable and portable. There are some design decisions on earmuffs that are mirrored in many headphones, though. One is the ear cup material and how well it seals against the wearer's head.

As a good comparison, we did a spot-check with the Logitech G PRO X 2 LIGHTSPEED Wireless headphones. While they don't have ANC, they come with two pairs of interchangeable ear cup materials: cloth and leatherette. Besides the material difference, the ear cups are identical, giving us a great apples-to-apples comparison of how ear cup material impacts attenuation.

It's clear that the leatherette ear cup is much better at attenuating noise than the cloth ear cup. The massive improvement in attenuation that the leatherette ear cup provides in the mid and treble range makes it clear that the seal provided by leather and leatherette ear cups improves isolation.

You will notice that almost all ANC headphones on the market use a leather/leatherette material for the ear cup, one of the reasons being improved noise isolation. As you can see from the list below, seven of our eight top-performing over-ear ANC headphones have ear cups that are leather/leatherette. These materials allow for a consistently good seal for most wearers and provide the best conditions for strong noise isolation.

The only headphones that don't use the leather/leatherette material are the Apple AirPods Max Wireless, which uses a woven fabric. It seems likely that Apple consciously focused on comfort and/or breathability with the woven fabric ear cups. However, the trade-off is that they might leave some passive isolation performance on the table.

ANC Headphones Behave Differently In Common Scenarios

More and more manufacturers are producing ANC headphones that behave differently than what our initial tests were designed for. Similar to what we observe with TV response times, some headphones take time to adapt to the incoming background noise and even have an overshoot of their target attenuation. In practice, this means that some headphones will attenuate less background noise for a few seconds until they have time to respond and adapt to the external noise. This behavior indicated that we needed to adapt our testing methodology to allow time for these headphones to reach a steady state before assessing their ANC performance. This also means that headphones exhibiting this behavior might perform a bit worse when the background noise changes frequently.

AirPods Max vs WH-1000XM4

One of the most popular ANC headphones, the Apple AirPods Max, exhibit an overshoot that is audible when a new constant background noise is introduced. The best way to experience this is to compare it to the Sony WH-1000XM4, which don't have any noticeable overshoot. Below are audio samples of both headphones with airplane noise playing (audio samples used are from ETSI, accessible via the ETSI Open database).

You'll notice that the AirPods Max audio sample starts louder, then levels off to a constant level after about two to three seconds, giving the impression that they're listening and responding to the current background noise. This can be best visualized over time by playing a 1000Hz tone at a constant level, which should highlight the AirPods Max's capacity to adapt. The graph below shows the loudness in dB SPL over time (not to be confused with the reduction seen in the previous graphs) of the Sony WH-1000XM4 and the Apple AirPods Max subjected to the 1000 Hz tone. The AirPods Max have a clear overshoot that lasts about three seconds and settles to about 33 dB SPL. The WH-1000XM4 don't take time to adapt, and they plateau at about 49 dB SPL. These results align with our Full Range test results, which indicate that the AirPods Max attenuate better at 1000 Hz.

It should be noted that by using a 1000Hz tone, there's only one sound wave to handle, while in real content, the AirPods Max would have worse overall attenuation since they need to adjust to varying frequencies over time.

How we adapted our tests for overshoot

Our previous methodology used a sine sweep to test the headphones for all frequencies in the audible range. Although this method works well for headphones that don't need time to react, headphones like the AirPods Max were unfairly penalized due to their response time. To avoid scoring the overshoot in our reviews, we adapted our noise isolation tests to provide headphones with time to respond to the stimuli.

For our Noise Isolation – Full Range test, we use a pink noise signal at 60, 70, and 80 dBA and allow 5 seconds before recording for the headphones to respond to the noise. As well as being constant over time, pink noise is used because it has equal energy per octave, which is perceived as flat for humans. We can track the difference between our new and old tests by comparing the score gap between the AirPods Max and the WH-1000XM4. Although we made other changes, like a new speaker set-up and a more advanced head simulator (B&K 5128 Type B), the gap between the AirPods Max and WH-1000XM4 is more in line with how they perform in the real world.

A brand new test added in v1.7 is Noise Isolation – Common Scenarios, where we test using airplane noise, office noise, and street noise. What stands out is that the gap in scoring between the AirPods Max and WH-1000XM4 is much larger than in the Full Range test. This is likely for multiple reasons, one of which is that the common scenario audio samples are more bass-heavy, and the WH-1000XM4 attenuate the bass range better than the AirPods Max.

Another key reason the Apple AirPods Max perform a bit worse in common scenarios is that if we focus on the street noise sample, much of the noise is changing quickly, so the AirPods Max likely can't respond in time to attenuate as much as they would for a constant noise. Voices are another example of sound that changes quickly in tone and volume. To account for this, we've added a few additional recordings in our new Noise Isolation – Voice Handling test so that you can directly compare how headphones attenuate the voices of those around you. Although it's not scored, you can easily listen to how each model attenuates human voices with pink noise playing in the background to simulate pink noise. This is another way we test noise that isn't constant, and it provides a unique challenge for headphones that need time to respond. The original voice recordings are from Peter Kabal, Telecommunications & Signal Processing Laboratory Multimedia Signal Processing at McGill University.

Having to take time to respond to background noises seems to be a drawback for headphones that are attempting to cancel noises that change constantly. Despite that, it's also possible that it isn't a drawback but rather an intended feature. This could increase the listener's awareness of their surroundings by allowing sudden sounds to pass through. We aren't sure whether this trend will continue in future headphone designs or if this is a limitation for current headphones that only some models exhibit. Either way, our new tests should allow us to fairly compare headphones for the foreseeable future, whether they need time to adjust or not.

Even The Best ANC Headphones Still Struggle With Wind

You wouldn't be alone in thinking that headphones with good ANC can reduce the noise of wind. However, there's little correlation between ANC performance and wind handling. One of the biggest pitfalls that ANC headphones fall into is that when introduced to wind, they can make more noise than if they didn't have ANC at all. This noise is a major concern for people who like to be active outdoors with their headphones but don't want to be deafened by wind.

The best example of this wind problem is the Sony WH-1000XM4 headphones. As previously mentioned, the WH-1000XM4 are top-performing ANC headphones; however, compared to other premium ANC headphones, they struggle significantly with wind noise. Thanks to our new testing methodology, we created a test to allow you to listen for yourself what it sounds like to wear the WH-1000XM4 in a windy environment compared to the 3M Peltor X5A earmuffs (you may need to increase your volume to hear the clips):

Even earmuffs will exhibit some small amount of wind noise, but it isn't to the same degree as seen in ANC headphones. With earmuffs, most of the wind noise is blocked by passive isolation, and the wearer ends up hearing the small amount of noise that gets through. To understand the interaction between the ANC headphones and wind, we must first understand the different types of ANC implementation.

Different types of ANC implementation

There are three primary types of ANC implementation for consumer ANC headphones: feedforward, feedback, and hybrid. Each of these implementations refers to where the ANC microphones are positioned on the headphones.

Feedforward is the most common type of ANC headphones. They'll have their ANC microphones located on the outside of the ear cup (or ear canal for in-ears/earbuds). The sound is captured before it gets to the wearer, is processed by the ANC algorithm, and then anti-noise is played through the speaker.

The next type of ANC implementation is feedback ANC, where the microphone is placed inside the ear cup (or ear canal for in-ears/earbuds). This method is more complex because the microphone inside the ear cup/canal will now pick up the audio being played out of the speaker. Because of this, the algorithm needs to account for the added step of filtering out the undesired noise from the desired audio being played. After that's done, the anti-noise can be added to the audio signal to attenuate the background noise.

Hybrid ANC, the third type, is simply a combination of feedback and feedforward techniques. In this case, the headphones will have microphones both outside and inside the speaker. These are typically the best at attenuating noise and are often found in top-of-the-line headphone models.

The cause of the increased wind noise

Now that we know where ANC microphones are typically located, we can better understand why wind is such an issue for so many ANC headphones. At first, it might seem counterintuitive that ANC headphones can't simply cancel out wind noise like any other noise. However, it's not as simple as it seems.

Compared to typical background noises, wind noise doesn't come from an external source; it's generated through the physical interaction of the wind with the headphones. Keep in mind this is different from the noise generated when wind hits other objects, like trees or windows. Those sounds will reach the headphones the same as any other external noise. The interaction between wind and the headphones creates a unique problem for many ANC headphones, specifically those that have microphones on the outside of the ear cup/canal (feedforward and hybrid).

When wind hits a pair of headphones, the incident velocity is important in determining if there will be noise. At low velocity, wind will tend to stay in a laminar flow, not causing much noise. If the wind begins to move fast enough over the headphones, it can create turbulent airflow, which creates a shift in pressure over the microphones (source). The microphones interpret this change in pressure as noise, so the ANC algorithm counters the perceived noise with anti-noise. Additionally, since the airflow is turbulent, the velocity and pressure at any given point are likely to be different, so if there are multiple microphones, each microphone is recording a different perceived noise (source). The result is the ANC algorithm playing anti-noise in the headphones for a noise that would never have made it to your ear, resulting in louder noise than if the ANC weren't on in the first place.

How companies adapt their headphones for wind noise

One of the least expensive and most commonly used methods for dealing with ANC wind noise is to physically block the wind from interacting with the microphones. As shown below, the middle microphone of the WH-1000XM4 is recessed and covered with a grille, both of which help physically isolate the microphone from the wind. Although this method works at lower wind speeds, more wind-blocking material is required at high speeds, and this becomes unrealistic to implement on a pair of headphones while keeping them a comfortable size and weight (source).

Another method for dealing with ANC wind noise is to adaptively turn off the ANC microphones when wind is present. For example, Bose advertises their Wind Block feature for the QuietComfort Ultra headphones, which they say mutes all microphones on the outside of the ear cups. On the other hand, Sony advertises their Automatic Wind Noise Reduction feature for their top-of-the-line WF-1000XM4 earbuds, which turns off some of the exterior microphones.

The obvious drawback to muting the microphones is the drop in noise cancelling performance for all sounds other than wind. Regarding wind noise, most consumer ANC headphones with external microphones either struggle or turn back to passive isolation, neither of which is perfect.

Conclusion

ANC headphones are an incredible way to listen to your favorite music or podcast while blocking out all the unwanted noise of your surroundings. We built our new test methodology to highlight ANC headphone performance and allow for a fairer and more representative ranking of headphones. Using our latest tests, we compared top-of-the-line ANC headphones to earmuffs. Though they're made for very different purposes, it's clear that ANC headphones can still take design inspiration from the incredible passive isolation of earmuffs.

Our new Full Range, Common Scenarios, and Voice Handling tests allow us to evaluate headphones that need time to respond to background noise. Though we don't know whether this is a limitation or feature, our tests are built to handle the behavior fairly and comparably.

Our new ANC Wind Handling test also highlights a glaring weakness of ANC: wind noise. Despite that, understanding the underlying cause of ANC wind noise allows some companies to handle it better than others.

We would love your thoughts on our new test methodology and how your favorite ANC headphones stack up!