How to Skew a Blind Listening Test

We've talked a lot about how to set up a listening test on this website. There has been a lot of buzz lately on the results of the Paradigm blind listening tests results. For those that aren't in the know, Paradigm put some of their least expensive ($300/pair) bookshelf speakers in a number of blind tests with a pair of speakers that cost over $1000 (reportedly B&Ws) and another cheaper pair of speakers (reportedly Klipsch). A reporter over at CEPro wrote up their experience. Let's be clear that we are NOT saying that Paradigm skewed their tests. We are NOT suggesting that the entire thing was rigged. What we are doing is taking this opportunity (striking while the iron is hot so to speak) to talk about how listening tests can be skewed (or affected) both purposefully and accidentally. 

Seeing is Believing

Obviously, the most harmful thing for a participant to have in a listening test is preconceptions. If they have already heard and formed an opinion about one or more of the speaker pairs, they won't be able to help but think that the test will support their initial conclusions. The best thing is if you can find people that have never heard the speakers in question and in fact have never heard an opinion from anyone else about the speakers in question. Not likely? We agree. So what can you do? Hide the speakers.

The "blind" test is one of the most common forms of punishment on a forum. Say you like a speaker (or amp or receiver or CD player)? All someone needs to do to discredit your (probably detracting) opinion is to ask, "Well, did you do a blind test?" Of course you didn't because no one does. But suddenly they can unilaterally discount any opinion you have because of your lack of rigorous research into your opinions. Fair? No. But it happens every day.

A blind test is (generally) where the persons involved in the listening test (the participants) are unaware of what they are reviewing. This usually takes the form of a screen across the front of the room hiding the speakers/equipment. There are such things as double blind tests. This is where the person that is conducting the test (the experimenter) also doesn't know what speakers/equipment is playing until the test is over. This second form is much harder to do because it usually requires some sort of computerized equipment to switch and monitor what is going on.

Are blind tests necessary? Well, you don't need one to "prove" that a small cubed speaker doesn't have the bass response of a floorstanding speaker. But in speakers of the same type, a blind test can take out such biases as price, preconceptions, aesthetics, and more. It certainly will make your conclusions more convincing (as is the case with the Paradigm test). But can those tests be skewed or manipulated? Oh yes.

Volume

One thing that is widely known but easily manipulated is the volume/quality connection. Consistently, people when hearing two different sources will equate loudness with quality. A small adjustment in volume can completely change the outcome of a test. Most tests these days start off with, "....and the speakers were level matched..."

Expectancy Bias

A little known though easily understood form of affecting a test is through expectancy bias. In short, if the experimenter (person running the test) expects a certain result, that result is more likely to be obtained. The classic example of how this works is counting horses. Most of us have seen horses (or animals) that can count on TV. Someone asks a mathematical question of the owner (standing nearby) and the animal stamps or barks out the correct answer. What has been shown, however, is that if the owner doesn't know the answer, neither does the animal. This is not because the owner hasn't been able to tutor the animal in high level calculus but because the animal is gleaning non-verbal and very subtle cues from the owner. The owner, without knowing it themselves, is conveying to the animal the correct answer. This isn't (usually) malicious but simple human nature.

This is why when Audioholics does blind listening tests (like the $1500-$2000 Floorstanding Speaker Shootout) we have the experimenter either behind the screen with the speakers or out of the room. An in-room experimenter is just too likely to unconsciously convey information. We also require that the experimenter answer as few questions during the test as possible with as few words as possible. It is just too easy for information to be unconsciously passed. With the Paradigm test, the experimenter (who we guess was a Paradigm employee) was at the front of the room controlling the switching of the speakers. He was using a computer but was in clear view of the listeners. While we aren't saying this in any way affected the results of their listening tests, it certainly could have.

Tuning the Room

Most Audioholics know that the room can play a large effect on the sound quality you experience from any given speaker. Rooms can also be tuned to make all speakers sound better or maybe even worse. In particular, two of our reviewers had heard a speaker in a room and thought they were, without a doubt, the worst speakers ever made. When those speakers were placed in a different room, that opinion changed to, "Well, they aren't that bad. I kinda like them." That's a huge difference just in a room. The paradigm test had a room, reportedly, that could be changed for hard or soft surfaces. But it is easy to envision a less scrupulous company picking and tuning a room to accentuate the strengths and hide the weaknesses of their speakers. One could also envision using the same sorts of techniques to make another speaker sound worse. Again, this is not an accusation but we are merely pointing out possibilities.

Along the same lines, there are tons of different room tuning systems out there. While not long ago it was manual EQ or nothing, now we've got nearly every receiver manufacturer with either their own propitiatory room correction system or one provided by a third party. It wouldn't be hard to use one of these only on the speaker you wanted to win. When we did our speaker shootout, we used the same outputs from a receiver into two different (but identical) amps. We then manually turned the amps on and off as requested by the participants. If we had used a room correction system on just one pair, the other would have suffered (presumably). We didn't but we could have. Of course, we didn't have anything invested in the results so there was no motivation to do so.

Switching Order

Audio memory is about 3 seconds long. That means you have about 3 seconds before your recollection of the quality/timbre of a sound starts to fade. This is why quick switches from one set of speakers/equipment is so important. Most everyone knows this and provides for it. But what isn't often discussed is the switching order. If you have 3 or more pairs of speakers in a lineup, you can make sure to switch them in such a way that favors one pair. For example, during our listening test, we had a pair that mostly were considered bright. But in one comparison they were described as painful and another they were described as articulate. This is the same speaker with the same material. The difference? The speaker they were compared to.

When described as painful, they were being compared to a speaker that was very laid back with not a lot of high end energy. In comparison, the "bright" speaker sounded much worse. Whereas in the other comparison, they were up against another speaker that was also fairly bright. When compared to that speaker, their "brightness" wasn't nearly as different and so not nearly as negatively reviewed. With multiple speakers in a test, it is easy to see how you could plan to switch them in such a way so that the speaker you are favoring will get a bit of a boost.

Music Selection

In the same vein, you could pick music/material that favors your speakers. The easiest way of combating this is to let the participants choose/bring their own music. Of course, then you risk people bringing inappropriate music (ZZ Top, while fun to listen to isn't exactly great testing material) to your test. Providing some while allowing for others would be prudent.

Comparators Chosen

The easiest way to skew a listening test is to make sure the speakers you are comparing to your own don't measure up OR are different enough sounding that they will not fare well. Sometimes, all it takes is one to sound different in a blind environment the color the responses. While we're not suggesting that you need days and days listening to speakers to know whether or not you like them, we think a quick switching session may favor a certain type of speaker. We haven't seen any research to back this up but we'd like to see someone take it on.

Conclusion

While some of these methods of skewing a listening test can be malicious, most (if not all) can be done accidentally or unconsciously. If you are a speaker manufacturer, it makes sense that you'd tune a room for your speakers. It also makes sense that you'd bring other speakers to your room rather than find a new neutral place for the test. Music selection could easily fall in the same category. Choosing comparators and switching order to be accidental or serendipitous and having the experimenter in the room could be a function of the layout of the room/equipment.

We always recommend you take with a grain of salt any listening test - if for no other reason than they were not conducted in your room. You won't really know whether or not you like a speaker/piece of gear until you get it home and try it out. We do hope that you will read such test reports with the same critical eye we do. It is very easy to skew some tests, even by the most well intentioned of people.