Small Room Syndrome

Let's talk about some of the coolest jobs on the planet. Guy who holds those reflectors on the SI Swimsuit photo shoots. Supermodel for that matter. Billionaire playboy that moonlights as a crime-fighter. But I guarantee that if you pin down any of those people and they'll complain about their jobs. Any of them. The reflector guy is bored, the supermodel is hungry, and the billionaire playboy is sleep deprived. But if you are an Audioholic, one of your dream jobs is audio reviewer. You can't think of anything better than getting paid for playing with the gear you love so much. Maybe second would be speaker/amp/whatever designer. But that second one probably requires more schooling. So let's talk about the review process.

• Step 1: Contact (or get contacted by) manufacturer
• Step 2: Receive gear

• Step 3: Write review

• Step 4: Publish review

Easy right? Well, there are a few extra steps in there. Usually between steps 1 and 2 there are a number of emails about what would be the best model to review for the audience and (if you're me) a bunch of phone calls to dodge from the manufacturer who wants to "prep" you for the review in an attempt to bias you toward the product. In other reviews on other sites (and print mags), they play this part up. They love to talk about how they hobnobbed with the best and the brightest at whatever company. Personally, it reeks of the introduction of bias to me so I avoid it at all costs. Usually between Steps 2 and 3 there is more of this attempted contact as well.

But I digress.

You only have to publish one review without the manufacturer seeing it first to cause an absolute crap-storm of trouble. No good ever comes of it. And honestly, it doesn't do the reader any good. While we know that others don't, we ALWAYS send our reviews to the manufacturer for technical review. This is the review, just as you see it on the site, without the ratings at the end. That's all we change. We then ask them to look it over and make sure there isn't anything that either we missed or that doesn't line up with what they know about their product. Very often we get a few grammar/spelling edits and a few notes asking for clarification but occasionally they'll suggest something we missed. Especially on a complicated product like a receiver or HTPC, it is easy to overlook something a manufacturer considers vital. We're happy to go back and take a look at it and include it in the review. When we've completed this process, we publish the review.

But sometimes something happens. Something bad. Something out of the manufacturer's control. We'll receive a product that is defective in some way. We put it through its paces and it bottoms out, catches on fire, or just doesn't perform as it is supposed to. At this point we have a problem. The problem is that we have a really bad review and a manufacturer that REALLY doesn't want it to hit the web. Well, you can't blame them for that can you? But they can cast blame. And they do. Here are some of the most famous reasons why it isn't the manufacturer's fault:

• We can't replicate that problem on our end (the "This is the first time anyone has ever had this issue" argument)
  
• We've never had a customer complain about this (which doesn't mean it isn't real and that we didn't already know about it, by the way)
  
• We are aware of this issue and addressed the problem in production units (but we were hoping you were like other less rigorous mags and wouldn't figure it out or the "Holy crap, we'd better put out a firmware update like NOW" excuse)
  
• We didn’t realize we sent you prototypes (because, you know, sometimes they ship themselves or, as I like to call it, the "Gremlin excuse")
• Hmmm, that crossover/amp/driver/feature was probably damaged in shipping (UPS, FedEx, and DHL are always easy scapegoats)
  

What this equates to is that they want us to send the product back so they can send out new models. In the early days of Audioholics, we were naive. We were trusting. We let them. But we were sobered up to the reality pretty quickly. We've seen upwards of seven different iterations of the same product before we finally called it quits. Of course, these same products got rave reviews from other review sites. Because those sites, and most consumers for that matter, suffer from what we've coined Small Room Syndrome.

Small Room Syndrome

With speakers, you can always tell if they have design flaws if you put them in a large, well dampened room - a place where you can actually supply them large amounts of power and run them up to their spec. You'll force them to play at their limits and, if they don't melt, smoke, or cause your amp to shut down, you can at least tell that they have no major design flaws. They may not perform well, but that's completely different. The question is whether there is a fundamental flaw in their design. When a manufacturer says they've "never heard of this problem from any of their customers" before, it simply means that we are pushing their products to their limits. Which, you have to admit, as reviewers we are expected to do. A review is supposed to be critical. And if it isn't, it really isn't worth much is it?

But most consumers won't run into the problems we do because they have the proverbial "small room." They aren't pushing their gear to the limit. They are probably using only a fraction of the features of their receiver, a minuscule amount of the amplifier power, and are listening at volumes that are far below the physical limits of their speakers. These design flaws never rear their ugly heads. Just like brake issues don't typically affect Toyota's... but when they do it is catastrophic.

So, what do we do? We end up at a crossroads. You, the reader, expect reviews. And if we give you nothing but positive, danceable, reviews, you'll see us for what we'd be: shills. But every manufacturer has a dud in there somewhere. You can't put out a line of eight receivers and not have one that isn't sort of overpriced or underfeatured. Plus, things really do happen in shipping, some products come off the line defective, and occasionally the "excuses" listed above are true. So we try to be as fair as possible. We'll give them a chance, or two, but then that's it. When you send a product out for review, you have to be prepared for a bad review. Of course we understand that it is like us telling you your baby looks like you dropped them down a well one too many times but them's the breaks. It's not our fault your child is ugly.

Plus, we have a responsibility to our readership. If we don't put products through their paces and find out exactly how well they perform, we're accused of pandering to our advertisers (even if the manufacturer isn't an advertiser). If we are too critical, the manufacturers either threaten to pull advertising or won't consider advertising. Plus, there is no pleasing the fanboys. If you pick one wrong word or phrasing, they are on you like you'd asked their girlfriend (as if they actually had one) for a lap dance. 

So there is only one solution. And it is our motto: Pursuing the Truth in Audio and Video. We're human, we make mistakes. And because we have a website and not a print mag, we have the ability to go back and correct them when they happen. In fact, it affords a manufacturer the chance to come out smelling like roses - they can fix a problem before it becomes too egregious and support their base with an even better product. But our goal has always been to stick by the truth. Because if we tell the truth, no matter how ugly it is, people can only complain. They can't say you lied or deceived. They'll try to prove you wrong but those that care about the truth, and we like to think our readership is mostly made up of those types, will see through the whining and BS to the facts. Some manufacturers will pull their advertising but we have to live with it. We've found that when that happens, someone else reads the same review and thinks, "Now there's a review site that isn't afraid to tell it like it is. I'm going to advertise with them." Sometimes a baby is ugly, sometimes a design is bad, and sometimes you just have to suck it up and realize that the messenger is not the problem, it's the flaming package of crap you put in his hands.